Hospitals and health systems are under mounting financial pressure to shorten hospitalizations and reduce readmissions. These priorities have led to an ever-increasing focus on postacute care (PAC), and more specifically on improving transitions from the hospital.^1,2 According to a 2013 Institute of Medicine report, PAC is the source of 73% of the variation in Medicare spending³ and readmissions during the postacute episode nearly double the average Medicare payment.⁴ Within the PAC landscape, discharges to skilled nursing facilities (SNFs) have received particular focus due to the high rates of readmission and associated care costs.⁵

Hospitals, hospital physicians, PAC providers, and payers need to improve SNF transitions in care. Hospitals are increasingly responsible for patient care beyond their walls through several mechanisms including rehospitalization penalties, value-based reimbursement strategies (eg, bundled payments), and risk-based contracting on the total cost of care through relationships with accountable care organizations (ACOs) and Medicare Advantage plans. Similarly, hospital-employed physicians and PAC providers are more engaged in achieving value-based goals through increased alignment of provider compensation models^6,7 with risk-based contracting.

Current evidence suggests that rehospitalizations could be reduced by focusing on a concentrated referral network of preferred high-quality SNFs;^8,9 however, less is known about how to develop and operate such linkages at the administrative or clinical levels.⁸ In this article, we propose a collaborative framework for the establishment of a preferred PAC network.

One mechanism employed to improve transitions to SNFs and reduce associated readmissions is to create a preferred provider network. Increasing the concentration of hospital discharges to higher performing facilities is associated with lower rehospitalization rates, particularly during the critical days following discharge.¹⁰

While the criteria applied for preferred provider networks vary, there are several emerging themes.¹⁰ Quality metrics are often applied, generally starting with Centers for Medicare and Medicaid Services (CMS) quality star ratings and Long-Term Care Minimum Data Set (MDS) metrics with additional criteria frequently layered upon those. Some examples include the extent of physician coverage,¹¹ the extent of nursing coverage (eg, nursing ratios or 24/7 nursing care), geographic access, and flexible admission times (including weekends and nights).¹² In addition, several outcome measures may be used such as 30-day readmission rates, patient/family satisfaction ratings, ED visits, primary care follow-up within seven days of PAC discharge, or impact on the total cost of care.

Beyond the specified criteria, some hospitals choose to build upon existing relationships when developing their preferred network. By selecting historically high-volume facilities, they are able to leverage the existing name recognition amongst patients and providers.¹³ This minimizes retraining of discharge planners, maintains institutional relationships, and aligns with the patients’ geographic preferences.^2,13 While the high volume SNFs may not have the highest quality ratings, some hospitals find they can leverage the value of preferred partner status to push behavior change and improve performance.¹³

Here we propose a framework for the establishment of a preferred provider network for a hospital or health system based on the early experience of establishing an SNF Collaborative within Johns Hopkins Medicine (JHM). JHM is a large integrated health care system, which includes five hospitals within the region, including two large academic hospitals and three community hospitals serving patients in Maryland and the District of Columbia.¹⁴

JHM identified a need for improved coordination with PAC providers and saw opportunities to build upon successful individual hospital efforts to create a system-level approach with a PAC partnership sharing the goals of improving care and reducing costs. Additional opportunities exist given the unique Maryland all-payer Global Budget Revenue system managed by the Health Services Cost Review Commission. This system imposes hospital-level penalties for readmissions or poor quality measure performance and is moving to a new phase that will place hospitals directly at risk for the total Part A and Part B Medicare expenditures for a cohort of attributed Medicare patients, inclusive of their PAC expenses. This state-wide program is one example of a shift in payment structures from volume to value that is occurring throughout the healthcare sector.

Developing a formal collaboration inclusive of the five local hospitals, Johns Hopkins HealthCare (JHHC)—the managed care division of JHM—and the JHM ACO (Johns Hopkins Medicine Alliance for Patients, JMAP), we established a JHM SNF Collaborative. This group was tasked with improving the continuum of care for our patients discharged to PAC facilities. Given the number and diversity of entities involved, we sought to draw on efforts already managed and piloted locally, while disseminating best practices and providing added services at the collaborative level. We propose a collaborative multistakeholder model (Figure) that we anticipate will be adaptable to other health systems.

In the process of establishing the SNF Collaborative at JHM, there were a number of early challenges faced and lessons learned:

• In a large integrated delivery system, there is a need to balance the benefits of central coordination with the support for ongoing local efforts to promote partner engagement at the hospital and SNF level. The forums created within the collaborative governance structure can facilitate sharing of the prior health system, hospital or SNF initiatives to grow upon successes and avoid prior pitfalls.
• Early identification of risk-adjusted PAC data sources is central to the collaborative establishment and goal setting. This requires assessment of internal analytic resources, budget, and desired timeline for implementation to determine the optimal arrangement. Similarly, identification of available data sources to drive the analytic efforts is essential and should include a health information exchange, claims, and MDS among others.
• Partnering with local QIOs provides support for facility-level quality improvement efforts. They have the staff and onsite expertise to facilitate process implementation within individual SNFs.
• Larger preferred provider networks require considerable administrative support to facilitate communication with the entities, coordinate completion of network agreements, and manage the dissemination of SNF- and hospital-specific performance data.
• Legal and contractual support related to data sharing and HIPAA compliance is needed due to the complexity of the health system and SNF legal structure. Multiple JHM legal entities were involved in this collaborative as were a mixture of freestanding SNFs and corporate chains. There was a significant effort required to execute both data-sharing agreements as well as charters to enable QIO participation.
• Physician leadership and insight are key to implementing meaningful and broad change. When devising system-wide solutions, incorporation and respect for local processes and needs are paramount for provider engagement and behavior change. This process will likely identify gaps in understanding the PAC patient’s experience and needs. It may also reveal practice variability and foster opportunities for provider education on the needs of PAC teams and how to best facilitate quality transitions.

We proposed a framework for establishing a collaborative partnership with a preferred network of SNF providers. Depending on organizational readiness, significant upfront investment of time and resources could be needed to establish a coordinated network of SNF providers. However, once established, such networks can be leveraged to support ongoing process improvement efforts within a hospital or delivery system and can be used strategically by such health systems as they implement value-based health strategies. Furthermore, the lessons learned from transitions to SNFs can be applied more broadly in the PAC landscape including transitions to home from both the hospital and SNF.

Acknowledgments

The authors wish to acknowledge all the members and participants in the Johns Hopkins Medicine Skilled Nursing Facility Collaborative and the executive sponsors and JHM hospital presidents for their support of this work.

Disclosures

Michele Bellantoni receives intramural salary support for being the medical director of the JHM SNF Collaborative. Damien Doyle is a part-time geriatrician at the Hebrew Home of Greater Washington, a skilled nursing facility. He received travel expense support for GAPNA, a local Advanced Practice Nurse Association meeting.The authors otherwise have no potential conflicts of interest to disclose.

Funding

The authors state that there were no external sponsors for this work.

Hospitals and health systems are under mounting financial pressure to shorten hospitalizations and reduce readmissions. These priorities have led to an ever-increasing focus on postacute care (PAC), and more specifically on improving transitions from the hospital.^1,2 According to a 2013 Institute of Medicine report, PAC is the source of 73% of the variation in Medicare spending³ and readmissions during the postacute episode nearly double the average Medicare payment.⁴ Within the PAC landscape, discharges to skilled nursing facilities (SNFs) have received particular focus due to the high rates of readmission and associated care costs.⁵

Hospitals, hospital physicians, PAC providers, and payers need to improve SNF transitions in care. Hospitals are increasingly responsible for patient care beyond their walls through several mechanisms including rehospitalization penalties, value-based reimbursement strategies (eg, bundled payments), and risk-based contracting on the total cost of care through relationships with accountable care organizations (ACOs) and Medicare Advantage plans. Similarly, hospital-employed physicians and PAC providers are more engaged in achieving value-based goals through increased alignment of provider compensation models^6,7 with risk-based contracting.

Current evidence suggests that rehospitalizations could be reduced by focusing on a concentrated referral network of preferred high-quality SNFs;^8,9 however, less is known about how to develop and operate such linkages at the administrative or clinical levels.⁸ In this article, we propose a collaborative framework for the establishment of a preferred PAC network.

One mechanism employed to improve transitions to SNFs and reduce associated readmissions is to create a preferred provider network. Increasing the concentration of hospital discharges to higher performing facilities is associated with lower rehospitalization rates, particularly during the critical days following discharge.¹⁰

While the criteria applied for preferred provider networks vary, there are several emerging themes.¹⁰ Quality metrics are often applied, generally starting with Centers for Medicare and Medicaid Services (CMS) quality star ratings and Long-Term Care Minimum Data Set (MDS) metrics with additional criteria frequently layered upon those. Some examples include the extent of physician coverage,¹¹ the extent of nursing coverage (eg, nursing ratios or 24/7 nursing care), geographic access, and flexible admission times (including weekends and nights).¹² In addition, several outcome measures may be used such as 30-day readmission rates, patient/family satisfaction ratings, ED visits, primary care follow-up within seven days of PAC discharge, or impact on the total cost of care.

Beyond the specified criteria, some hospitals choose to build upon existing relationships when developing their preferred network. By selecting historically high-volume facilities, they are able to leverage the existing name recognition amongst patients and providers.¹³ This minimizes retraining of discharge planners, maintains institutional relationships, and aligns with the patients’ geographic preferences.^2,13 While the high volume SNFs may not have the highest quality ratings, some hospitals find they can leverage the value of preferred partner status to push behavior change and improve performance.¹³

Here we propose a framework for the establishment of a preferred provider network for a hospital or health system based on the early experience of establishing an SNF Collaborative within Johns Hopkins Medicine (JHM). JHM is a large integrated health care system, which includes five hospitals within the region, including two large academic hospitals and three community hospitals serving patients in Maryland and the District of Columbia.¹⁴

JHM identified a need for improved coordination with PAC providers and saw opportunities to build upon successful individual hospital efforts to create a system-level approach with a PAC partnership sharing the goals of improving care and reducing costs. Additional opportunities exist given the unique Maryland all-payer Global Budget Revenue system managed by the Health Services Cost Review Commission. This system imposes hospital-level penalties for readmissions or poor quality measure performance and is moving to a new phase that will place hospitals directly at risk for the total Part A and Part B Medicare expenditures for a cohort of attributed Medicare patients, inclusive of their PAC expenses. This state-wide program is one example of a shift in payment structures from volume to value that is occurring throughout the healthcare sector.

Developing a formal collaboration inclusive of the five local hospitals, Johns Hopkins HealthCare (JHHC)—the managed care division of JHM—and the JHM ACO (Johns Hopkins Medicine Alliance for Patients, JMAP), we established a JHM SNF Collaborative. This group was tasked with improving the continuum of care for our patients discharged to PAC facilities. Given the number and diversity of entities involved, we sought to draw on efforts already managed and piloted locally, while disseminating best practices and providing added services at the collaborative level. We propose a collaborative multistakeholder model (Figure) that we anticipate will be adaptable to other health systems.

In the process of establishing the SNF Collaborative at JHM, there were a number of early challenges faced and lessons learned:

• In a large integrated delivery system, there is a need to balance the benefits of central coordination with the support for ongoing local efforts to promote partner engagement at the hospital and SNF level. The forums created within the collaborative governance structure can facilitate sharing of the prior health system, hospital or SNF initiatives to grow upon successes and avoid prior pitfalls.
• Early identification of risk-adjusted PAC data sources is central to the collaborative establishment and goal setting. This requires assessment of internal analytic resources, budget, and desired timeline for implementation to determine the optimal arrangement. Similarly, identification of available data sources to drive the analytic efforts is essential and should include a health information exchange, claims, and MDS among others.
• Partnering with local QIOs provides support for facility-level quality improvement efforts. They have the staff and onsite expertise to facilitate process implementation within individual SNFs.
• Larger preferred provider networks require considerable administrative support to facilitate communication with the entities, coordinate completion of network agreements, and manage the dissemination of SNF- and hospital-specific performance data.
• Legal and contractual support related to data sharing and HIPAA compliance is needed due to the complexity of the health system and SNF legal structure. Multiple JHM legal entities were involved in this collaborative as were a mixture of freestanding SNFs and corporate chains. There was a significant effort required to execute both data-sharing agreements as well as charters to enable QIO participation.
• Physician leadership and insight are key to implementing meaningful and broad change. When devising system-wide solutions, incorporation and respect for local processes and needs are paramount for provider engagement and behavior change. This process will likely identify gaps in understanding the PAC patient’s experience and needs. It may also reveal practice variability and foster opportunities for provider education on the needs of PAC teams and how to best facilitate quality transitions.

We proposed a framework for establishing a collaborative partnership with a preferred network of SNF providers. Depending on organizational readiness, significant upfront investment of time and resources could be needed to establish a coordinated network of SNF providers. However, once established, such networks can be leveraged to support ongoing process improvement efforts within a hospital or delivery system and can be used strategically by such health systems as they implement value-based health strategies. Furthermore, the lessons learned from transitions to SNFs can be applied more broadly in the PAC landscape including transitions to home from both the hospital and SNF.

Acknowledgments

The authors wish to acknowledge all the members and participants in the Johns Hopkins Medicine Skilled Nursing Facility Collaborative and the executive sponsors and JHM hospital presidents for their support of this work.

Disclosures

Michele Bellantoni receives intramural salary support for being the medical director of the JHM SNF Collaborative. Damien Doyle is a part-time geriatrician at the Hebrew Home of Greater Washington, a skilled nursing facility. He received travel expense support for GAPNA, a local Advanced Practice Nurse Association meeting.The authors otherwise have no potential conflicts of interest to disclose.

Funding

The authors state that there were no external sponsors for this work.

Hospitals and health systems are under mounting financial pressure to shorten hospitalizations and reduce readmissions. These priorities have led to an ever-increasing focus on postacute care (PAC), and more specifically on improving transitions from the hospital.^1,2 According to a 2013 Institute of Medicine report, PAC is the source of 73% of the variation in Medicare spending³ and readmissions during the postacute episode nearly double the average Medicare payment.⁴ Within the PAC landscape, discharges to skilled nursing facilities (SNFs) have received particular focus due to the high rates of readmission and associated care costs.⁵

Hospitals, hospital physicians, PAC providers, and payers need to improve SNF transitions in care. Hospitals are increasingly responsible for patient care beyond their walls through several mechanisms including rehospitalization penalties, value-based reimbursement strategies (eg, bundled payments), and risk-based contracting on the total cost of care through relationships with accountable care organizations (ACOs) and Medicare Advantage plans. Similarly, hospital-employed physicians and PAC providers are more engaged in achieving value-based goals through increased alignment of provider compensation models^6,7 with risk-based contracting.

Current evidence suggests that rehospitalizations could be reduced by focusing on a concentrated referral network of preferred high-quality SNFs;^8,9 however, less is known about how to develop and operate such linkages at the administrative or clinical levels.⁸ In this article, we propose a collaborative framework for the establishment of a preferred PAC network.

One mechanism employed to improve transitions to SNFs and reduce associated readmissions is to create a preferred provider network. Increasing the concentration of hospital discharges to higher performing facilities is associated with lower rehospitalization rates, particularly during the critical days following discharge.¹⁰

While the criteria applied for preferred provider networks vary, there are several emerging themes.¹⁰ Quality metrics are often applied, generally starting with Centers for Medicare and Medicaid Services (CMS) quality star ratings and Long-Term Care Minimum Data Set (MDS) metrics with additional criteria frequently layered upon those. Some examples include the extent of physician coverage,¹¹ the extent of nursing coverage (eg, nursing ratios or 24/7 nursing care), geographic access, and flexible admission times (including weekends and nights).¹² In addition, several outcome measures may be used such as 30-day readmission rates, patient/family satisfaction ratings, ED visits, primary care follow-up within seven days of PAC discharge, or impact on the total cost of care.

Beyond the specified criteria, some hospitals choose to build upon existing relationships when developing their preferred network. By selecting historically high-volume facilities, they are able to leverage the existing name recognition amongst patients and providers.¹³ This minimizes retraining of discharge planners, maintains institutional relationships, and aligns with the patients’ geographic preferences.^2,13 While the high volume SNFs may not have the highest quality ratings, some hospitals find they can leverage the value of preferred partner status to push behavior change and improve performance.¹³

Here we propose a framework for the establishment of a preferred provider network for a hospital or health system based on the early experience of establishing an SNF Collaborative within Johns Hopkins Medicine (JHM). JHM is a large integrated health care system, which includes five hospitals within the region, including two large academic hospitals and three community hospitals serving patients in Maryland and the District of Columbia.¹⁴

JHM identified a need for improved coordination with PAC providers and saw opportunities to build upon successful individual hospital efforts to create a system-level approach with a PAC partnership sharing the goals of improving care and reducing costs. Additional opportunities exist given the unique Maryland all-payer Global Budget Revenue system managed by the Health Services Cost Review Commission. This system imposes hospital-level penalties for readmissions or poor quality measure performance and is moving to a new phase that will place hospitals directly at risk for the total Part A and Part B Medicare expenditures for a cohort of attributed Medicare patients, inclusive of their PAC expenses. This state-wide program is one example of a shift in payment structures from volume to value that is occurring throughout the healthcare sector.

Developing a formal collaboration inclusive of the five local hospitals, Johns Hopkins HealthCare (JHHC)—the managed care division of JHM—and the JHM ACO (Johns Hopkins Medicine Alliance for Patients, JMAP), we established a JHM SNF Collaborative. This group was tasked with improving the continuum of care for our patients discharged to PAC facilities. Given the number and diversity of entities involved, we sought to draw on efforts already managed and piloted locally, while disseminating best practices and providing added services at the collaborative level. We propose a collaborative multistakeholder model (Figure) that we anticipate will be adaptable to other health systems.

In the process of establishing the SNF Collaborative at JHM, there were a number of early challenges faced and lessons learned:

• In a large integrated delivery system, there is a need to balance the benefits of central coordination with the support for ongoing local efforts to promote partner engagement at the hospital and SNF level. The forums created within the collaborative governance structure can facilitate sharing of the prior health system, hospital or SNF initiatives to grow upon successes and avoid prior pitfalls.
• Early identification of risk-adjusted PAC data sources is central to the collaborative establishment and goal setting. This requires assessment of internal analytic resources, budget, and desired timeline for implementation to determine the optimal arrangement. Similarly, identification of available data sources to drive the analytic efforts is essential and should include a health information exchange, claims, and MDS among others.
• Partnering with local QIOs provides support for facility-level quality improvement efforts. They have the staff and onsite expertise to facilitate process implementation within individual SNFs.
• Larger preferred provider networks require considerable administrative support to facilitate communication with the entities, coordinate completion of network agreements, and manage the dissemination of SNF- and hospital-specific performance data.
• Legal and contractual support related to data sharing and HIPAA compliance is needed due to the complexity of the health system and SNF legal structure. Multiple JHM legal entities were involved in this collaborative as were a mixture of freestanding SNFs and corporate chains. There was a significant effort required to execute both data-sharing agreements as well as charters to enable QIO participation.
• Physician leadership and insight are key to implementing meaningful and broad change. When devising system-wide solutions, incorporation and respect for local processes and needs are paramount for provider engagement and behavior change. This process will likely identify gaps in understanding the PAC patient’s experience and needs. It may also reveal practice variability and foster opportunities for provider education on the needs of PAC teams and how to best facilitate quality transitions.

We proposed a framework for establishing a collaborative partnership with a preferred network of SNF providers. Depending on organizational readiness, significant upfront investment of time and resources could be needed to establish a coordinated network of SNF providers. However, once established, such networks can be leveraged to support ongoing process improvement efforts within a hospital or delivery system and can be used strategically by such health systems as they implement value-based health strategies. Furthermore, the lessons learned from transitions to SNFs can be applied more broadly in the PAC landscape including transitions to home from both the hospital and SNF.

Acknowledgments

The authors wish to acknowledge all the members and participants in the Johns Hopkins Medicine Skilled Nursing Facility Collaborative and the executive sponsors and JHM hospital presidents for their support of this work.

Disclosures

Michele Bellantoni receives intramural salary support for being the medical director of the JHM SNF Collaborative. Damien Doyle is a part-time geriatrician at the Hebrew Home of Greater Washington, a skilled nursing facility. He received travel expense support for GAPNA, a local Advanced Practice Nurse Association meeting.The authors otherwise have no potential conflicts of interest to disclose.

Funding

The authors state that there were no external sponsors for this work.

Ensuring the delivery of safe and cost-effective care is the core mission of hospitals,¹ but nearly 90% of unplanned patient transfers to critical care may be the result of a new or worsening condition.² The cost of treatment of sepsis, respiratory failure, and arrest, which are among the deadliest conditions for hospitalized patients,^3,4 are estimated to be $30.7 billion annually (8.1% of national hospital costs).⁵ As many as 44% of adverse events may be avoidable,⁶ and concerns about patient safety have motivated hospitals and health systems to find solutions to identify and treat deteriorating patients expeditiously. Evidence suggests that many hospitalized patients presenting with rapid decline showed warning signs 24-48 hours before the event.⁷ Therefore, ample time may be available for early identification and intervention in many patients.

As early as 1997, hospitals have used early warning systems (EWSs) to identify at-risk patients and proactively inform clinicians.⁸ EWSs can predict a proportion of patients who are at risk for clinical deterioration (this benefit is measured with sensitivity) with the tradeoff that some alerts are false (as measured with positive predictive value [PPV] or its inverse, workup-to-detection ratio [WDR]^9-11). Historically, EWS tools were paper-based instruments designed for fast manual calculation by hospital staff. Many aggregate-weighted EWS instruments continue to be used for research and practice, including the Modified Early Warning Systems (MEWS)¹² and National Early Warning System (NEWS).^13,14 Aggregate-weighted EWSs lack predictive precision because they use simple addition of a few clinical parameter scores, including vital signs and level of consciousness.¹⁵ Recently, a new category has emerged, which use multivariable regression or machine learning; we refer to this category as “EWSs using statistical modeling”. This type of EWS uses more computationally intensive risk stratification methods to predict risk¹⁶ by adjusting for a larger set of clinical covariates, thereby reducing the degree of unexplained variance. Although these EWSs are thought to be more precise and to generate fewer false positive alarms compared with others,^14,17-19 no review to date has systematically synthesized and compared their performance against aggregate-weighted EWSs.

Purpose

The purpose of this systematic review was to evaluate the recent literature regarding prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling versus aggregate-weighted systems.

Search Strategy

Adhering to PRISMA protocol guidelines for systematic reviews, we searched the peer-reviewed literature in PubMed and CINAHL Plus, as well as conference proceedings and online repositories of patient safety organizations published between January 1, 2012 and September 15, 2018. We selected this timeframe because EWSs using statistical modeling are relatively new approaches compared with the body of evidence concerning aggregate-weighted EWSs. An expert PhD researcher confirmed the search results in a blinded independent query.

Inclusion and Exclusion Criteria

We included peer-reviewed articles reporting the area under the receiver operator curve (AUC),²⁰ or the equivalent c-statistic, of models predicting clinical deterioration (measured as the composite of transfer to intensive care unit (ICU) and/or mortality) among adult patients in general hospital wards. We excluded studies if they did not compare an EWS using statistical modeling with an aggregate-weighted EWS, did not report AUC, or only reported on an aggregate-weighted EWS. Excluded settings were pediatrics, obstetrics, emergency departments, ICUs, transitional care units, and oncology. We also excluded studies with samples limited to physiological monitoring, sepsis, or postsurgical subpopulations.

Data Abstraction

Following the TRIPOD guidelines for the reporting of predictive models,²¹ and the PRISMA and Cochrane Collaboration guidelines for systematic reviews,^22-24 we extracted study characteristics (Table 1), sample demographics (Appendix Table 4), model characteristics and performance (Appendix Table 5), and level of scientific evidence and risk of bias (Appendix Table 6). To address the potential for overfitting, we selected model performance results of the validation dataset rather than the derivation dataset, if reported. If studies reported multiple models in either EWS category, we selected the best-performing model for comparison.

Measures of Model Performance

Because predictive models can achieve good case identification at the expense of high clinical workloads, an assessment of model performance would be incomplete without measures of clinical utility. For clinicians, this aspect can be measured as the model’s PPV (the percentage of true positive alerts among all alerts), or more intelligibly, as the WDR, which equals 1/PPV. WDR indicates the number of patients requiring evaluation to identify and treat one true positive case.^9-11 It is known that differences in event rates (prevalence or pretest probability) influence a model’s PPV²⁵ and its reciprocal WDR. However, for systematic comparison, PPV and WDR can be standardized using a fixed representative event rate across studies.^24,26 We abstracted the reported PPV and WDR, and computed standardized PPV and WDR for an event rate of 4%.

Other measures included the area under the receiver operator curve (AUC),²⁰ sensitivity, and specificity. AUC plots a model’s false positive rate (x-axis) against its true positive rate (y-axis), with an ideal scenario of very high y-values and very low x-values.²⁷ Sensitivity (the model’s ability to detect a true positive case among all cases) and specificity (the model’s ability to detect a true noncase among all noncases²⁸) are influenced by chosen alert thresholds. It is incorrect to assume that a given model produces only one sensitivity/specificity result; for systematic comparison, we therefore selected results in the 50% sensitivity range, and separately, in the 92% specificity range for EWSs using statistical modeling. Then, we simulated a fixed sensitivity of 0.51 and assumed specificity of 0.87 in aggregate-weighted EWSs.

Search Results

The PubMed search for “early warning score OR early warning system AND deterioration OR predict transfer ICU” returned 285 peer-reviewed articles. A search on CINAHL Plus using the same filters and query terms returned 219 articles with no additional matches (Figure 1). Of the 285 articles, we excluded 269 during the abstract screen and 10 additional articles during full-text review (Figure 1). A final review of the reference lists of the six selected studies did not yield additional articles.

Study Characteristics

There were several similarities across the selected studies (Table 1). All occurred in the United States; all compared their model’s performance against at least one aggregate-weighted EWS model;^14,17-19,29 and all used retrospective cohort designs. Of the six studies, one took place in a single hospital;²⁹ three pooled data from five hospitals;^17,18,30 and two occurred in a large integrated healthcare delivery system using data from 14 and, subsequently, 21 hospitals.^14,19 The largest study¹⁴ included nearly 650,000 admissions, while the smallest study²⁹ reported slightly less than 7,500 admissions. Of the six studies, four used multivariable regression,^14,17,19,29 and two used machine learning techniques for outcome prediction.^18,30

Outcome Variables

The primary outcome for inclusion in this review was clinical deterioration measured by the composite of transfer to ICU and some measure of mortality. Churpek et al.^10,11 and Green et al.³⁰ also included cardiac arrest, and Alvarez et al.²² included respiratory compromise in their outcome composite.

Researchers used varying definitions of mortality, including “death outside the ICU in a patient whose care directive was full code;”^14,19 “death on the wards without attempted resuscitation;”^17,18 “an in-hospital death in patients without a DNR order at admission that occurred on the medical ward or in ICU within 24 hours after transfer;”²⁹ or “death within 24 hours.”³⁰

Predictor Variables

We observed a broad assortment of predictor variables. All models included vital signs (heart rate, respiratory rate, blood pressure, and venous oxygen saturation); mental state; laboratory data; age; and sex. Additional variables included comorbidity, shock index,³¹ severity of illness score, length of stay, event time of day, season, admission category, and length of stay,^14,19 among others.

Model Performance

Reported PPV ranged from 0.16 to 0.42 (mean = 0.27) in EWSs using statistical modeling and 0.15 to 0.28 (mean = 0.19) in aggregate-weighted EWS models. The weighted mean standardized PPV, adjusted for an event rate of 4% across studies (Table 2), was 0.21 in EWSs using statistical modeling versus 0.14 in aggregate-weighted EWS models (simulated at 0.51 sensitivity and 0.87 specificity).

Only two studies^14,19 reported the WDR metric (alerts generated to identify one true positive case) explicitly. Based on the above PPV results, EWSs using statistical modeling generated a standardized WDR of 4.9 in models using statistical modeling versus 7.1 in aggregate-weighted models (Figure 2). The delta of 2.2 evaluations to find and treat one true positive case equals a 45% relative increase in RRT evaluation workloads using aggregate-weighted EWSs.

AUC values ranged from 0.77 to 0.85 (weighted mean = 0.80) in EWSs using statistical modeling, indicating good model discrimination. AUCs of aggregate-weighted EWSs ranged from 0.70 to 0.76 (weighted mean = 0.73), indicating fair model discrimination (Figure 2). The overall AUC delta was 0.07. However, our estimates may possibly be favoring EWSs that use statistical modeling by virtue of their derivation in an original research population compared with aggregate-weighted EWSs that were derived externally. For example, sensitivity analysis of eCART,¹⁸ an EWS using machine learning, showed an AUC drop of 1% in a large external patient population,¹⁴ while NEWS AUCs¹³ dropped between 11% and 15% in two large external populations (Appendix Table 7).^14,30 For hospitals adopting an externally developed EWS using statistical modeling, these results suggest that an AUC delta of approximately 5% can be expected and 7% for an internally developed EWS.

Risk of Bias Assessment

We scored the studies by adapting the Cochrane Collaboration tool for assessing risk of bias ³² (Appendix Table 5). Of the six studies, five received total scores between 1.0 and 2.0 (indicating relatively low bias risk), and one study had a score of 3.5 (indicating higher bias risk). Low bias studies^14,17-19,30 used large samples across multiple hospitals, discussed the choice of predictor variables and outcomes more precisely, and reported their measurement approaches and analytic methods in more detail, including imputation of missing data and model calibration.

In this systematic review, we assessed the predictive ability of EWSs using statistical modeling versus aggregate-weighted EWS models to detect clinical deterioration risk in hospitalized adults in general wards. From 2007 to 2018, at least five systematic reviews examined aggregate-weighted EWSs in adult inpatient settings.^33-37 No systematic review, however, has synthesized the evidence of EWSs using statistical modeling.

The recent evidence is limited to six studies, of which five had favorable risk of bias scores. All studies included in this review demonstrated superior model performance of the EWSs using statistical modeling compared with an aggregate-weighted EWS, and at least five of the six studies employed rigor in design, measurement, and analytic method. The AUC absolute difference between EWSs using statistical modeling and aggregate-weighted EWSs was 7% overall, moving model performance from fair to good (Table 2; Figure 2). Although this increase in discriminative power may appear modest, it translates into avoiding a 45% increase in WDR workload generated by an aggregate-weighted EWS, approximately two patient evaluations for each true positive case.

Results of our review suggest that EWSs using statistical modeling predict clinical deterioration risk with better precision. This is an important finding for the following reasons: (1) Better risk prediction can support the activation of rescue; (2) Given federal mandates to curb spending, the elimination of some resource-intensive false positive evaluations supports high-value care;³⁸ and (3) The Quadruple Aim³⁹ accounts for clinician wellbeing. EWSs using statistical modeling may offer benefits in terms of clinician satisfaction with the human–system interface because better discrimination reduces the daily evaluation workload/cognitive burden and because the reduction of false positive alerts may reduce alert fatigue.^40,41

Still, an important issue with risk detection is that it is unknown which percentage of patients are uniquely identified by an EWS and not already under evaluation by the clinical team. For example, a recent study by Bedoya et al.⁴² found that using NEWS did not improve clinical outcomes and nurses frequently disregarded the alert. Another study⁴³ found that the combined clinical judgment of physicians and nurses had an AUC of 0.90 in predicting mortality. These results suggest that at certain times, an EWS alert may not add new useful information for clinicians even when it correctly identifies deterioration risk. It remains difficult to define exactly how many patients an EWS would have to uniquely identify to have clinical utility.

Even EWSs that use statistical modeling cannot detect all true deterioration cases perfectly, and they may at times trigger an alert only when the clinical team is already aware of a patient’s clinical decline. Consequently, EWSs using statistical modeling can at best augment and support—but not replace—RRT rounding, physician workup, and vigilant frontline staff. However, clinicians, too, are not perfect, and the failure-to-rescue literature suggests that certain human factors are antecedents to patient crises (eg, stress and distraction,^44-46 judging by precedent/experience,^44,47 and innate limitations of human cognition⁴⁷). Because neither clinicians nor EWSs can predict deterioration perfectly, the best possible rescue response combines clinical vigilance, RRT rounding, and EWSs using statistical modeling as complementary solutions.

Our findings suggest that predictive models cannot be judged purely on AUC (in fact, it would be ill-advised) but also by their clinical utility (expressed in WDR and PPV): How many patients does a clinician need to evaluate?^9-11 Precision is not meaningful if it comes at the expense of unmanageable evaluation workloads, and our findings suggest that clinicians should evaluate models based on their clinical utility. Hospitals considering adoption of an EWS using statistical modeling should consider that externally developed EWSs appear to experience a performance drop when applied to a new patient population; a slightly higher WDR and slightly lower AUC can be expected. EWSs using statistical modeling appear to perform best when tailored to the targeted patient population (or are derived in-house). Model depreciation over time will likely require recalibration. In addition, adoption of a machine learning algorithm may mean that original model results are obscured by the black box output of the algorithm.^48-50

Findings from this systematic review are subject to several limitations. First, we applied strict inclusion criteria, which led us to exclude studies that offered findings in specialty units and specific patient subpopulations, among others. In the interest of systematic comparison, our findings are limited to general wards. We also restricted our search to recent studies that reported on models predicting clinical deterioration, which we defined as the composite of ICU transfer and/or death. Clinically, deteriorating patients in general wards either die or are transferred to ICU. This criterion resulted in exclusion of the Rothman Index,⁵¹ which predicts “death within 24 hours” but not ICU transfer. The AUC in this study was higher than those selected in this review (0.93 compared to 0.82 for MEWS; AUC delta: 0.09). The higher AUC may be a function of the outcome definition (30-day mortality would be more challenging to predict). Therefore, hospitals or health systems interested in purchasing an EWS using statistical modeling should carefully consider the outcome selection and definition.

Second, as is true for systematic reviews in general,⁵² the degree of clinical and methodological heterogeneity across the selected studies may limit our findings. Studies occurred in various settings (university hospital, teaching hospitals, and community hospitals), which may serve diverging patient populations. We observed that studies in university-based settings had a higher event rate ranging from 5.6% to 7.8%, which may result in higher PPV results in these settings. However, this increase would apply to both EWS types equally. To arrive at a “true” reflection of model performance, the simulations for PPV and WDR have used a more conservative event rate of 4%. We observed heterogenous mortality definitions, which did not always account for the reality that a patient’s death may be an appropriate outcome (ie, it was concordant with treatment wishes in the context of severe illness or an end-of-life trajectory). Studies also used different sampling procedures; some allowed multiple observations although most did not. The variation in sampling may change PPV and limit our systematic comparison. However, regardless of methodological differences, our review suggests that EWSs using statistical modeling perform better than aggregate-weighted EWSs in each of the selected studies.

Third, systematic reviews may be subject to the issue of publication bias because they can only compare published results and could possibly omit an unknown number of unpublished studies. However, the selected studies uniformly demonstrated similar model improvements, which are plausibly related to the larger number of covariates, statistical methods, and shrinkage of random error.

Finally, this review was limited to the comparison of observational studies, which aimed to answer how the two EWS classes compared. These studies did not address whether an alert had an impact on clinical care and patient outcomes. Results from at least one randomized nonblinded controlled trial suggest that alert-driven RRT activation may reduce the length of stay by 24 hours and use of oximetry, but has no impact on mortality, ICU transfer, and ICU length of stay.⁵³

Our findings point to three areas of need for the field of predictive EWS research: (1) a standardized set of clinical deterioration outcome measures, (2) a standardized set of measures capturing clinical evaluation workload and alert frequency, and (3) cost estimates of clinical workloads with and without deployment of an EWS using statistical modeling. Given the present divergence of outcome definitions, EWS research may benefit from a common “clinical deterioration” outcome standard, including transfer to ICU, inpatient/30-day/90-day mortality, and death with DNR, comfort care, or hospice. The field is lacking a standardized clinical workload measure and an understanding of the net percentage of patients uniquely identified by an EWS.

By using predictive analytics, health systems may be better able to achieve the goals of high-value care and patient safety and support the Quadruple Aim. Still, gaps in knowledge exist regarding the measurement of the clinical processes triggered by EWSs, evaluation workloads, alert fatigue, clinician burnout associated with the human-alert interface, and costs versus benefits. Future research should evaluate the degree to which EWSs can identify risk among patients who are not already under evaluation by the clinical team, assess the balanced treatment effects of RRT interventions between decedents and survivors, and investigate clinical process times relative to the time of an EWS alert using statistical modeling.

Acknowledgments

The authors would like to thank Ms. Jill Pope at the Kaiser Permanente Center for Health Research in Portland, OR for her assistance with manuscript preparation. Daniel Linnen would like to thank Dr. Linda Franck, PhD, RN, FAAN, Professor at the University of California, San Francisco, School of Nursing for reviewing the manuscript.

Disclosures

The authors declare no conflicts of interest.

Funding

The Maribelle & Stephen Leavitt Scholarship, the Jonas Nurse Scholars Scholarship at the University of California, San Francisco, and the Nurse Scholars Academy Predoctoral Research Fellowship at Kaiser Permanente Northern California supported this study during Daniel Linnen’s doctoral training at the University of California, San Francisco. Dr. Vincent Liu was funded by National Institute of General Medical Sciences Grant K23GM112018.

Ensuring the delivery of safe and cost-effective care is the core mission of hospitals,¹ but nearly 90% of unplanned patient transfers to critical care may be the result of a new or worsening condition.² The cost of treatment of sepsis, respiratory failure, and arrest, which are among the deadliest conditions for hospitalized patients,^3,4 are estimated to be $30.7 billion annually (8.1% of national hospital costs).⁵ As many as 44% of adverse events may be avoidable,⁶ and concerns about patient safety have motivated hospitals and health systems to find solutions to identify and treat deteriorating patients expeditiously. Evidence suggests that many hospitalized patients presenting with rapid decline showed warning signs 24-48 hours before the event.⁷ Therefore, ample time may be available for early identification and intervention in many patients.

As early as 1997, hospitals have used early warning systems (EWSs) to identify at-risk patients and proactively inform clinicians.⁸ EWSs can predict a proportion of patients who are at risk for clinical deterioration (this benefit is measured with sensitivity) with the tradeoff that some alerts are false (as measured with positive predictive value [PPV] or its inverse, workup-to-detection ratio [WDR]^9-11). Historically, EWS tools were paper-based instruments designed for fast manual calculation by hospital staff. Many aggregate-weighted EWS instruments continue to be used for research and practice, including the Modified Early Warning Systems (MEWS)¹² and National Early Warning System (NEWS).^13,14 Aggregate-weighted EWSs lack predictive precision because they use simple addition of a few clinical parameter scores, including vital signs and level of consciousness.¹⁵ Recently, a new category has emerged, which use multivariable regression or machine learning; we refer to this category as “EWSs using statistical modeling”. This type of EWS uses more computationally intensive risk stratification methods to predict risk¹⁶ by adjusting for a larger set of clinical covariates, thereby reducing the degree of unexplained variance. Although these EWSs are thought to be more precise and to generate fewer false positive alarms compared with others,^14,17-19 no review to date has systematically synthesized and compared their performance against aggregate-weighted EWSs.

Purpose

The purpose of this systematic review was to evaluate the recent literature regarding prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling versus aggregate-weighted systems.

Search Strategy

Adhering to PRISMA protocol guidelines for systematic reviews, we searched the peer-reviewed literature in PubMed and CINAHL Plus, as well as conference proceedings and online repositories of patient safety organizations published between January 1, 2012 and September 15, 2018. We selected this timeframe because EWSs using statistical modeling are relatively new approaches compared with the body of evidence concerning aggregate-weighted EWSs. An expert PhD researcher confirmed the search results in a blinded independent query.

Inclusion and Exclusion Criteria

We included peer-reviewed articles reporting the area under the receiver operator curve (AUC),²⁰ or the equivalent c-statistic, of models predicting clinical deterioration (measured as the composite of transfer to intensive care unit (ICU) and/or mortality) among adult patients in general hospital wards. We excluded studies if they did not compare an EWS using statistical modeling with an aggregate-weighted EWS, did not report AUC, or only reported on an aggregate-weighted EWS. Excluded settings were pediatrics, obstetrics, emergency departments, ICUs, transitional care units, and oncology. We also excluded studies with samples limited to physiological monitoring, sepsis, or postsurgical subpopulations.

Data Abstraction

Following the TRIPOD guidelines for the reporting of predictive models,²¹ and the PRISMA and Cochrane Collaboration guidelines for systematic reviews,^22-24 we extracted study characteristics (Table 1), sample demographics (Appendix Table 4), model characteristics and performance (Appendix Table 5), and level of scientific evidence and risk of bias (Appendix Table 6). To address the potential for overfitting, we selected model performance results of the validation dataset rather than the derivation dataset, if reported. If studies reported multiple models in either EWS category, we selected the best-performing model for comparison.

Measures of Model Performance

Because predictive models can achieve good case identification at the expense of high clinical workloads, an assessment of model performance would be incomplete without measures of clinical utility. For clinicians, this aspect can be measured as the model’s PPV (the percentage of true positive alerts among all alerts), or more intelligibly, as the WDR, which equals 1/PPV. WDR indicates the number of patients requiring evaluation to identify and treat one true positive case.^9-11 It is known that differences in event rates (prevalence or pretest probability) influence a model’s PPV²⁵ and its reciprocal WDR. However, for systematic comparison, PPV and WDR can be standardized using a fixed representative event rate across studies.^24,26 We abstracted the reported PPV and WDR, and computed standardized PPV and WDR for an event rate of 4%.

Other measures included the area under the receiver operator curve (AUC),²⁰ sensitivity, and specificity. AUC plots a model’s false positive rate (x-axis) against its true positive rate (y-axis), with an ideal scenario of very high y-values and very low x-values.²⁷ Sensitivity (the model’s ability to detect a true positive case among all cases) and specificity (the model’s ability to detect a true noncase among all noncases²⁸) are influenced by chosen alert thresholds. It is incorrect to assume that a given model produces only one sensitivity/specificity result; for systematic comparison, we therefore selected results in the 50% sensitivity range, and separately, in the 92% specificity range for EWSs using statistical modeling. Then, we simulated a fixed sensitivity of 0.51 and assumed specificity of 0.87 in aggregate-weighted EWSs.

Search Results

The PubMed search for “early warning score OR early warning system AND deterioration OR predict transfer ICU” returned 285 peer-reviewed articles. A search on CINAHL Plus using the same filters and query terms returned 219 articles with no additional matches (Figure 1). Of the 285 articles, we excluded 269 during the abstract screen and 10 additional articles during full-text review (Figure 1). A final review of the reference lists of the six selected studies did not yield additional articles.

Study Characteristics

There were several similarities across the selected studies (Table 1). All occurred in the United States; all compared their model’s performance against at least one aggregate-weighted EWS model;^14,17-19,29 and all used retrospective cohort designs. Of the six studies, one took place in a single hospital;²⁹ three pooled data from five hospitals;^17,18,30 and two occurred in a large integrated healthcare delivery system using data from 14 and, subsequently, 21 hospitals.^14,19 The largest study¹⁴ included nearly 650,000 admissions, while the smallest study²⁹ reported slightly less than 7,500 admissions. Of the six studies, four used multivariable regression,^14,17,19,29 and two used machine learning techniques for outcome prediction.^18,30

Outcome Variables

The primary outcome for inclusion in this review was clinical deterioration measured by the composite of transfer to ICU and some measure of mortality. Churpek et al.^10,11 and Green et al.³⁰ also included cardiac arrest, and Alvarez et al.²² included respiratory compromise in their outcome composite.

Researchers used varying definitions of mortality, including “death outside the ICU in a patient whose care directive was full code;”^14,19 “death on the wards without attempted resuscitation;”^17,18 “an in-hospital death in patients without a DNR order at admission that occurred on the medical ward or in ICU within 24 hours after transfer;”²⁹ or “death within 24 hours.”³⁰

Predictor Variables

We observed a broad assortment of predictor variables. All models included vital signs (heart rate, respiratory rate, blood pressure, and venous oxygen saturation); mental state; laboratory data; age; and sex. Additional variables included comorbidity, shock index,³¹ severity of illness score, length of stay, event time of day, season, admission category, and length of stay,^14,19 among others.

Model Performance

Reported PPV ranged from 0.16 to 0.42 (mean = 0.27) in EWSs using statistical modeling and 0.15 to 0.28 (mean = 0.19) in aggregate-weighted EWS models. The weighted mean standardized PPV, adjusted for an event rate of 4% across studies (Table 2), was 0.21 in EWSs using statistical modeling versus 0.14 in aggregate-weighted EWS models (simulated at 0.51 sensitivity and 0.87 specificity).

Only two studies^14,19 reported the WDR metric (alerts generated to identify one true positive case) explicitly. Based on the above PPV results, EWSs using statistical modeling generated a standardized WDR of 4.9 in models using statistical modeling versus 7.1 in aggregate-weighted models (Figure 2). The delta of 2.2 evaluations to find and treat one true positive case equals a 45% relative increase in RRT evaluation workloads using aggregate-weighted EWSs.

AUC values ranged from 0.77 to 0.85 (weighted mean = 0.80) in EWSs using statistical modeling, indicating good model discrimination. AUCs of aggregate-weighted EWSs ranged from 0.70 to 0.76 (weighted mean = 0.73), indicating fair model discrimination (Figure 2). The overall AUC delta was 0.07. However, our estimates may possibly be favoring EWSs that use statistical modeling by virtue of their derivation in an original research population compared with aggregate-weighted EWSs that were derived externally. For example, sensitivity analysis of eCART,¹⁸ an EWS using machine learning, showed an AUC drop of 1% in a large external patient population,¹⁴ while NEWS AUCs¹³ dropped between 11% and 15% in two large external populations (Appendix Table 7).^14,30 For hospitals adopting an externally developed EWS using statistical modeling, these results suggest that an AUC delta of approximately 5% can be expected and 7% for an internally developed EWS.

Risk of Bias Assessment

We scored the studies by adapting the Cochrane Collaboration tool for assessing risk of bias ³² (Appendix Table 5). Of the six studies, five received total scores between 1.0 and 2.0 (indicating relatively low bias risk), and one study had a score of 3.5 (indicating higher bias risk). Low bias studies^14,17-19,30 used large samples across multiple hospitals, discussed the choice of predictor variables and outcomes more precisely, and reported their measurement approaches and analytic methods in more detail, including imputation of missing data and model calibration.

In this systematic review, we assessed the predictive ability of EWSs using statistical modeling versus aggregate-weighted EWS models to detect clinical deterioration risk in hospitalized adults in general wards. From 2007 to 2018, at least five systematic reviews examined aggregate-weighted EWSs in adult inpatient settings.^33-37 No systematic review, however, has synthesized the evidence of EWSs using statistical modeling.

The recent evidence is limited to six studies, of which five had favorable risk of bias scores. All studies included in this review demonstrated superior model performance of the EWSs using statistical modeling compared with an aggregate-weighted EWS, and at least five of the six studies employed rigor in design, measurement, and analytic method. The AUC absolute difference between EWSs using statistical modeling and aggregate-weighted EWSs was 7% overall, moving model performance from fair to good (Table 2; Figure 2). Although this increase in discriminative power may appear modest, it translates into avoiding a 45% increase in WDR workload generated by an aggregate-weighted EWS, approximately two patient evaluations for each true positive case.

Results of our review suggest that EWSs using statistical modeling predict clinical deterioration risk with better precision. This is an important finding for the following reasons: (1) Better risk prediction can support the activation of rescue; (2) Given federal mandates to curb spending, the elimination of some resource-intensive false positive evaluations supports high-value care;³⁸ and (3) The Quadruple Aim³⁹ accounts for clinician wellbeing. EWSs using statistical modeling may offer benefits in terms of clinician satisfaction with the human–system interface because better discrimination reduces the daily evaluation workload/cognitive burden and because the reduction of false positive alerts may reduce alert fatigue.^40,41

Still, an important issue with risk detection is that it is unknown which percentage of patients are uniquely identified by an EWS and not already under evaluation by the clinical team. For example, a recent study by Bedoya et al.⁴² found that using NEWS did not improve clinical outcomes and nurses frequently disregarded the alert. Another study⁴³ found that the combined clinical judgment of physicians and nurses had an AUC of 0.90 in predicting mortality. These results suggest that at certain times, an EWS alert may not add new useful information for clinicians even when it correctly identifies deterioration risk. It remains difficult to define exactly how many patients an EWS would have to uniquely identify to have clinical utility.

Even EWSs that use statistical modeling cannot detect all true deterioration cases perfectly, and they may at times trigger an alert only when the clinical team is already aware of a patient’s clinical decline. Consequently, EWSs using statistical modeling can at best augment and support—but not replace—RRT rounding, physician workup, and vigilant frontline staff. However, clinicians, too, are not perfect, and the failure-to-rescue literature suggests that certain human factors are antecedents to patient crises (eg, stress and distraction,^44-46 judging by precedent/experience,^44,47 and innate limitations of human cognition⁴⁷). Because neither clinicians nor EWSs can predict deterioration perfectly, the best possible rescue response combines clinical vigilance, RRT rounding, and EWSs using statistical modeling as complementary solutions.

Our findings suggest that predictive models cannot be judged purely on AUC (in fact, it would be ill-advised) but also by their clinical utility (expressed in WDR and PPV): How many patients does a clinician need to evaluate?^9-11 Precision is not meaningful if it comes at the expense of unmanageable evaluation workloads, and our findings suggest that clinicians should evaluate models based on their clinical utility. Hospitals considering adoption of an EWS using statistical modeling should consider that externally developed EWSs appear to experience a performance drop when applied to a new patient population; a slightly higher WDR and slightly lower AUC can be expected. EWSs using statistical modeling appear to perform best when tailored to the targeted patient population (or are derived in-house). Model depreciation over time will likely require recalibration. In addition, adoption of a machine learning algorithm may mean that original model results are obscured by the black box output of the algorithm.^48-50

Findings from this systematic review are subject to several limitations. First, we applied strict inclusion criteria, which led us to exclude studies that offered findings in specialty units and specific patient subpopulations, among others. In the interest of systematic comparison, our findings are limited to general wards. We also restricted our search to recent studies that reported on models predicting clinical deterioration, which we defined as the composite of ICU transfer and/or death. Clinically, deteriorating patients in general wards either die or are transferred to ICU. This criterion resulted in exclusion of the Rothman Index,⁵¹ which predicts “death within 24 hours” but not ICU transfer. The AUC in this study was higher than those selected in this review (0.93 compared to 0.82 for MEWS; AUC delta: 0.09). The higher AUC may be a function of the outcome definition (30-day mortality would be more challenging to predict). Therefore, hospitals or health systems interested in purchasing an EWS using statistical modeling should carefully consider the outcome selection and definition.

Second, as is true for systematic reviews in general,⁵² the degree of clinical and methodological heterogeneity across the selected studies may limit our findings. Studies occurred in various settings (university hospital, teaching hospitals, and community hospitals), which may serve diverging patient populations. We observed that studies in university-based settings had a higher event rate ranging from 5.6% to 7.8%, which may result in higher PPV results in these settings. However, this increase would apply to both EWS types equally. To arrive at a “true” reflection of model performance, the simulations for PPV and WDR have used a more conservative event rate of 4%. We observed heterogenous mortality definitions, which did not always account for the reality that a patient’s death may be an appropriate outcome (ie, it was concordant with treatment wishes in the context of severe illness or an end-of-life trajectory). Studies also used different sampling procedures; some allowed multiple observations although most did not. The variation in sampling may change PPV and limit our systematic comparison. However, regardless of methodological differences, our review suggests that EWSs using statistical modeling perform better than aggregate-weighted EWSs in each of the selected studies.

Third, systematic reviews may be subject to the issue of publication bias because they can only compare published results and could possibly omit an unknown number of unpublished studies. However, the selected studies uniformly demonstrated similar model improvements, which are plausibly related to the larger number of covariates, statistical methods, and shrinkage of random error.

Finally, this review was limited to the comparison of observational studies, which aimed to answer how the two EWS classes compared. These studies did not address whether an alert had an impact on clinical care and patient outcomes. Results from at least one randomized nonblinded controlled trial suggest that alert-driven RRT activation may reduce the length of stay by 24 hours and use of oximetry, but has no impact on mortality, ICU transfer, and ICU length of stay.⁵³

Our findings point to three areas of need for the field of predictive EWS research: (1) a standardized set of clinical deterioration outcome measures, (2) a standardized set of measures capturing clinical evaluation workload and alert frequency, and (3) cost estimates of clinical workloads with and without deployment of an EWS using statistical modeling. Given the present divergence of outcome definitions, EWS research may benefit from a common “clinical deterioration” outcome standard, including transfer to ICU, inpatient/30-day/90-day mortality, and death with DNR, comfort care, or hospice. The field is lacking a standardized clinical workload measure and an understanding of the net percentage of patients uniquely identified by an EWS.

By using predictive analytics, health systems may be better able to achieve the goals of high-value care and patient safety and support the Quadruple Aim. Still, gaps in knowledge exist regarding the measurement of the clinical processes triggered by EWSs, evaluation workloads, alert fatigue, clinician burnout associated with the human-alert interface, and costs versus benefits. Future research should evaluate the degree to which EWSs can identify risk among patients who are not already under evaluation by the clinical team, assess the balanced treatment effects of RRT interventions between decedents and survivors, and investigate clinical process times relative to the time of an EWS alert using statistical modeling.

Acknowledgments

The authors would like to thank Ms. Jill Pope at the Kaiser Permanente Center for Health Research in Portland, OR for her assistance with manuscript preparation. Daniel Linnen would like to thank Dr. Linda Franck, PhD, RN, FAAN, Professor at the University of California, San Francisco, School of Nursing for reviewing the manuscript.

Disclosures

The authors declare no conflicts of interest.

Funding

The Maribelle & Stephen Leavitt Scholarship, the Jonas Nurse Scholars Scholarship at the University of California, San Francisco, and the Nurse Scholars Academy Predoctoral Research Fellowship at Kaiser Permanente Northern California supported this study during Daniel Linnen’s doctoral training at the University of California, San Francisco. Dr. Vincent Liu was funded by National Institute of General Medical Sciences Grant K23GM112018.

Ensuring the delivery of safe and cost-effective care is the core mission of hospitals,¹ but nearly 90% of unplanned patient transfers to critical care may be the result of a new or worsening condition.² The cost of treatment of sepsis, respiratory failure, and arrest, which are among the deadliest conditions for hospitalized patients,^3,4 are estimated to be $30.7 billion annually (8.1% of national hospital costs).⁵ As many as 44% of adverse events may be avoidable,⁶ and concerns about patient safety have motivated hospitals and health systems to find solutions to identify and treat deteriorating patients expeditiously. Evidence suggests that many hospitalized patients presenting with rapid decline showed warning signs 24-48 hours before the event.⁷ Therefore, ample time may be available for early identification and intervention in many patients.

As early as 1997, hospitals have used early warning systems (EWSs) to identify at-risk patients and proactively inform clinicians.⁸ EWSs can predict a proportion of patients who are at risk for clinical deterioration (this benefit is measured with sensitivity) with the tradeoff that some alerts are false (as measured with positive predictive value [PPV] or its inverse, workup-to-detection ratio [WDR]^9-11). Historically, EWS tools were paper-based instruments designed for fast manual calculation by hospital staff. Many aggregate-weighted EWS instruments continue to be used for research and practice, including the Modified Early Warning Systems (MEWS)¹² and National Early Warning System (NEWS).^13,14 Aggregate-weighted EWSs lack predictive precision because they use simple addition of a few clinical parameter scores, including vital signs and level of consciousness.¹⁵ Recently, a new category has emerged, which use multivariable regression or machine learning; we refer to this category as “EWSs using statistical modeling”. This type of EWS uses more computationally intensive risk stratification methods to predict risk¹⁶ by adjusting for a larger set of clinical covariates, thereby reducing the degree of unexplained variance. Although these EWSs are thought to be more precise and to generate fewer false positive alarms compared with others,^14,17-19 no review to date has systematically synthesized and compared their performance against aggregate-weighted EWSs.

Purpose

The purpose of this systematic review was to evaluate the recent literature regarding prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling versus aggregate-weighted systems.

Search Strategy

Adhering to PRISMA protocol guidelines for systematic reviews, we searched the peer-reviewed literature in PubMed and CINAHL Plus, as well as conference proceedings and online repositories of patient safety organizations published between January 1, 2012 and September 15, 2018. We selected this timeframe because EWSs using statistical modeling are relatively new approaches compared with the body of evidence concerning aggregate-weighted EWSs. An expert PhD researcher confirmed the search results in a blinded independent query.

Inclusion and Exclusion Criteria

We included peer-reviewed articles reporting the area under the receiver operator curve (AUC),²⁰ or the equivalent c-statistic, of models predicting clinical deterioration (measured as the composite of transfer to intensive care unit (ICU) and/or mortality) among adult patients in general hospital wards. We excluded studies if they did not compare an EWS using statistical modeling with an aggregate-weighted EWS, did not report AUC, or only reported on an aggregate-weighted EWS. Excluded settings were pediatrics, obstetrics, emergency departments, ICUs, transitional care units, and oncology. We also excluded studies with samples limited to physiological monitoring, sepsis, or postsurgical subpopulations.

Data Abstraction

Following the TRIPOD guidelines for the reporting of predictive models,²¹ and the PRISMA and Cochrane Collaboration guidelines for systematic reviews,^22-24 we extracted study characteristics (Table 1), sample demographics (Appendix Table 4), model characteristics and performance (Appendix Table 5), and level of scientific evidence and risk of bias (Appendix Table 6). To address the potential for overfitting, we selected model performance results of the validation dataset rather than the derivation dataset, if reported. If studies reported multiple models in either EWS category, we selected the best-performing model for comparison.

Measures of Model Performance

Because predictive models can achieve good case identification at the expense of high clinical workloads, an assessment of model performance would be incomplete without measures of clinical utility. For clinicians, this aspect can be measured as the model’s PPV (the percentage of true positive alerts among all alerts), or more intelligibly, as the WDR, which equals 1/PPV. WDR indicates the number of patients requiring evaluation to identify and treat one true positive case.^9-11 It is known that differences in event rates (prevalence or pretest probability) influence a model’s PPV²⁵ and its reciprocal WDR. However, for systematic comparison, PPV and WDR can be standardized using a fixed representative event rate across studies.^24,26 We abstracted the reported PPV and WDR, and computed standardized PPV and WDR for an event rate of 4%.

Other measures included the area under the receiver operator curve (AUC),²⁰ sensitivity, and specificity. AUC plots a model’s false positive rate (x-axis) against its true positive rate (y-axis), with an ideal scenario of very high y-values and very low x-values.²⁷ Sensitivity (the model’s ability to detect a true positive case among all cases) and specificity (the model’s ability to detect a true noncase among all noncases²⁸) are influenced by chosen alert thresholds. It is incorrect to assume that a given model produces only one sensitivity/specificity result; for systematic comparison, we therefore selected results in the 50% sensitivity range, and separately, in the 92% specificity range for EWSs using statistical modeling. Then, we simulated a fixed sensitivity of 0.51 and assumed specificity of 0.87 in aggregate-weighted EWSs.

Search Results

The PubMed search for “early warning score OR early warning system AND deterioration OR predict transfer ICU” returned 285 peer-reviewed articles. A search on CINAHL Plus using the same filters and query terms returned 219 articles with no additional matches (Figure 1). Of the 285 articles, we excluded 269 during the abstract screen and 10 additional articles during full-text review (Figure 1). A final review of the reference lists of the six selected studies did not yield additional articles.

Study Characteristics

There were several similarities across the selected studies (Table 1). All occurred in the United States; all compared their model’s performance against at least one aggregate-weighted EWS model;^14,17-19,29 and all used retrospective cohort designs. Of the six studies, one took place in a single hospital;²⁹ three pooled data from five hospitals;^17,18,30 and two occurred in a large integrated healthcare delivery system using data from 14 and, subsequently, 21 hospitals.^14,19 The largest study¹⁴ included nearly 650,000 admissions, while the smallest study²⁹ reported slightly less than 7,500 admissions. Of the six studies, four used multivariable regression,^14,17,19,29 and two used machine learning techniques for outcome prediction.^18,30

Outcome Variables

The primary outcome for inclusion in this review was clinical deterioration measured by the composite of transfer to ICU and some measure of mortality. Churpek et al.^10,11 and Green et al.³⁰ also included cardiac arrest, and Alvarez et al.²² included respiratory compromise in their outcome composite.

Researchers used varying definitions of mortality, including “death outside the ICU in a patient whose care directive was full code;”^14,19 “death on the wards without attempted resuscitation;”^17,18 “an in-hospital death in patients without a DNR order at admission that occurred on the medical ward or in ICU within 24 hours after transfer;”²⁹ or “death within 24 hours.”³⁰

Predictor Variables

We observed a broad assortment of predictor variables. All models included vital signs (heart rate, respiratory rate, blood pressure, and venous oxygen saturation); mental state; laboratory data; age; and sex. Additional variables included comorbidity, shock index,³¹ severity of illness score, length of stay, event time of day, season, admission category, and length of stay,^14,19 among others.

Model Performance

Reported PPV ranged from 0.16 to 0.42 (mean = 0.27) in EWSs using statistical modeling and 0.15 to 0.28 (mean = 0.19) in aggregate-weighted EWS models. The weighted mean standardized PPV, adjusted for an event rate of 4% across studies (Table 2), was 0.21 in EWSs using statistical modeling versus 0.14 in aggregate-weighted EWS models (simulated at 0.51 sensitivity and 0.87 specificity).

Only two studies^14,19 reported the WDR metric (alerts generated to identify one true positive case) explicitly. Based on the above PPV results, EWSs using statistical modeling generated a standardized WDR of 4.9 in models using statistical modeling versus 7.1 in aggregate-weighted models (Figure 2). The delta of 2.2 evaluations to find and treat one true positive case equals a 45% relative increase in RRT evaluation workloads using aggregate-weighted EWSs.

AUC values ranged from 0.77 to 0.85 (weighted mean = 0.80) in EWSs using statistical modeling, indicating good model discrimination. AUCs of aggregate-weighted EWSs ranged from 0.70 to 0.76 (weighted mean = 0.73), indicating fair model discrimination (Figure 2). The overall AUC delta was 0.07. However, our estimates may possibly be favoring EWSs that use statistical modeling by virtue of their derivation in an original research population compared with aggregate-weighted EWSs that were derived externally. For example, sensitivity analysis of eCART,¹⁸ an EWS using machine learning, showed an AUC drop of 1% in a large external patient population,¹⁴ while NEWS AUCs¹³ dropped between 11% and 15% in two large external populations (Appendix Table 7).^14,30 For hospitals adopting an externally developed EWS using statistical modeling, these results suggest that an AUC delta of approximately 5% can be expected and 7% for an internally developed EWS.

Risk of Bias Assessment

We scored the studies by adapting the Cochrane Collaboration tool for assessing risk of bias ³² (Appendix Table 5). Of the six studies, five received total scores between 1.0 and 2.0 (indicating relatively low bias risk), and one study had a score of 3.5 (indicating higher bias risk). Low bias studies^14,17-19,30 used large samples across multiple hospitals, discussed the choice of predictor variables and outcomes more precisely, and reported their measurement approaches and analytic methods in more detail, including imputation of missing data and model calibration.

In this systematic review, we assessed the predictive ability of EWSs using statistical modeling versus aggregate-weighted EWS models to detect clinical deterioration risk in hospitalized adults in general wards. From 2007 to 2018, at least five systematic reviews examined aggregate-weighted EWSs in adult inpatient settings.^33-37 No systematic review, however, has synthesized the evidence of EWSs using statistical modeling.

The recent evidence is limited to six studies, of which five had favorable risk of bias scores. All studies included in this review demonstrated superior model performance of the EWSs using statistical modeling compared with an aggregate-weighted EWS, and at least five of the six studies employed rigor in design, measurement, and analytic method. The AUC absolute difference between EWSs using statistical modeling and aggregate-weighted EWSs was 7% overall, moving model performance from fair to good (Table 2; Figure 2). Although this increase in discriminative power may appear modest, it translates into avoiding a 45% increase in WDR workload generated by an aggregate-weighted EWS, approximately two patient evaluations for each true positive case.

Results of our review suggest that EWSs using statistical modeling predict clinical deterioration risk with better precision. This is an important finding for the following reasons: (1) Better risk prediction can support the activation of rescue; (2) Given federal mandates to curb spending, the elimination of some resource-intensive false positive evaluations supports high-value care;³⁸ and (3) The Quadruple Aim³⁹ accounts for clinician wellbeing. EWSs using statistical modeling may offer benefits in terms of clinician satisfaction with the human–system interface because better discrimination reduces the daily evaluation workload/cognitive burden and because the reduction of false positive alerts may reduce alert fatigue.^40,41

Still, an important issue with risk detection is that it is unknown which percentage of patients are uniquely identified by an EWS and not already under evaluation by the clinical team. For example, a recent study by Bedoya et al.⁴² found that using NEWS did not improve clinical outcomes and nurses frequently disregarded the alert. Another study⁴³ found that the combined clinical judgment of physicians and nurses had an AUC of 0.90 in predicting mortality. These results suggest that at certain times, an EWS alert may not add new useful information for clinicians even when it correctly identifies deterioration risk. It remains difficult to define exactly how many patients an EWS would have to uniquely identify to have clinical utility.

Even EWSs that use statistical modeling cannot detect all true deterioration cases perfectly, and they may at times trigger an alert only when the clinical team is already aware of a patient’s clinical decline. Consequently, EWSs using statistical modeling can at best augment and support—but not replace—RRT rounding, physician workup, and vigilant frontline staff. However, clinicians, too, are not perfect, and the failure-to-rescue literature suggests that certain human factors are antecedents to patient crises (eg, stress and distraction,^44-46 judging by precedent/experience,^44,47 and innate limitations of human cognition⁴⁷). Because neither clinicians nor EWSs can predict deterioration perfectly, the best possible rescue response combines clinical vigilance, RRT rounding, and EWSs using statistical modeling as complementary solutions.

Our findings suggest that predictive models cannot be judged purely on AUC (in fact, it would be ill-advised) but also by their clinical utility (expressed in WDR and PPV): How many patients does a clinician need to evaluate?^9-11 Precision is not meaningful if it comes at the expense of unmanageable evaluation workloads, and our findings suggest that clinicians should evaluate models based on their clinical utility. Hospitals considering adoption of an EWS using statistical modeling should consider that externally developed EWSs appear to experience a performance drop when applied to a new patient population; a slightly higher WDR and slightly lower AUC can be expected. EWSs using statistical modeling appear to perform best when tailored to the targeted patient population (or are derived in-house). Model depreciation over time will likely require recalibration. In addition, adoption of a machine learning algorithm may mean that original model results are obscured by the black box output of the algorithm.^48-50

Findings from this systematic review are subject to several limitations. First, we applied strict inclusion criteria, which led us to exclude studies that offered findings in specialty units and specific patient subpopulations, among others. In the interest of systematic comparison, our findings are limited to general wards. We also restricted our search to recent studies that reported on models predicting clinical deterioration, which we defined as the composite of ICU transfer and/or death. Clinically, deteriorating patients in general wards either die or are transferred to ICU. This criterion resulted in exclusion of the Rothman Index,⁵¹ which predicts “death within 24 hours” but not ICU transfer. The AUC in this study was higher than those selected in this review (0.93 compared to 0.82 for MEWS; AUC delta: 0.09). The higher AUC may be a function of the outcome definition (30-day mortality would be more challenging to predict). Therefore, hospitals or health systems interested in purchasing an EWS using statistical modeling should carefully consider the outcome selection and definition.

Second, as is true for systematic reviews in general,⁵² the degree of clinical and methodological heterogeneity across the selected studies may limit our findings. Studies occurred in various settings (university hospital, teaching hospitals, and community hospitals), which may serve diverging patient populations. We observed that studies in university-based settings had a higher event rate ranging from 5.6% to 7.8%, which may result in higher PPV results in these settings. However, this increase would apply to both EWS types equally. To arrive at a “true” reflection of model performance, the simulations for PPV and WDR have used a more conservative event rate of 4%. We observed heterogenous mortality definitions, which did not always account for the reality that a patient’s death may be an appropriate outcome (ie, it was concordant with treatment wishes in the context of severe illness or an end-of-life trajectory). Studies also used different sampling procedures; some allowed multiple observations although most did not. The variation in sampling may change PPV and limit our systematic comparison. However, regardless of methodological differences, our review suggests that EWSs using statistical modeling perform better than aggregate-weighted EWSs in each of the selected studies.

Third, systematic reviews may be subject to the issue of publication bias because they can only compare published results and could possibly omit an unknown number of unpublished studies. However, the selected studies uniformly demonstrated similar model improvements, which are plausibly related to the larger number of covariates, statistical methods, and shrinkage of random error.

Finally, this review was limited to the comparison of observational studies, which aimed to answer how the two EWS classes compared. These studies did not address whether an alert had an impact on clinical care and patient outcomes. Results from at least one randomized nonblinded controlled trial suggest that alert-driven RRT activation may reduce the length of stay by 24 hours and use of oximetry, but has no impact on mortality, ICU transfer, and ICU length of stay.⁵³

Our findings point to three areas of need for the field of predictive EWS research: (1) a standardized set of clinical deterioration outcome measures, (2) a standardized set of measures capturing clinical evaluation workload and alert frequency, and (3) cost estimates of clinical workloads with and without deployment of an EWS using statistical modeling. Given the present divergence of outcome definitions, EWS research may benefit from a common “clinical deterioration” outcome standard, including transfer to ICU, inpatient/30-day/90-day mortality, and death with DNR, comfort care, or hospice. The field is lacking a standardized clinical workload measure and an understanding of the net percentage of patients uniquely identified by an EWS.

By using predictive analytics, health systems may be better able to achieve the goals of high-value care and patient safety and support the Quadruple Aim. Still, gaps in knowledge exist regarding the measurement of the clinical processes triggered by EWSs, evaluation workloads, alert fatigue, clinician burnout associated with the human-alert interface, and costs versus benefits. Future research should evaluate the degree to which EWSs can identify risk among patients who are not already under evaluation by the clinical team, assess the balanced treatment effects of RRT interventions between decedents and survivors, and investigate clinical process times relative to the time of an EWS alert using statistical modeling.

Acknowledgments

The authors would like to thank Ms. Jill Pope at the Kaiser Permanente Center for Health Research in Portland, OR for her assistance with manuscript preparation. Daniel Linnen would like to thank Dr. Linda Franck, PhD, RN, FAAN, Professor at the University of California, San Francisco, School of Nursing for reviewing the manuscript.

Disclosures

The authors declare no conflicts of interest.

Funding

The Maribelle & Stephen Leavitt Scholarship, the Jonas Nurse Scholars Scholarship at the University of California, San Francisco, and the Nurse Scholars Academy Predoctoral Research Fellowship at Kaiser Permanente Northern California supported this study during Daniel Linnen’s doctoral training at the University of California, San Francisco. Dr. Vincent Liu was funded by National Institute of General Medical Sciences Grant K23GM112018.

Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.¹ The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).² Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.

Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.² However, the limitation of this and other studies evaluating the sensitivity of early warning systems^3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.

To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)⁷ and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.

Patients and Setting

This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.

Definition of Critical Deterioration Events

Critical deterioration events (CDEs) were defined according to an existing, validated measure⁷ as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.

The Pediatric Rothman Index

The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.² The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.^2,8 (See Supplement 1 for a complete list of variables.)

The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.

Development of the Source Dataset

In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.

Data Collection

Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.⁹

To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.

To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.

We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team, and the determination was made by consensus.

To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.

During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).

Patient Characteristics

The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.

Acuity Alerts

A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.

Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).

We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.

Escalation Orders

To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.

Provider Notes

Temporal Relationships

In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.

The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.

The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.

The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.¹⁰ In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.¹¹

Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).¹² Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.

Prior work has described a range of mechanisms by which early warning scores can impact patient safety.¹³ The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.

We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.

This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.

Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.

Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.

In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.

Disclosures

The authors have nothing to disclose

Funding

The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.

Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.¹ The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).² Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.

Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.² However, the limitation of this and other studies evaluating the sensitivity of early warning systems^3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.

To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)⁷ and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.

Patients and Setting

This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.

Definition of Critical Deterioration Events

Critical deterioration events (CDEs) were defined according to an existing, validated measure⁷ as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.

The Pediatric Rothman Index

The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.² The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.^2,8 (See Supplement 1 for a complete list of variables.)

The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.

Development of the Source Dataset

In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.

Data Collection

Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.⁹

To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.

To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.

We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team, and the determination was made by consensus.

To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.

During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).

Patient Characteristics

The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.

Acuity Alerts

A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.

Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).

We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.

Escalation Orders

To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.

Provider Notes

Temporal Relationships

In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.

The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.

The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.

The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.¹⁰ In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.¹¹

Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).¹² Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.

Prior work has described a range of mechanisms by which early warning scores can impact patient safety.¹³ The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.

We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.

This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.

Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.

Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.

In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.

Disclosures

The authors have nothing to disclose

Funding

The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.

Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.¹ The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).² Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.

Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.² However, the limitation of this and other studies evaluating the sensitivity of early warning systems^3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.

To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)⁷ and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.

Patients and Setting

This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.

Definition of Critical Deterioration Events

Critical deterioration events (CDEs) were defined according to an existing, validated measure⁷ as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.

The Pediatric Rothman Index

The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.² The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.^2,8 (See Supplement 1 for a complete list of variables.)

The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.

Development of the Source Dataset

In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.

Data Collection

Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.⁹

To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.

To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.

We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team, and the determination was made by consensus.

To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.

During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).

Patient Characteristics

The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.

Acuity Alerts

A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.

Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).

We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.

Escalation Orders

To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.

Provider Notes

Temporal Relationships

In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.

The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.

The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.

The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.¹⁰ In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.¹¹

Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).¹² Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.

Prior work has described a range of mechanisms by which early warning scores can impact patient safety.¹³ The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.

We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.

This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.

Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.

Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.

In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.

Disclosures

The authors have nothing to disclose

Funding

The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.

With the presence of hypertension in 25% of patients admitted to the hospital,¹ its proper management is imperative. A hypertensive crisis is a severe elevation of blood pressure, defined as systolic ≥180 mm Hg and/or diastolic ≥120 mm Hg. It is further classified as either a hypertensive emergency which includes the presence of end-organ damage,² or hypertensive urgency, defined as asymptomatic blood pressure elevation.³ Although hypertensive emergencies account for only 1%-2% of patients with hypertension,⁴ they are associated with a high one-year mortality rate (>79%).⁵ Hypertensive emergency requires immediate reduction of blood pressure with IV antihypertensive drugs to limit organ damage. In contrast, as per national guidelines, inpatient management of hypertensive urgency requires gradual reductions of blood pressure over hours to days using oral antihypertensives.² It is also recommended that alternative etiologies, such as anxiety or pain, be considered before treatment is initiated.¹

Clinicians often inappropriately treat asymptomatic hypertension in the inpatient setting,^6,7 using intravenous (IV) antihypertensive medications despite evidence showing potential harm.^5,8 This can lead to unpredictable reductions in blood pressure.^7,9 A recent retrospective analysis demonstrated that 32.6% of patients had a blood pressure reduction greater than 25% after the use of an IV antihypertensive.⁷ Reductions greater than 25% lead to shifts in autoregulation, which may result in patient harm, such as hypotension, decreased renal perfusion, and stroke.⁹ IV medications are also more expensive than oral agents, due to the additional cost of administration.

Although overtreatment of asymptomatic hypertension with IV antihypertensive medications is common,⁷ initiatives to address this in inpatient settings are lacking in the literature. The aim of this quality improvement initiative was to reduce unnecessary IV antihypertensive treatment for hypertensive urgency in the inpatient setting.

Setting

An interdisciplinary quality improvement intervention was initiated on two inpatient medicine units at an urban, 1,134-bed tertiary medical center affiliated with the Icahn School of Medicine at Mount Sinai. Members of the Mount Sinai High Value Care Committee and the Student High Value Care Initiative¹⁰ developed this project. The intervention was implemented in stages from March 2017 to February 2018. It targeted nurses, housestaff, nurse practitioners, and attendings on general medical teaching and nonteaching services. The components of the intervention included education, a treatment algorithm, audit and feedback, and electronic medical record (EMR) change. This project was submitted to the Quality Committee in the Department of Medicine and determined to be a quality improvement project rather than research and thus, an IRB submission was not required.

Treatment Algorithm and Education

A clinical algorithm was designed with nursing and cardiology representatives to provide guidance for nurses regarding the best practice for evaluation of inpatient hypertension, focusing on assessing patients before recommending treatment (“Assess Before Rx”; Figure 1). Educational sessions reinforcing the clinical algorithm were held monthly at nursing huddles. These involved an introduction session providing the background and purpose of the project, with follow-up sessions including interactive mock cases on the assessment of hypertensive urgency.

A second treatment algorithm was designed, with housestaff and cardiology input, to provide guidance for the internal medicine housestaff and nurse practitioners. It utilized a similar approach regarding identification, evaluation, and assessment of alternate etiologies but included more detailed treatment recommendations with a table outlining the oral medications used for hypertensive urgency (Figure 2). The flowchart and table were uploaded to an existing mobile application used by housestaff and nurse practitioners for quick access. The mobile application is frequently used by housestaff and contains many clinical resources. Additionally, e-mails including the purpose of the project and the treatment algorithm were sent to rotating housestaff at the start of each new medicine rotation.

Audit and Feedback

Monthly feedback was e-mailed to the nurses, which reinforced the goals and provided positive feedback on outcomes with an announcement of the “Nurse of the Month.” The winners were selected based on the most accurate and appropriate documentation of their assessments determined through retrospective chart review.

Targeted e-mail feedback was also sent to providers who ordered IV antihypertensives without the appropriate indication. The e-mails included the medical record number, date and time of the order, any alternate etiologies that were documented, and any adverse events that occurred as a result of the medication.

Systems Change: Electronic Medical Record Orders

EMR advisory warnings were placed on IV antihypertensive orders of labetalol and hydralazine. The alerts served to nonintrusively remind providers to assess for symptoms before placing the order to ensure that the order was appropriate.

Data Collection and Assessment

Seven-month preintervention (January-July 2016) and 12-month postintervention (March 2017-February 2018) data were compared. The months prior to intervention were excluded to account for project development and educational lag. Data were obtained from EMR utilization reports of one-time orders of IV labetalol and hydralazine, and retrospective chart review. Patients who were pregnant, less than 18 years of age, or postoperative were excluded. Orders were designated as inappropriate if there was no evidence of hypertensive emergency through documentation in progress notes, or if the patient was able to take oral medication (not NPO). Adverse events were defined as a blood pressure drop of more than 25%, a change in the heart rate by more than 20 beats per minute, or the need for IV fluids, based on previous studies.⁷ Although decreased blood pressure is not necessarily dangerous in and of itself, adverse events arising from blood pressure decreasing too rapidly from IV antihypertensives are well documented.^9,11 The presence of alternate etiologies of high blood pressure that were documented in progress notes, including pain, anxiety, agitation, and holding of home blood pressure medications, were recorded. The numbers of inappropriate orders pre- and postintervention were compared. Confounding factors of patient age and length of stay (LOS) were compared pre- and postintervention in order to rule out other factors to which the intervention’s effect could be attributed. Additionally, as a balancing measure, a random sample of patients with elevated blood pressure were monitored on a biweekly basis for adverse events that occurred as a result of not receiving IV treatment, including stroke, myocardial infarction, and pulmonary edema.

For this study, orders were reported on the standardized form of orders per 1,000 patient days. This was calculated as the number of orders divided by the total number of patient days from the two medicine units. For the univariate analysis, pre- and postintervention orders were compared for the different order categories using a t-test. Results were considered statistically significant at P < .05. Data analysis was conducted using SAS v. 9.4 (SAS Institute, Cary, North Carolina).

Additionally, a cost analysis was performed to estimate the hospital-wide annual cost of inappropriate orders. The analysis used the cost per dose¹² and included nurse-time derived from the median salary of those on our units. The hospital-wide cost was extrapolated to estimate the potential annual savings for the institution.

A total of 260 one-time orders of IV antihypertensives were analyzed in this study, 127 in the seven-month preintervention period and 133 in the 12-month postintervention period. The majority, 67.3% (n = 175), were labetalol orders. Inappropriate orders (ie, neither NPO nor hypertensive emergency) decreased from 8.3 to 3.3 orders per 1,000 patient days (P = .0099; Figure 3).

In total, there were 86 adverse events (33.1%), the majority of which (94.2%, n = 81) were a >25% decrease in blood pressure (Table 1). The number of adverse events per 1,000 patient days decreased from 4.4 in the preintervention period to 1.9 postintervention, P = .0112. Of the inappropriate orders, adverse events decreased from 3.7 to 0.8 per 1,000 patient days, P = .0072. Overall, there were 76 orders (29.2%) with documented alternate etiologies. The number of orders per 1,000 patient days with an alternate etiology decreased from 4.7 in the preintervention period to 1.2 postintervention, P =.0044 (Table 2). Descriptive analysis of patient characteristics pre- and postintervention were not statistically significant; for age 68.4 vs 70.7, P = .0823 and for LOS 14.8 vs 15.4, P = .0769. As a balancing measure, 111 patients with elevated blood pressure were monitored for adverse events during the postintervention period. Among patients who did not receive IV medication based on our algorithm, there were no adverse events.

Our initiative successfully demonstrated a significant reduction of 60% in inappropriate one-time orders of IV antihypertensives per 1,000 patient days. Accordingly, the number of adverse events per 1,000 patient days decreased by 57%. There was also a decrease in the number and percentage of IV orders with documented alternate etiologies. We hypothesize that this was due to nurses and physicians assessing and treating these conditions prior to treating hypertension in the intervention period, consequently avoiding an IV order.

The goal of the intervention was to have nurses assess for end-organ damage and alternate etiologies and include this information on their assessment provided to the physician, which would result in appropriate treatment of elevated blood pressure. By performing an interdisciplinary intervention, we addressed the knowledge deficit of both nurses and physicians, improved the triage of elevated blood pressure, and likely decreased the number of pages to providers.

To our knowledge, this is the first intervention addressing the inpatient overuse of IV antihypertensive medications for the treatment of asymptomatic hypertension. Additionally, this study bolsters prior evidence that the use of IV antihypertensives in asymptomatic patients leads to a large number of adverse events.⁷ A third of patients in the preintervention period had documented alternate etiologies of their blood pressure elevation, highlighting the need to assess and potentially treat these causes prior to treating blood pressure itself.

Reducing unnecessary treatment of asymptomatic blood pressure elevation is challenging. Evidence shows that both clinicians and patients overestimate the benefits and underestimate the harms of medical interventions.^13,14 This unfortunately leads to unjustified enthusiasm for medical treatments, which can worsen outcomes.¹⁵ Additionally, there may be a lack of knowledge of the guidelines, as well as the amount of time required in the full assessment of hypertensive urgency, that creates a culture of “treating the number.”

Changing physician behavior is difficult.¹⁶ However, active forms of continuing education and multifaceted interventions, such as ours, are most effective.¹⁷ Our message focused on patient safety and harm reduction, addressed clinicians’ safety concerns, and included stories of real cases where this overuse led to adverse events—all of which are encouraged in order to facilitate clinician engagement.¹⁸

There were limitations to this study. Only blood pressure elevations associated with an IV antihypertensive order and not all blood pressure elevations meeting the criteria for hypertensive urgency in general were examined. Additionally, our documentation of symptoms of hypertensive emergency and alternate etiologies was based only on documentation in the medical record. Ideally, we would have liked to conduct an interrupted time series analysis to assess the effect of the intervention over time; however, there were not enough orders of IV antihypertensives to perform such an analysis.

Treatment of asymptomatic blood pressure with IV antihypertensive medications can lead to patient harm. To reduce inappropriate treatment, our Student High Value Care team set out to challenge this common practice. Our interdisciplinary intervention successfully reduced unnecessary IV antihypertensive treatment. This may serve as a model for other institutions.

Disclosures

There are no relevant conflicts of interest to disclose for any authors.

With the presence of hypertension in 25% of patients admitted to the hospital,¹ its proper management is imperative. A hypertensive crisis is a severe elevation of blood pressure, defined as systolic ≥180 mm Hg and/or diastolic ≥120 mm Hg. It is further classified as either a hypertensive emergency which includes the presence of end-organ damage,² or hypertensive urgency, defined as asymptomatic blood pressure elevation.³ Although hypertensive emergencies account for only 1%-2% of patients with hypertension,⁴ they are associated with a high one-year mortality rate (>79%).⁵ Hypertensive emergency requires immediate reduction of blood pressure with IV antihypertensive drugs to limit organ damage. In contrast, as per national guidelines, inpatient management of hypertensive urgency requires gradual reductions of blood pressure over hours to days using oral antihypertensives.² It is also recommended that alternative etiologies, such as anxiety or pain, be considered before treatment is initiated.¹

Clinicians often inappropriately treat asymptomatic hypertension in the inpatient setting,^6,7 using intravenous (IV) antihypertensive medications despite evidence showing potential harm.^5,8 This can lead to unpredictable reductions in blood pressure.^7,9 A recent retrospective analysis demonstrated that 32.6% of patients had a blood pressure reduction greater than 25% after the use of an IV antihypertensive.⁷ Reductions greater than 25% lead to shifts in autoregulation, which may result in patient harm, such as hypotension, decreased renal perfusion, and stroke.⁹ IV medications are also more expensive than oral agents, due to the additional cost of administration.

Although overtreatment of asymptomatic hypertension with IV antihypertensive medications is common,⁷ initiatives to address this in inpatient settings are lacking in the literature. The aim of this quality improvement initiative was to reduce unnecessary IV antihypertensive treatment for hypertensive urgency in the inpatient setting.

Setting

An interdisciplinary quality improvement intervention was initiated on two inpatient medicine units at an urban, 1,134-bed tertiary medical center affiliated with the Icahn School of Medicine at Mount Sinai. Members of the Mount Sinai High Value Care Committee and the Student High Value Care Initiative¹⁰ developed this project. The intervention was implemented in stages from March 2017 to February 2018. It targeted nurses, housestaff, nurse practitioners, and attendings on general medical teaching and nonteaching services. The components of the intervention included education, a treatment algorithm, audit and feedback, and electronic medical record (EMR) change. This project was submitted to the Quality Committee in the Department of Medicine and determined to be a quality improvement project rather than research and thus, an IRB submission was not required.

Treatment Algorithm and Education

A clinical algorithm was designed with nursing and cardiology representatives to provide guidance for nurses regarding the best practice for evaluation of inpatient hypertension, focusing on assessing patients before recommending treatment (“Assess Before Rx”; Figure 1). Educational sessions reinforcing the clinical algorithm were held monthly at nursing huddles. These involved an introduction session providing the background and purpose of the project, with follow-up sessions including interactive mock cases on the assessment of hypertensive urgency.

A second treatment algorithm was designed, with housestaff and cardiology input, to provide guidance for the internal medicine housestaff and nurse practitioners. It utilized a similar approach regarding identification, evaluation, and assessment of alternate etiologies but included more detailed treatment recommendations with a table outlining the oral medications used for hypertensive urgency (Figure 2). The flowchart and table were uploaded to an existing mobile application used by housestaff and nurse practitioners for quick access. The mobile application is frequently used by housestaff and contains many clinical resources. Additionally, e-mails including the purpose of the project and the treatment algorithm were sent to rotating housestaff at the start of each new medicine rotation.

Audit and Feedback

Monthly feedback was e-mailed to the nurses, which reinforced the goals and provided positive feedback on outcomes with an announcement of the “Nurse of the Month.” The winners were selected based on the most accurate and appropriate documentation of their assessments determined through retrospective chart review.

Targeted e-mail feedback was also sent to providers who ordered IV antihypertensives without the appropriate indication. The e-mails included the medical record number, date and time of the order, any alternate etiologies that were documented, and any adverse events that occurred as a result of the medication.

Systems Change: Electronic Medical Record Orders

EMR advisory warnings were placed on IV antihypertensive orders of labetalol and hydralazine. The alerts served to nonintrusively remind providers to assess for symptoms before placing the order to ensure that the order was appropriate.

Data Collection and Assessment

Seven-month preintervention (January-July 2016) and 12-month postintervention (March 2017-February 2018) data were compared. The months prior to intervention were excluded to account for project development and educational lag. Data were obtained from EMR utilization reports of one-time orders of IV labetalol and hydralazine, and retrospective chart review. Patients who were pregnant, less than 18 years of age, or postoperative were excluded. Orders were designated as inappropriate if there was no evidence of hypertensive emergency through documentation in progress notes, or if the patient was able to take oral medication (not NPO). Adverse events were defined as a blood pressure drop of more than 25%, a change in the heart rate by more than 20 beats per minute, or the need for IV fluids, based on previous studies.⁷ Although decreased blood pressure is not necessarily dangerous in and of itself, adverse events arising from blood pressure decreasing too rapidly from IV antihypertensives are well documented.^9,11 The presence of alternate etiologies of high blood pressure that were documented in progress notes, including pain, anxiety, agitation, and holding of home blood pressure medications, were recorded. The numbers of inappropriate orders pre- and postintervention were compared. Confounding factors of patient age and length of stay (LOS) were compared pre- and postintervention in order to rule out other factors to which the intervention’s effect could be attributed. Additionally, as a balancing measure, a random sample of patients with elevated blood pressure were monitored on a biweekly basis for adverse events that occurred as a result of not receiving IV treatment, including stroke, myocardial infarction, and pulmonary edema.

For this study, orders were reported on the standardized form of orders per 1,000 patient days. This was calculated as the number of orders divided by the total number of patient days from the two medicine units. For the univariate analysis, pre- and postintervention orders were compared for the different order categories using a t-test. Results were considered statistically significant at P < .05. Data analysis was conducted using SAS v. 9.4 (SAS Institute, Cary, North Carolina).

Additionally, a cost analysis was performed to estimate the hospital-wide annual cost of inappropriate orders. The analysis used the cost per dose¹² and included nurse-time derived from the median salary of those on our units. The hospital-wide cost was extrapolated to estimate the potential annual savings for the institution.

A total of 260 one-time orders of IV antihypertensives were analyzed in this study, 127 in the seven-month preintervention period and 133 in the 12-month postintervention period. The majority, 67.3% (n = 175), were labetalol orders. Inappropriate orders (ie, neither NPO nor hypertensive emergency) decreased from 8.3 to 3.3 orders per 1,000 patient days (P = .0099; Figure 3).

In total, there were 86 adverse events (33.1%), the majority of which (94.2%, n = 81) were a >25% decrease in blood pressure (Table 1). The number of adverse events per 1,000 patient days decreased from 4.4 in the preintervention period to 1.9 postintervention, P = .0112. Of the inappropriate orders, adverse events decreased from 3.7 to 0.8 per 1,000 patient days, P = .0072. Overall, there were 76 orders (29.2%) with documented alternate etiologies. The number of orders per 1,000 patient days with an alternate etiology decreased from 4.7 in the preintervention period to 1.2 postintervention, P =.0044 (Table 2). Descriptive analysis of patient characteristics pre- and postintervention were not statistically significant; for age 68.4 vs 70.7, P = .0823 and for LOS 14.8 vs 15.4, P = .0769. As a balancing measure, 111 patients with elevated blood pressure were monitored for adverse events during the postintervention period. Among patients who did not receive IV medication based on our algorithm, there were no adverse events.

Our initiative successfully demonstrated a significant reduction of 60% in inappropriate one-time orders of IV antihypertensives per 1,000 patient days. Accordingly, the number of adverse events per 1,000 patient days decreased by 57%. There was also a decrease in the number and percentage of IV orders with documented alternate etiologies. We hypothesize that this was due to nurses and physicians assessing and treating these conditions prior to treating hypertension in the intervention period, consequently avoiding an IV order.

The goal of the intervention was to have nurses assess for end-organ damage and alternate etiologies and include this information on their assessment provided to the physician, which would result in appropriate treatment of elevated blood pressure. By performing an interdisciplinary intervention, we addressed the knowledge deficit of both nurses and physicians, improved the triage of elevated blood pressure, and likely decreased the number of pages to providers.

To our knowledge, this is the first intervention addressing the inpatient overuse of IV antihypertensive medications for the treatment of asymptomatic hypertension. Additionally, this study bolsters prior evidence that the use of IV antihypertensives in asymptomatic patients leads to a large number of adverse events.⁷ A third of patients in the preintervention period had documented alternate etiologies of their blood pressure elevation, highlighting the need to assess and potentially treat these causes prior to treating blood pressure itself.

Reducing unnecessary treatment of asymptomatic blood pressure elevation is challenging. Evidence shows that both clinicians and patients overestimate the benefits and underestimate the harms of medical interventions.^13,14 This unfortunately leads to unjustified enthusiasm for medical treatments, which can worsen outcomes.¹⁵ Additionally, there may be a lack of knowledge of the guidelines, as well as the amount of time required in the full assessment of hypertensive urgency, that creates a culture of “treating the number.”

Changing physician behavior is difficult.¹⁶ However, active forms of continuing education and multifaceted interventions, such as ours, are most effective.¹⁷ Our message focused on patient safety and harm reduction, addressed clinicians’ safety concerns, and included stories of real cases where this overuse led to adverse events—all of which are encouraged in order to facilitate clinician engagement.¹⁸

There were limitations to this study. Only blood pressure elevations associated with an IV antihypertensive order and not all blood pressure elevations meeting the criteria for hypertensive urgency in general were examined. Additionally, our documentation of symptoms of hypertensive emergency and alternate etiologies was based only on documentation in the medical record. Ideally, we would have liked to conduct an interrupted time series analysis to assess the effect of the intervention over time; however, there were not enough orders of IV antihypertensives to perform such an analysis.

Treatment of asymptomatic blood pressure with IV antihypertensive medications can lead to patient harm. To reduce inappropriate treatment, our Student High Value Care team set out to challenge this common practice. Our interdisciplinary intervention successfully reduced unnecessary IV antihypertensive treatment. This may serve as a model for other institutions.

Disclosures

There are no relevant conflicts of interest to disclose for any authors.

With the presence of hypertension in 25% of patients admitted to the hospital,¹ its proper management is imperative. A hypertensive crisis is a severe elevation of blood pressure, defined as systolic ≥180 mm Hg and/or diastolic ≥120 mm Hg. It is further classified as either a hypertensive emergency which includes the presence of end-organ damage,² or hypertensive urgency, defined as asymptomatic blood pressure elevation.³ Although hypertensive emergencies account for only 1%-2% of patients with hypertension,⁴ they are associated with a high one-year mortality rate (>79%).⁵ Hypertensive emergency requires immediate reduction of blood pressure with IV antihypertensive drugs to limit organ damage. In contrast, as per national guidelines, inpatient management of hypertensive urgency requires gradual reductions of blood pressure over hours to days using oral antihypertensives.² It is also recommended that alternative etiologies, such as anxiety or pain, be considered before treatment is initiated.¹

Clinicians often inappropriately treat asymptomatic hypertension in the inpatient setting,^6,7 using intravenous (IV) antihypertensive medications despite evidence showing potential harm.^5,8 This can lead to unpredictable reductions in blood pressure.^7,9 A recent retrospective analysis demonstrated that 32.6% of patients had a blood pressure reduction greater than 25% after the use of an IV antihypertensive.⁷ Reductions greater than 25% lead to shifts in autoregulation, which may result in patient harm, such as hypotension, decreased renal perfusion, and stroke.⁹ IV medications are also more expensive than oral agents, due to the additional cost of administration.

Although overtreatment of asymptomatic hypertension with IV antihypertensive medications is common,⁷ initiatives to address this in inpatient settings are lacking in the literature. The aim of this quality improvement initiative was to reduce unnecessary IV antihypertensive treatment for hypertensive urgency in the inpatient setting.

Setting

An interdisciplinary quality improvement intervention was initiated on two inpatient medicine units at an urban, 1,134-bed tertiary medical center affiliated with the Icahn School of Medicine at Mount Sinai. Members of the Mount Sinai High Value Care Committee and the Student High Value Care Initiative¹⁰ developed this project. The intervention was implemented in stages from March 2017 to February 2018. It targeted nurses, housestaff, nurse practitioners, and attendings on general medical teaching and nonteaching services. The components of the intervention included education, a treatment algorithm, audit and feedback, and electronic medical record (EMR) change. This project was submitted to the Quality Committee in the Department of Medicine and determined to be a quality improvement project rather than research and thus, an IRB submission was not required.

Treatment Algorithm and Education

A clinical algorithm was designed with nursing and cardiology representatives to provide guidance for nurses regarding the best practice for evaluation of inpatient hypertension, focusing on assessing patients before recommending treatment (“Assess Before Rx”; Figure 1). Educational sessions reinforcing the clinical algorithm were held monthly at nursing huddles. These involved an introduction session providing the background and purpose of the project, with follow-up sessions including interactive mock cases on the assessment of hypertensive urgency.

A second treatment algorithm was designed, with housestaff and cardiology input, to provide guidance for the internal medicine housestaff and nurse practitioners. It utilized a similar approach regarding identification, evaluation, and assessment of alternate etiologies but included more detailed treatment recommendations with a table outlining the oral medications used for hypertensive urgency (Figure 2). The flowchart and table were uploaded to an existing mobile application used by housestaff and nurse practitioners for quick access. The mobile application is frequently used by housestaff and contains many clinical resources. Additionally, e-mails including the purpose of the project and the treatment algorithm were sent to rotating housestaff at the start of each new medicine rotation.

Audit and Feedback

Monthly feedback was e-mailed to the nurses, which reinforced the goals and provided positive feedback on outcomes with an announcement of the “Nurse of the Month.” The winners were selected based on the most accurate and appropriate documentation of their assessments determined through retrospective chart review.

Targeted e-mail feedback was also sent to providers who ordered IV antihypertensives without the appropriate indication. The e-mails included the medical record number, date and time of the order, any alternate etiologies that were documented, and any adverse events that occurred as a result of the medication.

Systems Change: Electronic Medical Record Orders

EMR advisory warnings were placed on IV antihypertensive orders of labetalol and hydralazine. The alerts served to nonintrusively remind providers to assess for symptoms before placing the order to ensure that the order was appropriate.

Data Collection and Assessment

Seven-month preintervention (January-July 2016) and 12-month postintervention (March 2017-February 2018) data were compared. The months prior to intervention were excluded to account for project development and educational lag. Data were obtained from EMR utilization reports of one-time orders of IV labetalol and hydralazine, and retrospective chart review. Patients who were pregnant, less than 18 years of age, or postoperative were excluded. Orders were designated as inappropriate if there was no evidence of hypertensive emergency through documentation in progress notes, or if the patient was able to take oral medication (not NPO). Adverse events were defined as a blood pressure drop of more than 25%, a change in the heart rate by more than 20 beats per minute, or the need for IV fluids, based on previous studies.⁷ Although decreased blood pressure is not necessarily dangerous in and of itself, adverse events arising from blood pressure decreasing too rapidly from IV antihypertensives are well documented.^9,11 The presence of alternate etiologies of high blood pressure that were documented in progress notes, including pain, anxiety, agitation, and holding of home blood pressure medications, were recorded. The numbers of inappropriate orders pre- and postintervention were compared. Confounding factors of patient age and length of stay (LOS) were compared pre- and postintervention in order to rule out other factors to which the intervention’s effect could be attributed. Additionally, as a balancing measure, a random sample of patients with elevated blood pressure were monitored on a biweekly basis for adverse events that occurred as a result of not receiving IV treatment, including stroke, myocardial infarction, and pulmonary edema.

For this study, orders were reported on the standardized form of orders per 1,000 patient days. This was calculated as the number of orders divided by the total number of patient days from the two medicine units. For the univariate analysis, pre- and postintervention orders were compared for the different order categories using a t-test. Results were considered statistically significant at P < .05. Data analysis was conducted using SAS v. 9.4 (SAS Institute, Cary, North Carolina).

Additionally, a cost analysis was performed to estimate the hospital-wide annual cost of inappropriate orders. The analysis used the cost per dose¹² and included nurse-time derived from the median salary of those on our units. The hospital-wide cost was extrapolated to estimate the potential annual savings for the institution.

A total of 260 one-time orders of IV antihypertensives were analyzed in this study, 127 in the seven-month preintervention period and 133 in the 12-month postintervention period. The majority, 67.3% (n = 175), were labetalol orders. Inappropriate orders (ie, neither NPO nor hypertensive emergency) decreased from 8.3 to 3.3 orders per 1,000 patient days (P = .0099; Figure 3).

In total, there were 86 adverse events (33.1%), the majority of which (94.2%, n = 81) were a >25% decrease in blood pressure (Table 1). The number of adverse events per 1,000 patient days decreased from 4.4 in the preintervention period to 1.9 postintervention, P = .0112. Of the inappropriate orders, adverse events decreased from 3.7 to 0.8 per 1,000 patient days, P = .0072. Overall, there were 76 orders (29.2%) with documented alternate etiologies. The number of orders per 1,000 patient days with an alternate etiology decreased from 4.7 in the preintervention period to 1.2 postintervention, P =.0044 (Table 2). Descriptive analysis of patient characteristics pre- and postintervention were not statistically significant; for age 68.4 vs 70.7, P = .0823 and for LOS 14.8 vs 15.4, P = .0769. As a balancing measure, 111 patients with elevated blood pressure were monitored for adverse events during the postintervention period. Among patients who did not receive IV medication based on our algorithm, there were no adverse events.

Our initiative successfully demonstrated a significant reduction of 60% in inappropriate one-time orders of IV antihypertensives per 1,000 patient days. Accordingly, the number of adverse events per 1,000 patient days decreased by 57%. There was also a decrease in the number and percentage of IV orders with documented alternate etiologies. We hypothesize that this was due to nurses and physicians assessing and treating these conditions prior to treating hypertension in the intervention period, consequently avoiding an IV order.

The goal of the intervention was to have nurses assess for end-organ damage and alternate etiologies and include this information on their assessment provided to the physician, which would result in appropriate treatment of elevated blood pressure. By performing an interdisciplinary intervention, we addressed the knowledge deficit of both nurses and physicians, improved the triage of elevated blood pressure, and likely decreased the number of pages to providers.

To our knowledge, this is the first intervention addressing the inpatient overuse of IV antihypertensive medications for the treatment of asymptomatic hypertension. Additionally, this study bolsters prior evidence that the use of IV antihypertensives in asymptomatic patients leads to a large number of adverse events.⁷ A third of patients in the preintervention period had documented alternate etiologies of their blood pressure elevation, highlighting the need to assess and potentially treat these causes prior to treating blood pressure itself.

Reducing unnecessary treatment of asymptomatic blood pressure elevation is challenging. Evidence shows that both clinicians and patients overestimate the benefits and underestimate the harms of medical interventions.^13,14 This unfortunately leads to unjustified enthusiasm for medical treatments, which can worsen outcomes.¹⁵ Additionally, there may be a lack of knowledge of the guidelines, as well as the amount of time required in the full assessment of hypertensive urgency, that creates a culture of “treating the number.”

Changing physician behavior is difficult.¹⁶ However, active forms of continuing education and multifaceted interventions, such as ours, are most effective.¹⁷ Our message focused on patient safety and harm reduction, addressed clinicians’ safety concerns, and included stories of real cases where this overuse led to adverse events—all of which are encouraged in order to facilitate clinician engagement.¹⁸

There were limitations to this study. Only blood pressure elevations associated with an IV antihypertensive order and not all blood pressure elevations meeting the criteria for hypertensive urgency in general were examined. Additionally, our documentation of symptoms of hypertensive emergency and alternate etiologies was based only on documentation in the medical record. Ideally, we would have liked to conduct an interrupted time series analysis to assess the effect of the intervention over time; however, there were not enough orders of IV antihypertensives to perform such an analysis.

Treatment of asymptomatic blood pressure with IV antihypertensive medications can lead to patient harm. To reduce inappropriate treatment, our Student High Value Care team set out to challenge this common practice. Our interdisciplinary intervention successfully reduced unnecessary IV antihypertensive treatment. This may serve as a model for other institutions.

Disclosures

There are no relevant conflicts of interest to disclose for any authors.

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.

The CDC says the United States has hit a plateau with HIV infections. The feds’ infectious disease leadership testifies on the measles outbreaks. Big pharma tells Congress it can’t drop drug list prices alone. And why there may be too much screening for Barrett’s esophagus in patients with uncomplicated GERD.

Amazon Alexa
Apple Podcasts
Google Podcasts
Spotify

The CDC says the United States has hit a plateau with HIV infections. The feds’ infectious disease leadership testifies on the measles outbreaks. Big pharma tells Congress it can’t drop drug list prices alone. And why there may be too much screening for Barrett’s esophagus in patients with uncomplicated GERD.

Amazon Alexa
Apple Podcasts
Google Podcasts
Spotify

The CDC says the United States has hit a plateau with HIV infections. The feds’ infectious disease leadership testifies on the measles outbreaks. Big pharma tells Congress it can’t drop drug list prices alone. And why there may be too much screening for Barrett’s esophagus in patients with uncomplicated GERD.

Amazon Alexa
Apple Podcasts
Google Podcasts
Spotify

Kirti Magudia, MD, and Thomas Ng, MD, continue their conversation with host Nick Andrews, sharing their experiences as parents of young children as well as early career physicians.

Dr. Magudia and Dr. Ng, both diagnostic radiology fellows at Brigham and Women’s Hospital, Boston, discuss traditional ways of mitigating against burnout – practices such as exercise, meditation, and carving out self-care time – as well as finding meaning in researching and advocating for work-life integration for residents.
Apple Podcasts
Google Podcasts
Spotify

Contact us: [email protected]
Nick Andrews @tribnic on Twitter

Kirti Magudia, MD, and Thomas Ng, MD, continue their conversation with host Nick Andrews, sharing their experiences as parents of young children as well as early career physicians.

Dr. Magudia and Dr. Ng, both diagnostic radiology fellows at Brigham and Women’s Hospital, Boston, discuss traditional ways of mitigating against burnout – practices such as exercise, meditation, and carving out self-care time – as well as finding meaning in researching and advocating for work-life integration for residents.
Apple Podcasts
Google Podcasts
Spotify

Contact us: [email protected]
Nick Andrews @tribnic on Twitter

Kirti Magudia, MD, and Thomas Ng, MD, continue their conversation with host Nick Andrews, sharing their experiences as parents of young children as well as early career physicians.

Dr. Magudia and Dr. Ng, both diagnostic radiology fellows at Brigham and Women’s Hospital, Boston, discuss traditional ways of mitigating against burnout – practices such as exercise, meditation, and carving out self-care time – as well as finding meaning in researching and advocating for work-life integration for residents.
Apple Podcasts
Google Podcasts
Spotify

Contact us: [email protected]
Nick Andrews @tribnic on Twitter

Your pager beeps. Your patient, Mrs. Jones, who was admitted with cellulitis and is improving, now has a blood pressure of 188/103 on routine vitals. Her nurse reports that she is comfortable and asymptomatic, but she met the “call parameters.” You review her chart and find that since admission her systolic blood pressure (SBP) has ranged from 149 to 157 mm Hg and her diastolic blood pressure (DBP) from 84 to 96 mm Hg. Her nurse asks how you would like to treat her.

While over half of inpatients have at least one hypertensive episode during their stay, evidence suggests that nearly all such episodes—estimates are between 98% and 99%^1,2—should be treated over several days with oral antihypertensives, not acutely with intravenous medications.^3-6 Current guidelines recommend that intravenous medications should be reserved for severe hypertensive episodes (SBP > 180, DBP > 120) with acute end-organ damage,^7,8 but such “hypertensive emergencies” are rare on the general medicine wards. Still, hospitalists regularly face the dilemma posed by Mrs. Jones, and evidence shows they often prescribe intravenous antihypertensives.^1,4,5 This unnecessary use can lead to unreliable drops in blood pressure and exposes our patients to potential harm.^5,6

In this issue of the Journal of Hospital Medicine, two papers describe the frequency of inappropriate intravenous antihypertensive use in their hospitals and the subsequent quality improvement efforts implemented to reduce this practice. The first, by Jacobs et al., found that over a 10-month period, 11% of patients who experienced “asymptomatic hypertension” on an urban academic hospital medicine service were treated inappropriately with intravenous antihypertensives,⁹ with 14% of those experiencing an adverse event. The second paper, by Pasik et al., found that in their urban academic medical center there were 8.3 inappropriate intravenous antihypertensive orders placed per 1,000 patient days,¹⁰ with nearly half of those treated experiencing an adverse event. Based on these findings, each group then led interventions to reduce the use of intravenous antihypertensives.

While both groups engaged physicians and nurses as primary stakeholders, Pasik et al.¹⁰ worked to further expand nursing staff roles by empowering them to assess for underlying causes of hypertension, such as pain or anxiety, as well as end-organ damage via specific guided algorithms prior to contacting physicians. In doing so, they reduced intravenous antihypertensive use by 60% during the postintervention period, with a proportional reduction in adverse events. In addition to their educational initiative, Jacobs et al. aimed to limit calls by liberalizing the “ceiling” on standard nursing call parameters for blood pressure from 160/80 to 180/90. Following their intervention, intravenous antihypertensive orders were reduced by 40%, with the mean orders per patient with asymptomatic hypertension decreasing from 11% to 7% .

While these results are admirable, some caution in their interpretation is needed. For example, Jacobs et al. used electronic health record data to retrospectively identify hypertension as “symptomatic” or “asymptomatic” using laboratory, electrocardiogram, and imaging diagnostics as surrogate markers for “provider concern for end-organ damage.” Although it appropriately focused on concern for end-organ damage as justification for intravenous antihypertensives, this approach potentially underappreciated true hypertensive emergencies, thereby overestimating the amount of inappropriate use of intravenous antihypertensives. Pasik et al. utilized chart review of patients prescribed intravenous antihypertensives and therefore did not explore how often symptomatic hypertension occurred in patients who did not receive intravenous antihypertensives. Subsequently, this limited their ability to evaluate unintended harms of their initiative. To address this limitation, the authors followed a group of 111 patients who had elevated hypertension but did not receive intravenous antihypertensives and found no adverse outcomes.¹⁰ Because both studies were retrospective in nature, they were subject to biases from providers choosing intravenous antihypertensives for reasons that were neither captured by their datasets nor adjusted for. Additionally, neither study reported downstream impacts such as an increase in symptomatic hypertensive episodes or more rare events such as kidney injury, stroke, or myocardial infarction.

Given that guidelines discourage using intravenous antihypertensives, why were the efforts of Jacobs et al.⁹ and Pasik et al.¹⁰ needed in the first place? In a recent installment of Choosing Wisely: Things We Do For No Reason, Breu et al.¹¹ cite two primary reasons: first, providers have unfounded fears that asymptomatic hypertension will quickly progress to cause organ damage; second, providers lack understanding of the potential harms from overtreatment. It is fitting, therefore, that both groups of authors focused on these topics in their education initiatives for physicians and nurses. Yet, as good quality improvement requires steps beyond education, it was promising to see that both authors additionally focused on intervening to change the systems and culture that existed around physician and nursing communication.

In the age of electronic health records, there has been a sustained focus on creating standardized order sets. While the value of these order sets has been widely demonstrated, there are downsides. For example, nursing call parameters in admission order sets are rarely patient-specific but account for a significant portion of nursing and physician communication. These one-size-fits-all orders limit nurses from using their clinical training and create unnecessary tensions as nurses are obligated to call covering hospitalists to address “abnormal” but clinically insignificant findings. Regular monitoring of vital signs is an integral part of caring for acutely ill inpatients but for most inpatients, the importance of vitals is to detect clinically meaningful changes, not to treat risk factors like hypertension that should be treated safely over the long term.

When inpatients become febrile, tachycardic, or hypoxic, hospitalists use critical thinking to diagnose the underlying causes. Unfortunately, high blood pressure is a vital sign that is treated differently. Many hospitalists see it as a number to fix, not a potential sign of a new underlying problem such as uncontrolled pain, anxiety, or medication side effects.⁸ Both groups of authors took the important first step of educating physicians to think critically when called about high blood pressure. Even more importantly, they took steps to change the system and culture in which providers make these decisions in the first place. Future work in this area would be wise to follow in these footsteps, by encouraging collaboration between hospitalist and nurses to create more logical and patient-specific call parameters that could potentially improve nursing-physician communication, and subsequently, patient care.

Changing the culture to limit the use of intravenous antihypertensives will not be easy, but it is necessary. We encourage readers to investigate intravenous antihypertensives in their own hospitals and consider how better communication between nurses and physicians could change their practice. Recalling Mrs. Jones above, the provider should engage her nurse to help confirm that her hypertension is “asymptomatic” and then consider underlying causes such as pain, anxiety, or withholding her home medications as reasons for her elevated blood pressure. After all, if nothing else, it seems clear that a call about inpatient hypertension is not a call to act, but to think.

Disclosures

The authors declare that they have no competing interests.

Funding

Dr. Lucas is supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development and Dartmouth SYNERGY, National Institutes of Health, National Center for Translational Science (UL1TR001086).

Your pager beeps. Your patient, Mrs. Jones, who was admitted with cellulitis and is improving, now has a blood pressure of 188/103 on routine vitals. Her nurse reports that she is comfortable and asymptomatic, but she met the “call parameters.” You review her chart and find that since admission her systolic blood pressure (SBP) has ranged from 149 to 157 mm Hg and her diastolic blood pressure (DBP) from 84 to 96 mm Hg. Her nurse asks how you would like to treat her.

While over half of inpatients have at least one hypertensive episode during their stay, evidence suggests that nearly all such episodes—estimates are between 98% and 99%^1,2—should be treated over several days with oral antihypertensives, not acutely with intravenous medications.^3-6 Current guidelines recommend that intravenous medications should be reserved for severe hypertensive episodes (SBP > 180, DBP > 120) with acute end-organ damage,^7,8 but such “hypertensive emergencies” are rare on the general medicine wards. Still, hospitalists regularly face the dilemma posed by Mrs. Jones, and evidence shows they often prescribe intravenous antihypertensives.^1,4,5 This unnecessary use can lead to unreliable drops in blood pressure and exposes our patients to potential harm.^5,6

In this issue of the Journal of Hospital Medicine, two papers describe the frequency of inappropriate intravenous antihypertensive use in their hospitals and the subsequent quality improvement efforts implemented to reduce this practice. The first, by Jacobs et al., found that over a 10-month period, 11% of patients who experienced “asymptomatic hypertension” on an urban academic hospital medicine service were treated inappropriately with intravenous antihypertensives,⁹ with 14% of those experiencing an adverse event. The second paper, by Pasik et al., found that in their urban academic medical center there were 8.3 inappropriate intravenous antihypertensive orders placed per 1,000 patient days,¹⁰ with nearly half of those treated experiencing an adverse event. Based on these findings, each group then led interventions to reduce the use of intravenous antihypertensives.

While both groups engaged physicians and nurses as primary stakeholders, Pasik et al.¹⁰ worked to further expand nursing staff roles by empowering them to assess for underlying causes of hypertension, such as pain or anxiety, as well as end-organ damage via specific guided algorithms prior to contacting physicians. In doing so, they reduced intravenous antihypertensive use by 60% during the postintervention period, with a proportional reduction in adverse events. In addition to their educational initiative, Jacobs et al. aimed to limit calls by liberalizing the “ceiling” on standard nursing call parameters for blood pressure from 160/80 to 180/90. Following their intervention, intravenous antihypertensive orders were reduced by 40%, with the mean orders per patient with asymptomatic hypertension decreasing from 11% to 7% .

While these results are admirable, some caution in their interpretation is needed. For example, Jacobs et al. used electronic health record data to retrospectively identify hypertension as “symptomatic” or “asymptomatic” using laboratory, electrocardiogram, and imaging diagnostics as surrogate markers for “provider concern for end-organ damage.” Although it appropriately focused on concern for end-organ damage as justification for intravenous antihypertensives, this approach potentially underappreciated true hypertensive emergencies, thereby overestimating the amount of inappropriate use of intravenous antihypertensives. Pasik et al. utilized chart review of patients prescribed intravenous antihypertensives and therefore did not explore how often symptomatic hypertension occurred in patients who did not receive intravenous antihypertensives. Subsequently, this limited their ability to evaluate unintended harms of their initiative. To address this limitation, the authors followed a group of 111 patients who had elevated hypertension but did not receive intravenous antihypertensives and found no adverse outcomes.¹⁰ Because both studies were retrospective in nature, they were subject to biases from providers choosing intravenous antihypertensives for reasons that were neither captured by their datasets nor adjusted for. Additionally, neither study reported downstream impacts such as an increase in symptomatic hypertensive episodes or more rare events such as kidney injury, stroke, or myocardial infarction.

Given that guidelines discourage using intravenous antihypertensives, why were the efforts of Jacobs et al.⁹ and Pasik et al.¹⁰ needed in the first place? In a recent installment of Choosing Wisely: Things We Do For No Reason, Breu et al.¹¹ cite two primary reasons: first, providers have unfounded fears that asymptomatic hypertension will quickly progress to cause organ damage; second, providers lack understanding of the potential harms from overtreatment. It is fitting, therefore, that both groups of authors focused on these topics in their education initiatives for physicians and nurses. Yet, as good quality improvement requires steps beyond education, it was promising to see that both authors additionally focused on intervening to change the systems and culture that existed around physician and nursing communication.

In the age of electronic health records, there has been a sustained focus on creating standardized order sets. While the value of these order sets has been widely demonstrated, there are downsides. For example, nursing call parameters in admission order sets are rarely patient-specific but account for a significant portion of nursing and physician communication. These one-size-fits-all orders limit nurses from using their clinical training and create unnecessary tensions as nurses are obligated to call covering hospitalists to address “abnormal” but clinically insignificant findings. Regular monitoring of vital signs is an integral part of caring for acutely ill inpatients but for most inpatients, the importance of vitals is to detect clinically meaningful changes, not to treat risk factors like hypertension that should be treated safely over the long term.

When inpatients become febrile, tachycardic, or hypoxic, hospitalists use critical thinking to diagnose the underlying causes. Unfortunately, high blood pressure is a vital sign that is treated differently. Many hospitalists see it as a number to fix, not a potential sign of a new underlying problem such as uncontrolled pain, anxiety, or medication side effects.⁸ Both groups of authors took the important first step of educating physicians to think critically when called about high blood pressure. Even more importantly, they took steps to change the system and culture in which providers make these decisions in the first place. Future work in this area would be wise to follow in these footsteps, by encouraging collaboration between hospitalist and nurses to create more logical and patient-specific call parameters that could potentially improve nursing-physician communication, and subsequently, patient care.

Changing the culture to limit the use of intravenous antihypertensives will not be easy, but it is necessary. We encourage readers to investigate intravenous antihypertensives in their own hospitals and consider how better communication between nurses and physicians could change their practice. Recalling Mrs. Jones above, the provider should engage her nurse to help confirm that her hypertension is “asymptomatic” and then consider underlying causes such as pain, anxiety, or withholding her home medications as reasons for her elevated blood pressure. After all, if nothing else, it seems clear that a call about inpatient hypertension is not a call to act, but to think.

Disclosures

The authors declare that they have no competing interests.

Funding

Dr. Lucas is supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development and Dartmouth SYNERGY, National Institutes of Health, National Center for Translational Science (UL1TR001086).

Your pager beeps. Your patient, Mrs. Jones, who was admitted with cellulitis and is improving, now has a blood pressure of 188/103 on routine vitals. Her nurse reports that she is comfortable and asymptomatic, but she met the “call parameters.” You review her chart and find that since admission her systolic blood pressure (SBP) has ranged from 149 to 157 mm Hg and her diastolic blood pressure (DBP) from 84 to 96 mm Hg. Her nurse asks how you would like to treat her.

While over half of inpatients have at least one hypertensive episode during their stay, evidence suggests that nearly all such episodes—estimates are between 98% and 99%^1,2—should be treated over several days with oral antihypertensives, not acutely with intravenous medications.^3-6 Current guidelines recommend that intravenous medications should be reserved for severe hypertensive episodes (SBP > 180, DBP > 120) with acute end-organ damage,^7,8 but such “hypertensive emergencies” are rare on the general medicine wards. Still, hospitalists regularly face the dilemma posed by Mrs. Jones, and evidence shows they often prescribe intravenous antihypertensives.^1,4,5 This unnecessary use can lead to unreliable drops in blood pressure and exposes our patients to potential harm.^5,6

In this issue of the Journal of Hospital Medicine, two papers describe the frequency of inappropriate intravenous antihypertensive use in their hospitals and the subsequent quality improvement efforts implemented to reduce this practice. The first, by Jacobs et al., found that over a 10-month period, 11% of patients who experienced “asymptomatic hypertension” on an urban academic hospital medicine service were treated inappropriately with intravenous antihypertensives,⁹ with 14% of those experiencing an adverse event. The second paper, by Pasik et al., found that in their urban academic medical center there were 8.3 inappropriate intravenous antihypertensive orders placed per 1,000 patient days,¹⁰ with nearly half of those treated experiencing an adverse event. Based on these findings, each group then led interventions to reduce the use of intravenous antihypertensives.

While both groups engaged physicians and nurses as primary stakeholders, Pasik et al.¹⁰ worked to further expand nursing staff roles by empowering them to assess for underlying causes of hypertension, such as pain or anxiety, as well as end-organ damage via specific guided algorithms prior to contacting physicians. In doing so, they reduced intravenous antihypertensive use by 60% during the postintervention period, with a proportional reduction in adverse events. In addition to their educational initiative, Jacobs et al. aimed to limit calls by liberalizing the “ceiling” on standard nursing call parameters for blood pressure from 160/80 to 180/90. Following their intervention, intravenous antihypertensive orders were reduced by 40%, with the mean orders per patient with asymptomatic hypertension decreasing from 11% to 7% .

While these results are admirable, some caution in their interpretation is needed. For example, Jacobs et al. used electronic health record data to retrospectively identify hypertension as “symptomatic” or “asymptomatic” using laboratory, electrocardiogram, and imaging diagnostics as surrogate markers for “provider concern for end-organ damage.” Although it appropriately focused on concern for end-organ damage as justification for intravenous antihypertensives, this approach potentially underappreciated true hypertensive emergencies, thereby overestimating the amount of inappropriate use of intravenous antihypertensives. Pasik et al. utilized chart review of patients prescribed intravenous antihypertensives and therefore did not explore how often symptomatic hypertension occurred in patients who did not receive intravenous antihypertensives. Subsequently, this limited their ability to evaluate unintended harms of their initiative. To address this limitation, the authors followed a group of 111 patients who had elevated hypertension but did not receive intravenous antihypertensives and found no adverse outcomes.¹⁰ Because both studies were retrospective in nature, they were subject to biases from providers choosing intravenous antihypertensives for reasons that were neither captured by their datasets nor adjusted for. Additionally, neither study reported downstream impacts such as an increase in symptomatic hypertensive episodes or more rare events such as kidney injury, stroke, or myocardial infarction.

Given that guidelines discourage using intravenous antihypertensives, why were the efforts of Jacobs et al.⁹ and Pasik et al.¹⁰ needed in the first place? In a recent installment of Choosing Wisely: Things We Do For No Reason, Breu et al.¹¹ cite two primary reasons: first, providers have unfounded fears that asymptomatic hypertension will quickly progress to cause organ damage; second, providers lack understanding of the potential harms from overtreatment. It is fitting, therefore, that both groups of authors focused on these topics in their education initiatives for physicians and nurses. Yet, as good quality improvement requires steps beyond education, it was promising to see that both authors additionally focused on intervening to change the systems and culture that existed around physician and nursing communication.

In the age of electronic health records, there has been a sustained focus on creating standardized order sets. While the value of these order sets has been widely demonstrated, there are downsides. For example, nursing call parameters in admission order sets are rarely patient-specific but account for a significant portion of nursing and physician communication. These one-size-fits-all orders limit nurses from using their clinical training and create unnecessary tensions as nurses are obligated to call covering hospitalists to address “abnormal” but clinically insignificant findings. Regular monitoring of vital signs is an integral part of caring for acutely ill inpatients but for most inpatients, the importance of vitals is to detect clinically meaningful changes, not to treat risk factors like hypertension that should be treated safely over the long term.

When inpatients become febrile, tachycardic, or hypoxic, hospitalists use critical thinking to diagnose the underlying causes. Unfortunately, high blood pressure is a vital sign that is treated differently. Many hospitalists see it as a number to fix, not a potential sign of a new underlying problem such as uncontrolled pain, anxiety, or medication side effects.⁸ Both groups of authors took the important first step of educating physicians to think critically when called about high blood pressure. Even more importantly, they took steps to change the system and culture in which providers make these decisions in the first place. Future work in this area would be wise to follow in these footsteps, by encouraging collaboration between hospitalist and nurses to create more logical and patient-specific call parameters that could potentially improve nursing-physician communication, and subsequently, patient care.

Changing the culture to limit the use of intravenous antihypertensives will not be easy, but it is necessary. We encourage readers to investigate intravenous antihypertensives in their own hospitals and consider how better communication between nurses and physicians could change their practice. Recalling Mrs. Jones above, the provider should engage her nurse to help confirm that her hypertension is “asymptomatic” and then consider underlying causes such as pain, anxiety, or withholding her home medications as reasons for her elevated blood pressure. After all, if nothing else, it seems clear that a call about inpatient hypertension is not a call to act, but to think.

Disclosures

The authors declare that they have no competing interests.

Funding

Dr. Lucas is supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development and Dartmouth SYNERGY, National Institutes of Health, National Center for Translational Science (UL1TR001086).

Clinical guidelines are periodically released by medical societies with the overarching goal of improving deliverable medical care by standardizing disease management according to best available published literature and by reducing healthcare expenditure associated with unnecessary and superfluous testing.¹ Unfortunately, nonadherence to guidelines is common in clinical practice² and contributes to the rising cost of healthcare.³ Health resource utilization is particularly relevant in management of cirrhosis, a condition with an annual healthcare expenditure of $13 billion.⁴ Hepatic encephalopathy (HE), the most common complication of cirrhosis, is characterized by altered sensorium and is the leading indication for hospitalization among cirrhotics. The joint guidelines of the European Association for the Study of the Liver (EASL) and the American Association for the Study of Liver Diseases (AASLD) for diagnostic workup for HE recommend identification and treatment of potential precipitants.⁵ The guidelines also recommend against checking serum ammonia levels, which have not been shown to correlate with diagnosis or severity of HE.^6-8 Currently, limited data are available on practice patterns regarding guideline adherence and unnecessary serum ammonia testing for initial evaluation of HE in hospitals. To overcome this gap in knowledge, we conducted the present study to provide granular details regarding the diagnostic workup for hospitalized patients with HE.

This study adopted a retrospective design and recruited patients admitted to the Virginia Commonwealth University Medical Center between July 1, 2016 and July 1, 2017. The institutional review board approved the study, and the manuscript was reviewed and approved by all authors prior to submission. All chart reviews were performed by hepatologists with access to patients’ electronic medical record (EMR).

Patient Population

Patients were identified from the EMR system by using ICD-9 and ICD-10 codes for cirrhosis, hepatic encephalopathy, and altered mental status. All consecutive admissions with these diagnosis codes were considered for inclusion. Adult patients with cirrhosis resulting from any etiology of chronic liver diseases with primary reason for admission of HE were included. If patients were readmitted for HE during the study period, then only the data from index HE admission was included in the analysis and data from subsequent admissions were excluded. The other exclusion criteria included non-HE causes of confusion, acute liver failure, and those admitted with a preformulated plan (eg, direct hepatology clinic admission or outside hospital transfer). Patients who developed HE during their hospitalization where HE was not the indication for admission were also excluded. Finally, all patients admitted under the direct care of hepatology were excluded.

Diagnostic Workup

The recommendations of the AASLD and the EASL for workup for HE include obtaining detailed history and physical examination supplemented by diagnostic evaluation for potential HE precipitants including infections, electrolyte disturbances, dehydration, renal failure, glycemic disturbances, and toxin ingestion (eg, alcohol, illicit drugs).⁵ Based on the guideline recommendation, this study defined a “complete workup” as including all of the following elements: infection evaluation (blood culture, urinalysis/urine culture, chest radiograph, diagnostic paracentesis in the presence of ascites), electrolyte/renal evaluation (serum sodium, potassium, creatinine, and glucose), and toxin evaluation (urine drug screening). Any HE admission that was missing elements from the aforementioned battery of tests was defined as “incomplete workup.” In patients admitted with decompensated cirrhosis, serum ammonia testing was considered inappropriate unless there was a nuanced explanation supporting its use documented within the EMR. The frequency and specialty of the physician ordering serum ammonia level tests were determined. The financial burden of unnecessary ammonia testing was estimated by assigning a laboratory charge ($258) for each patient.

Statistical Analysis

Continuous and categorical variables are reported as means (± standard deviation), median (interquartile range or IQR), or proportion (%) as appropriate. Across-group differences were compared using Student t-test for normally distributed continuous variables and Mann-Whitney U test for skewed data. Fisher’s exact test was used to compare proportion. HE evaluations were quantified by the number of patients with complete workup and by the number of patients with missing components of the workup. A nominal P value of less than .05 was considered statistically significant. All statistical analyses were performed using SPSS Statistics version 24.0 (IBM Corporation, Armonk, New York).

Cohort Characteristics

The baseline cohort demographics are listed in the Table. Of the 145 patients identified using diagnostic codes for cirrhosis, 78 subjects met the study criteria. The most common exclusion criteria included non-HE etiology of altered mental status (n = 37) and patients with readmissions for HE during the study period (n = 30). The mean age of the study cohort was 59.3 years, and the most common etiology of cirrhosis was hepatitis C (n = 41), alcohol induced (n = 14), and nonalcoholic steatohepatitis (n = 13).

Initial Diagnostic Evaluation

The major precipitants of HE in the study cohort were ineffective lactulose dosing (n = 43), infections (n = 25), and electrolyte disturbances/renal injury (n = 6). At the time of admission, 53 patients were on therapy for HE. Only 17 (22%) patients had complete diagnostic workup within 24 hours of hospital admission. The individual components of the complete workup are shown in the Figure. Notably, 23 (30%) patients were missing blood cultures, 16 (21%) were missing urinalysis, 15 (20%) were missing chest radiograph, and 34 (44%) were missing urine drug screening. Of the 34 patients with ascites on admission, only eight (23%) had diagnostic paracentesis performed on admission to rule out spontaneous bacterial peritonitis.

Serum Ammonia Testing

Serum ammonia testing was performed on 74 patients (94.9%), and no patient met the criteria for appropriate testing. Forty patients already had a known diagnosis of HE prior to index admission. Furthermore, 10 (14%) patients had serum ammonia testing repeated after admission without documentation in the EMR to justify repeat testing. Emergency Department (ED) physicians ordered ammonia testing in 57 cases (77%), internists ordered the testing in 11 cases (15%), and intensivists ordered the testing in two cases (3%). The patient’s charges for serum ammonia testing at the time of admission and for repeat testing were $19,092 and $2,580, respectively.

This study utilized HE in patients with decompensated cirrhosis as a framework to analyze adherence to societal guidelines. The adherence rate to AALSD/EASL recommended inpatient evaluation of HE is surprisingly low, and most patients are missing key essential elements of the diagnostic work up. While the diagnostic tests that are ordered as part of a panel are completed universally (renal function, electrolytes, and glucose testing), individual testing is less inclined to be ordered (blood cultures, urine culture/urinalysis, CXR, UDS) and procedural testing, such as diagnostic paracentesis, is often missed. This last finding is in line with published literature showing that 40% of patients admitted with ascites or HE did not have diagnostic paracentesis during hospital admission despite 24% reduction of inhospital mortality among patients undergoing the procedure.⁹

Although serum ammonia testing is not endorsed by the AASLD/EASL guidelines for HE,⁵ it is ordered nearly universally. The cost of an individual test is relatively low, but the cumulative cost of serum ammonia testing can be substantial because HE is the most common indication for hospitalization among patients with cirrhosis.⁴ Initiatives, such as the Choosing Wisely^® campaign, encourage high-value and evidence-based care by limiting excessive and unnecessary diagnostic testing.¹⁰ The Canadian Choosing Wisely campaign specifically includes avoidance of serum ammonia testing for diagnosis of HE to provide high-value care in hepatology.¹¹

Although the exact reasons for nonadherence to recommended HE evaluations are unclear, a potential method to mitigate excessive testing is to utilize the EMR and ordering system.³ EMR-based strategies can curb unnecessary testing in inpatient settings.¹² The use of HE order sets, the inclusion of clinical decision support systems, and the restriction of access to specialized testing can be readily incorporated into the EMR to encourage adherence to guideline-based care while limiting unnecessary testing.

This study should be interpreted in the context of study limitations. Given the retrospective design of the study, salient factors in decisions behind diagnostic testing cannot be assessed. Future studies should utilize mixed-model methodology to elucidate reasons behind these decisions. The present study used a strict definition of complete workup including all the mentioned elements of the diagnostic workup for HE; however, in clinical practice, providers could be justified in not ordering certain tests if the specific clinical scenario does not lead to its use (eg, chest X-ray deferred in a patient with clear lung exam, no symptoms, or hypoxia). Similarly, UDS was included as a required element for a complete workup. While it may be ordered in a case-by-case basis to screen for illicit drug abuse, UDS is also a critical element of the workup to screen for opioid use as a precipitant of HE. Finally, considering the strict study entry criteria, we excluded repeated admissions for HE during the study period and therefore likely underestimate the cost burden of serum ammonia testing.

In conclusion, valuable guideline-based diagnostic testing is often missing in patients admitted for HE while serum ammonia testing is nearly universally ordered. These findings underscore the importance of implementing educational strategies, such as the Choosing Wisely^® campaign, and EMR-based clinical decision support systems to improve health resource utilization in patients with cirrhosis and HE.

Disclosures

The authors have nothing to disclose.

Clinical guidelines are periodically released by medical societies with the overarching goal of improving deliverable medical care by standardizing disease management according to best available published literature and by reducing healthcare expenditure associated with unnecessary and superfluous testing.¹ Unfortunately, nonadherence to guidelines is common in clinical practice² and contributes to the rising cost of healthcare.³ Health resource utilization is particularly relevant in management of cirrhosis, a condition with an annual healthcare expenditure of $13 billion.⁴ Hepatic encephalopathy (HE), the most common complication of cirrhosis, is characterized by altered sensorium and is the leading indication for hospitalization among cirrhotics. The joint guidelines of the European Association for the Study of the Liver (EASL) and the American Association for the Study of Liver Diseases (AASLD) for diagnostic workup for HE recommend identification and treatment of potential precipitants.⁵ The guidelines also recommend against checking serum ammonia levels, which have not been shown to correlate with diagnosis or severity of HE.^6-8 Currently, limited data are available on practice patterns regarding guideline adherence and unnecessary serum ammonia testing for initial evaluation of HE in hospitals. To overcome this gap in knowledge, we conducted the present study to provide granular details regarding the diagnostic workup for hospitalized patients with HE.

This study adopted a retrospective design and recruited patients admitted to the Virginia Commonwealth University Medical Center between July 1, 2016 and July 1, 2017. The institutional review board approved the study, and the manuscript was reviewed and approved by all authors prior to submission. All chart reviews were performed by hepatologists with access to patients’ electronic medical record (EMR).

Patient Population

Patients were identified from the EMR system by using ICD-9 and ICD-10 codes for cirrhosis, hepatic encephalopathy, and altered mental status. All consecutive admissions with these diagnosis codes were considered for inclusion. Adult patients with cirrhosis resulting from any etiology of chronic liver diseases with primary reason for admission of HE were included. If patients were readmitted for HE during the study period, then only the data from index HE admission was included in the analysis and data from subsequent admissions were excluded. The other exclusion criteria included non-HE causes of confusion, acute liver failure, and those admitted with a preformulated plan (eg, direct hepatology clinic admission or outside hospital transfer). Patients who developed HE during their hospitalization where HE was not the indication for admission were also excluded. Finally, all patients admitted under the direct care of hepatology were excluded.

Diagnostic Workup

The recommendations of the AASLD and the EASL for workup for HE include obtaining detailed history and physical examination supplemented by diagnostic evaluation for potential HE precipitants including infections, electrolyte disturbances, dehydration, renal failure, glycemic disturbances, and toxin ingestion (eg, alcohol, illicit drugs).⁵ Based on the guideline recommendation, this study defined a “complete workup” as including all of the following elements: infection evaluation (blood culture, urinalysis/urine culture, chest radiograph, diagnostic paracentesis in the presence of ascites), electrolyte/renal evaluation (serum sodium, potassium, creatinine, and glucose), and toxin evaluation (urine drug screening). Any HE admission that was missing elements from the aforementioned battery of tests was defined as “incomplete workup.” In patients admitted with decompensated cirrhosis, serum ammonia testing was considered inappropriate unless there was a nuanced explanation supporting its use documented within the EMR. The frequency and specialty of the physician ordering serum ammonia level tests were determined. The financial burden of unnecessary ammonia testing was estimated by assigning a laboratory charge ($258) for each patient.

Statistical Analysis

Continuous and categorical variables are reported as means (± standard deviation), median (interquartile range or IQR), or proportion (%) as appropriate. Across-group differences were compared using Student t-test for normally distributed continuous variables and Mann-Whitney U test for skewed data. Fisher’s exact test was used to compare proportion. HE evaluations were quantified by the number of patients with complete workup and by the number of patients with missing components of the workup. A nominal P value of less than .05 was considered statistically significant. All statistical analyses were performed using SPSS Statistics version 24.0 (IBM Corporation, Armonk, New York).

Cohort Characteristics

The baseline cohort demographics are listed in the Table. Of the 145 patients identified using diagnostic codes for cirrhosis, 78 subjects met the study criteria. The most common exclusion criteria included non-HE etiology of altered mental status (n = 37) and patients with readmissions for HE during the study period (n = 30). The mean age of the study cohort was 59.3 years, and the most common etiology of cirrhosis was hepatitis C (n = 41), alcohol induced (n = 14), and nonalcoholic steatohepatitis (n = 13).

Initial Diagnostic Evaluation

The major precipitants of HE in the study cohort were ineffective lactulose dosing (n = 43), infections (n = 25), and electrolyte disturbances/renal injury (n = 6). At the time of admission, 53 patients were on therapy for HE. Only 17 (22%) patients had complete diagnostic workup within 24 hours of hospital admission. The individual components of the complete workup are shown in the Figure. Notably, 23 (30%) patients were missing blood cultures, 16 (21%) were missing urinalysis, 15 (20%) were missing chest radiograph, and 34 (44%) were missing urine drug screening. Of the 34 patients with ascites on admission, only eight (23%) had diagnostic paracentesis performed on admission to rule out spontaneous bacterial peritonitis.

Serum Ammonia Testing

Serum ammonia testing was performed on 74 patients (94.9%), and no patient met the criteria for appropriate testing. Forty patients already had a known diagnosis of HE prior to index admission. Furthermore, 10 (14%) patients had serum ammonia testing repeated after admission without documentation in the EMR to justify repeat testing. Emergency Department (ED) physicians ordered ammonia testing in 57 cases (77%), internists ordered the testing in 11 cases (15%), and intensivists ordered the testing in two cases (3%). The patient’s charges for serum ammonia testing at the time of admission and for repeat testing were $19,092 and $2,580, respectively.

This study utilized HE in patients with decompensated cirrhosis as a framework to analyze adherence to societal guidelines. The adherence rate to AALSD/EASL recommended inpatient evaluation of HE is surprisingly low, and most patients are missing key essential elements of the diagnostic work up. While the diagnostic tests that are ordered as part of a panel are completed universally (renal function, electrolytes, and glucose testing), individual testing is less inclined to be ordered (blood cultures, urine culture/urinalysis, CXR, UDS) and procedural testing, such as diagnostic paracentesis, is often missed. This last finding is in line with published literature showing that 40% of patients admitted with ascites or HE did not have diagnostic paracentesis during hospital admission despite 24% reduction of inhospital mortality among patients undergoing the procedure.⁹

Although serum ammonia testing is not endorsed by the AASLD/EASL guidelines for HE,⁵ it is ordered nearly universally. The cost of an individual test is relatively low, but the cumulative cost of serum ammonia testing can be substantial because HE is the most common indication for hospitalization among patients with cirrhosis.⁴ Initiatives, such as the Choosing Wisely^® campaign, encourage high-value and evidence-based care by limiting excessive and unnecessary diagnostic testing.¹⁰ The Canadian Choosing Wisely campaign specifically includes avoidance of serum ammonia testing for diagnosis of HE to provide high-value care in hepatology.¹¹

Although the exact reasons for nonadherence to recommended HE evaluations are unclear, a potential method to mitigate excessive testing is to utilize the EMR and ordering system.³ EMR-based strategies can curb unnecessary testing in inpatient settings.¹² The use of HE order sets, the inclusion of clinical decision support systems, and the restriction of access to specialized testing can be readily incorporated into the EMR to encourage adherence to guideline-based care while limiting unnecessary testing.

This study should be interpreted in the context of study limitations. Given the retrospective design of the study, salient factors in decisions behind diagnostic testing cannot be assessed. Future studies should utilize mixed-model methodology to elucidate reasons behind these decisions. The present study used a strict definition of complete workup including all the mentioned elements of the diagnostic workup for HE; however, in clinical practice, providers could be justified in not ordering certain tests if the specific clinical scenario does not lead to its use (eg, chest X-ray deferred in a patient with clear lung exam, no symptoms, or hypoxia). Similarly, UDS was included as a required element for a complete workup. While it may be ordered in a case-by-case basis to screen for illicit drug abuse, UDS is also a critical element of the workup to screen for opioid use as a precipitant of HE. Finally, considering the strict study entry criteria, we excluded repeated admissions for HE during the study period and therefore likely underestimate the cost burden of serum ammonia testing.

In conclusion, valuable guideline-based diagnostic testing is often missing in patients admitted for HE while serum ammonia testing is nearly universally ordered. These findings underscore the importance of implementing educational strategies, such as the Choosing Wisely^® campaign, and EMR-based clinical decision support systems to improve health resource utilization in patients with cirrhosis and HE.

Disclosures

The authors have nothing to disclose.

Clinical guidelines are periodically released by medical societies with the overarching goal of improving deliverable medical care by standardizing disease management according to best available published literature and by reducing healthcare expenditure associated with unnecessary and superfluous testing.¹ Unfortunately, nonadherence to guidelines is common in clinical practice² and contributes to the rising cost of healthcare.³ Health resource utilization is particularly relevant in management of cirrhosis, a condition with an annual healthcare expenditure of $13 billion.⁴ Hepatic encephalopathy (HE), the most common complication of cirrhosis, is characterized by altered sensorium and is the leading indication for hospitalization among cirrhotics. The joint guidelines of the European Association for the Study of the Liver (EASL) and the American Association for the Study of Liver Diseases (AASLD) for diagnostic workup for HE recommend identification and treatment of potential precipitants.⁵ The guidelines also recommend against checking serum ammonia levels, which have not been shown to correlate with diagnosis or severity of HE.^6-8 Currently, limited data are available on practice patterns regarding guideline adherence and unnecessary serum ammonia testing for initial evaluation of HE in hospitals. To overcome this gap in knowledge, we conducted the present study to provide granular details regarding the diagnostic workup for hospitalized patients with HE.

This study adopted a retrospective design and recruited patients admitted to the Virginia Commonwealth University Medical Center between July 1, 2016 and July 1, 2017. The institutional review board approved the study, and the manuscript was reviewed and approved by all authors prior to submission. All chart reviews were performed by hepatologists with access to patients’ electronic medical record (EMR).

Patient Population

Patients were identified from the EMR system by using ICD-9 and ICD-10 codes for cirrhosis, hepatic encephalopathy, and altered mental status. All consecutive admissions with these diagnosis codes were considered for inclusion. Adult patients with cirrhosis resulting from any etiology of chronic liver diseases with primary reason for admission of HE were included. If patients were readmitted for HE during the study period, then only the data from index HE admission was included in the analysis and data from subsequent admissions were excluded. The other exclusion criteria included non-HE causes of confusion, acute liver failure, and those admitted with a preformulated plan (eg, direct hepatology clinic admission or outside hospital transfer). Patients who developed HE during their hospitalization where HE was not the indication for admission were also excluded. Finally, all patients admitted under the direct care of hepatology were excluded.

Diagnostic Workup

The recommendations of the AASLD and the EASL for workup for HE include obtaining detailed history and physical examination supplemented by diagnostic evaluation for potential HE precipitants including infections, electrolyte disturbances, dehydration, renal failure, glycemic disturbances, and toxin ingestion (eg, alcohol, illicit drugs).⁵ Based on the guideline recommendation, this study defined a “complete workup” as including all of the following elements: infection evaluation (blood culture, urinalysis/urine culture, chest radiograph, diagnostic paracentesis in the presence of ascites), electrolyte/renal evaluation (serum sodium, potassium, creatinine, and glucose), and toxin evaluation (urine drug screening). Any HE admission that was missing elements from the aforementioned battery of tests was defined as “incomplete workup.” In patients admitted with decompensated cirrhosis, serum ammonia testing was considered inappropriate unless there was a nuanced explanation supporting its use documented within the EMR. The frequency and specialty of the physician ordering serum ammonia level tests were determined. The financial burden of unnecessary ammonia testing was estimated by assigning a laboratory charge ($258) for each patient.

Statistical Analysis

Continuous and categorical variables are reported as means (± standard deviation), median (interquartile range or IQR), or proportion (%) as appropriate. Across-group differences were compared using Student t-test for normally distributed continuous variables and Mann-Whitney U test for skewed data. Fisher’s exact test was used to compare proportion. HE evaluations were quantified by the number of patients with complete workup and by the number of patients with missing components of the workup. A nominal P value of less than .05 was considered statistically significant. All statistical analyses were performed using SPSS Statistics version 24.0 (IBM Corporation, Armonk, New York).

Cohort Characteristics

The baseline cohort demographics are listed in the Table. Of the 145 patients identified using diagnostic codes for cirrhosis, 78 subjects met the study criteria. The most common exclusion criteria included non-HE etiology of altered mental status (n = 37) and patients with readmissions for HE during the study period (n = 30). The mean age of the study cohort was 59.3 years, and the most common etiology of cirrhosis was hepatitis C (n = 41), alcohol induced (n = 14), and nonalcoholic steatohepatitis (n = 13).

Initial Diagnostic Evaluation

The major precipitants of HE in the study cohort were ineffective lactulose dosing (n = 43), infections (n = 25), and electrolyte disturbances/renal injury (n = 6). At the time of admission, 53 patients were on therapy for HE. Only 17 (22%) patients had complete diagnostic workup within 24 hours of hospital admission. The individual components of the complete workup are shown in the Figure. Notably, 23 (30%) patients were missing blood cultures, 16 (21%) were missing urinalysis, 15 (20%) were missing chest radiograph, and 34 (44%) were missing urine drug screening. Of the 34 patients with ascites on admission, only eight (23%) had diagnostic paracentesis performed on admission to rule out spontaneous bacterial peritonitis.

Serum Ammonia Testing

Serum ammonia testing was performed on 74 patients (94.9%), and no patient met the criteria for appropriate testing. Forty patients already had a known diagnosis of HE prior to index admission. Furthermore, 10 (14%) patients had serum ammonia testing repeated after admission without documentation in the EMR to justify repeat testing. Emergency Department (ED) physicians ordered ammonia testing in 57 cases (77%), internists ordered the testing in 11 cases (15%), and intensivists ordered the testing in two cases (3%). The patient’s charges for serum ammonia testing at the time of admission and for repeat testing were $19,092 and $2,580, respectively.

This study utilized HE in patients with decompensated cirrhosis as a framework to analyze adherence to societal guidelines. The adherence rate to AALSD/EASL recommended inpatient evaluation of HE is surprisingly low, and most patients are missing key essential elements of the diagnostic work up. While the diagnostic tests that are ordered as part of a panel are completed universally (renal function, electrolytes, and glucose testing), individual testing is less inclined to be ordered (blood cultures, urine culture/urinalysis, CXR, UDS) and procedural testing, such as diagnostic paracentesis, is often missed. This last finding is in line with published literature showing that 40% of patients admitted with ascites or HE did not have diagnostic paracentesis during hospital admission despite 24% reduction of inhospital mortality among patients undergoing the procedure.⁹

Although serum ammonia testing is not endorsed by the AASLD/EASL guidelines for HE,⁵ it is ordered nearly universally. The cost of an individual test is relatively low, but the cumulative cost of serum ammonia testing can be substantial because HE is the most common indication for hospitalization among patients with cirrhosis.⁴ Initiatives, such as the Choosing Wisely^® campaign, encourage high-value and evidence-based care by limiting excessive and unnecessary diagnostic testing.¹⁰ The Canadian Choosing Wisely campaign specifically includes avoidance of serum ammonia testing for diagnosis of HE to provide high-value care in hepatology.¹¹

Although the exact reasons for nonadherence to recommended HE evaluations are unclear, a potential method to mitigate excessive testing is to utilize the EMR and ordering system.³ EMR-based strategies can curb unnecessary testing in inpatient settings.¹² The use of HE order sets, the inclusion of clinical decision support systems, and the restriction of access to specialized testing can be readily incorporated into the EMR to encourage adherence to guideline-based care while limiting unnecessary testing.

This study should be interpreted in the context of study limitations. Given the retrospective design of the study, salient factors in decisions behind diagnostic testing cannot be assessed. Future studies should utilize mixed-model methodology to elucidate reasons behind these decisions. The present study used a strict definition of complete workup including all the mentioned elements of the diagnostic workup for HE; however, in clinical practice, providers could be justified in not ordering certain tests if the specific clinical scenario does not lead to its use (eg, chest X-ray deferred in a patient with clear lung exam, no symptoms, or hypoxia). Similarly, UDS was included as a required element for a complete workup. While it may be ordered in a case-by-case basis to screen for illicit drug abuse, UDS is also a critical element of the workup to screen for opioid use as a precipitant of HE. Finally, considering the strict study entry criteria, we excluded repeated admissions for HE during the study period and therefore likely underestimate the cost burden of serum ammonia testing.

In conclusion, valuable guideline-based diagnostic testing is often missing in patients admitted for HE while serum ammonia testing is nearly universally ordered. These findings underscore the importance of implementing educational strategies, such as the Choosing Wisely^® campaign, and EMR-based clinical decision support systems to improve health resource utilization in patients with cirrhosis and HE.

Disclosures

The authors have nothing to disclose.

As part of the Journal of Hospital Medicine’s^® commitment to our readership, we are excited to announce innovative new review formats, designed for busy hospitalists. The state of knowledge in our field is changing rapidly, and the 21^st century poses a conundrum to clinicians in the form of increasingly complex studies and guidelines amidst ever-decreasing time to digest them. As a result, it can be challenging for hospitalists to access and interpret recently published research to inform their clinical practice. Because we are committed to practical innovation for hospitalists, starting in 2019, JHM will offer focused yet informative content that places important advances into relevant clinical or methodological context and provides our readers with information that is accessible, meaningful, and actionable—all in a more concise format.

Our new Clinical Guideline Highlights for the Hospitalist is a brief, targeted review of recently published clinical guidelines, distilling the major recommendations relevant to hospital medicine and placing them in context of the available evidence. This review format also offers a critique of gaps in the literature and notes areas ripe for future study. In this issue, we debut two articles using this new approach—one aimed at adult hospitalists and the other at pediatric hospitalists—regarding recently published studies and guidelines about maintenance intravenous fluids.^1-5

In 2019, we will also introduce a second new format, called Progress Notes. These reviews will be shorter than JHM’s traditional review format, and will accept two types of articles: clinical and methodological. The clinical Progress Notes will provide an update on the last several years of evidence related to diagnosis, treatment, risk stratification, and/or prevention of a clinical problem highly pertinent to hospitalists. The methodological Progress Notes will provide our readers with insight into the application of quantitative, qualitative, and quality improvement methods commonly used in work published in this journal. Our aim is to use Progress Notes as a way to enhance both clinical practice and scholarship efforts by our readers.

Finally, we will introduce “Hospital Medicine: The Year in Review,” an annual feature that concisely compiles and critiques the top articles in both adult and pediatric hospital medicine over the past year. The “Year in Review” will serve as a written corollary to the popular “Updates in Hospital Medicine” presentation at the Society of Hospital Medicine annual meeting, and will highlight important research that advanced our field or provided us a fresh perspective on hospitalist practice.

As we introduce these new review formats, it is important to note that JHM will continue to accept traditional, long-form reviews on any topic relevant to hospitalists, with a preference for rigorous systematic reviews or meta-analyses. Equally important is that JHM’s overarching commitment remains unchanged: support clinicians, leaders, and scholars in our field in their pursuit of delivering evidence-based, high-value clinical care. We hope you enjoy these new article formats and we look forward to your feedback.

Disclosures

The authors declare they have no conflicts of interest/competing interests.

As part of the Journal of Hospital Medicine’s^® commitment to our readership, we are excited to announce innovative new review formats, designed for busy hospitalists. The state of knowledge in our field is changing rapidly, and the 21^st century poses a conundrum to clinicians in the form of increasingly complex studies and guidelines amidst ever-decreasing time to digest them. As a result, it can be challenging for hospitalists to access and interpret recently published research to inform their clinical practice. Because we are committed to practical innovation for hospitalists, starting in 2019, JHM will offer focused yet informative content that places important advances into relevant clinical or methodological context and provides our readers with information that is accessible, meaningful, and actionable—all in a more concise format.

Our new Clinical Guideline Highlights for the Hospitalist is a brief, targeted review of recently published clinical guidelines, distilling the major recommendations relevant to hospital medicine and placing them in context of the available evidence. This review format also offers a critique of gaps in the literature and notes areas ripe for future study. In this issue, we debut two articles using this new approach—one aimed at adult hospitalists and the other at pediatric hospitalists—regarding recently published studies and guidelines about maintenance intravenous fluids.^1-5

In 2019, we will also introduce a second new format, called Progress Notes. These reviews will be shorter than JHM’s traditional review format, and will accept two types of articles: clinical and methodological. The clinical Progress Notes will provide an update on the last several years of evidence related to diagnosis, treatment, risk stratification, and/or prevention of a clinical problem highly pertinent to hospitalists. The methodological Progress Notes will provide our readers with insight into the application of quantitative, qualitative, and quality improvement methods commonly used in work published in this journal. Our aim is to use Progress Notes as a way to enhance both clinical practice and scholarship efforts by our readers.

Finally, we will introduce “Hospital Medicine: The Year in Review,” an annual feature that concisely compiles and critiques the top articles in both adult and pediatric hospital medicine over the past year. The “Year in Review” will serve as a written corollary to the popular “Updates in Hospital Medicine” presentation at the Society of Hospital Medicine annual meeting, and will highlight important research that advanced our field or provided us a fresh perspective on hospitalist practice.

As we introduce these new review formats, it is important to note that JHM will continue to accept traditional, long-form reviews on any topic relevant to hospitalists, with a preference for rigorous systematic reviews or meta-analyses. Equally important is that JHM’s overarching commitment remains unchanged: support clinicians, leaders, and scholars in our field in their pursuit of delivering evidence-based, high-value clinical care. We hope you enjoy these new article formats and we look forward to your feedback.

Disclosures

The authors declare they have no conflicts of interest/competing interests.

As part of the Journal of Hospital Medicine’s^® commitment to our readership, we are excited to announce innovative new review formats, designed for busy hospitalists. The state of knowledge in our field is changing rapidly, and the 21^st century poses a conundrum to clinicians in the form of increasingly complex studies and guidelines amidst ever-decreasing time to digest them. As a result, it can be challenging for hospitalists to access and interpret recently published research to inform their clinical practice. Because we are committed to practical innovation for hospitalists, starting in 2019, JHM will offer focused yet informative content that places important advances into relevant clinical or methodological context and provides our readers with information that is accessible, meaningful, and actionable—all in a more concise format.

Our new Clinical Guideline Highlights for the Hospitalist is a brief, targeted review of recently published clinical guidelines, distilling the major recommendations relevant to hospital medicine and placing them in context of the available evidence. This review format also offers a critique of gaps in the literature and notes areas ripe for future study. In this issue, we debut two articles using this new approach—one aimed at adult hospitalists and the other at pediatric hospitalists—regarding recently published studies and guidelines about maintenance intravenous fluids.^1-5

In 2019, we will also introduce a second new format, called Progress Notes. These reviews will be shorter than JHM’s traditional review format, and will accept two types of articles: clinical and methodological. The clinical Progress Notes will provide an update on the last several years of evidence related to diagnosis, treatment, risk stratification, and/or prevention of a clinical problem highly pertinent to hospitalists. The methodological Progress Notes will provide our readers with insight into the application of quantitative, qualitative, and quality improvement methods commonly used in work published in this journal. Our aim is to use Progress Notes as a way to enhance both clinical practice and scholarship efforts by our readers.

Finally, we will introduce “Hospital Medicine: The Year in Review,” an annual feature that concisely compiles and critiques the top articles in both adult and pediatric hospital medicine over the past year. The “Year in Review” will serve as a written corollary to the popular “Updates in Hospital Medicine” presentation at the Society of Hospital Medicine annual meeting, and will highlight important research that advanced our field or provided us a fresh perspective on hospitalist practice.

As we introduce these new review formats, it is important to note that JHM will continue to accept traditional, long-form reviews on any topic relevant to hospitalists, with a preference for rigorous systematic reviews or meta-analyses. Equally important is that JHM’s overarching commitment remains unchanged: support clinicians, leaders, and scholars in our field in their pursuit of delivering evidence-based, high-value clinical care. We hope you enjoy these new article formats and we look forward to your feedback.

Disclosures

The authors declare they have no conflicts of interest/competing interests.