Justin Iannello, DO, MBA

In 1997, Elizabeth McGlynn wrote, “Measuring quality is no simple task.”¹ We are reminded of this seminal Health Affairs article at a very pertinent point—as health care practice progresses, measuring the impact of performance improvement initiatives on clinical care delivery remains integral to monitoring overall effectiveness of quality. Mortality outcomes are a major focus of quality.

Inpatient mortality within the Veterans Health Administration (VHA) was measured as actual number of deaths (unadjusted mortality), and adjusted mortality was calculated using the standardized mortality ratio (SMR). SMR included actual number of deaths during hospitalization or within 1 day of hospital discharge divided by predicted number of deaths using a risk-adjusted formula and was calculated separately for acute level of care (LOC) and the intensive care unit (ICU). Using risk-adjusted SMR, if an observed/expected ratio was > 1.0, there were more inpatient deaths than expected; if < 1.0, fewer inpatient deaths occurred than predicted; and if 1.0, observed number of inpatient deaths was equivalent to expected number of deaths.²

Mortality reduction is a complex area of performance improvement. Health care facilities often focus their efforts on the biggest mortality contributors. According to Dantes and Epstein, sepsis results in about 265,000 deaths annually in the United States.³ Reinhart and colleagues demonstrated that sepsis is a worldwide issue resulting in approximately 30 million cases and 6 million deaths annually.⁴ Furthermore, Kumar and colleagues have noted that when sepsis progresses to septic shock, survival decreases by almost 8% for each hour delay in sepsis identification and treatment.⁵

Improvements in sepsis management have been multifaceted. The Surviving Sepsis Campaign guidelines created sepsis treatment bundles to guide early diagnosis/treatment of sepsis.⁶ In addition to awareness and sepsis care bundles, a plethora of informatics solutions within electronic health record (EHR) systems have demonstrated improved sepsis care.^7-16 Various approaches to early diagnosis and management of sepsis have been collectively referred to as an early warning sepsis system (EWSS).

An EWSS typically contains automated decision support tools that are integrated in the EHR and meant to assist health care professionals with clinical workflow decision-making. Automated decision support tools within the EHR have a variety of functions, such as clinical care reminders and alerts.¹⁷

Sepsis screening tools function as a form of automated decision support and may be incorporated into the EHR to support the EWSS. Although sepsis screening tools vary, they frequently include a combination of data involving vital signs, laboratory values and/or physical examination findings, such as mental status evaluation.The Modified Early Warning Signs (MEWS) + Sepsis Recognition Score (SRS) is one example of a sepsis screening tool.^7,16

At Malcom Randall Veterans Affairs Medical Center (MRVAMC) in Gainesville, Florida, we identified a quality improvement project opportunity to improve sepsis care in the emergency department (ED) and inpatient wards using the VHA EHR system, the Computerized Patient Record System (CPRS), which is supported by the Veterans Information Systems and Technology Architecture (VistA).¹⁸ A VistA/CPRS EWSS was developed using Lean Six Sigma DMAIC (define, measure, analyze, improve, and control) methodology.¹⁹ During the improve stage, informatics solutions were applied and included a combination of EHR interventions, such as template design, an order set, and clinical reminders. Clinical reminders have a wide variety of use, such as reminders for clinical tasks and as automated decision support within clinical workflows using Boolean logic.

To the best of our knowledge, there has been no published application of an EWSS within VistA/CPRS. In this study, we outline the strategic development of an EWSS in VistA/CPRS that assisted clinical staff with identification and treatment of sepsis; improved documentation of sepsis when present; and associated with improvement in unadjusted and adjusted inpatient mortality.

Methods 

According to policy activities that constitute research at MRVAMC, no institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

The North Florida/South Georgia Veterans Health System (NF/SGVHS) includes MRVAMC, a large academic hospital with rotating residents/fellows and multiple specialty care services. MRVAMC comprised 144 beds on the medicine/surgery wards; 48 beds in the psychiatry unit; 18 intermediate LOC beds; and 27 ICU beds. The MRVAMC SMR was identified as an improvement opportunity during fiscal year (FY) 2017 (Table 1). Its adjusted mortality for acute LOC demonstrated an observed/expected ratio of > 1.0 suggesting more inpatient deaths were observed than expected. The number of deaths (unadjusted mortality) on acute LOC at MRVAMC was noted to be rising during the first 3 quarters of FY 2017. A deeper examination of data by Pyramid Analytics (www.pyramidanalytics.com) discovered that sepsis was the primary driver for inpatient mortality on acute LOC at MRVAMC. Our goal was to reduce inpatient sepsis-related mortality via development of an EWSS that leveraged VistA/CPRS to improve early identification and treatment of sepsis in the ED and inpatient wards.

Emergency Department

Given the importance of recognizing sepsis early, the sepsis team focused on improvement opportunities at the initial point of patient contact: ED triage. The goal was to incorporate automated VistA/CPRS decision support to assist clinicians with identifying sepsis in triage using MEWS, which was chosen to optimize immediate hospital-wide buy-in. Clinical staff were already familiar with MEWS, which was in use on the inpatient wards.

Flow through the ED and availability of resources differed from the wards. Hence, modification to MEWS on the wards was necessary to fit clinical workflow in the ED. Temperature, heart rate (HR), respiratory rate (RR), systolic blood pressure (SBP), mental status, and white blood cell count (WBC) factored into a MEWS + SRS score on the wards (Table 2). For the ED, MEWS included temperature, HR, RR and SBP, but excluded mental status and WBC. Mental status assessment was excluded due to technical infeasibility (while vital signs could be automatically calculated in real time for a MEWS score, that was not possible for mental status changes). WBC was excluded from the ED as laboratory test results would not be available in triage.

MEWS + SRS scores were calculated in VistA by using clinical reminders. Clinical reminder logic included a series of conditional statements based on various combinations of MEWS + SRS clinical data entered in the EHR. When ED triage vital signs data were entered in CPRS, clinical data were stored and processed according to clinical reminder logic in VistA and displayed to the user in CPRS. While MEWS of ≥ 5 triggered a sepsis alert on the wards, the ≥ 4 threshold was used in the ED given mental status and WBC were excluded from calculations in triage (eAppendix 1 available at doi:10.12788/fp.0194).

Once a sepsis alert was triggered in triage for MEWS ≥ 4, ED nursing staff prioritized bed location and expedited staffing with an ED attending physician for early assessment. The ED attending then performed an assessment to confirm whether sepsis was present and direct early treatment. Although every patient who triggered a sepsis alert in triage did not meet clinical findings of sepsis, patients with MEWS ≥ 4 were frequently ill and required timely intervention.

If an ED attending physician agreed with a sepsis diagnosis, the physician had access to a sepsis workup and treatment order set in CPRS (eAppendix 2 available at doi:10.12788/fp.0194). The sepsis order set incorporated recommendations from the Surviving Sepsis Campaign guidelines and included orders for 2 large-bore peripheral IV lines; aggressive fluid resuscitation (30 mL/kg) for patients with clinical findings of hypoperfusion; broad-spectrum antibiotics; and frequent ordering of laboratory tests and imaging during initial sepsis workup.⁶ Vancomycin and cefepime were selected as routine broad-spectrum antibiotics in the order set when sepsis was suspected based on local antimicrobial stewardship and safety-efficacy profiles. For example, Luther and colleagues demonstrated that cefepime has lower rates of acute kidney injury when combined with vancomycin vs vancomycin + piperacillin-tazobactam.²⁰ If a β-lactam antibiotic could not be used due to a patient’s drug allergy history, aztreonam was available as an alternative option.

The design of the order set also functioned as a communication interface with clinical pharmacists. Given the large volume of antibiotics ordered in the ED, it was difficult for pharmacists to prioritize antibiotic order verification. While stat orders convey high priority, they often lack specificity. When antibiotic orders were selected from the sepsis order set, comments were already included that stated: “STAT. First dose for sepsis protocol” (eAppendix 3 available at doi:10.12788/fp.0194). This standardized communication conveyed a sense of urgency and a collective understanding that patients with suspected sepsis required timely order verification and administration of antibiotics.

Hospital Ward

Mental status and WBC were included on the wards to monitor for possible signs of sepsis, using MEWS + SRS, which was routinely monitored by nursing every 4 to 8 hours. When MEWS + SRS was ≥ 5 points, ward nursing staff called a sepsis alert.^7,16 Early response team (ERT) members received telephone notifications of the alert. ERT staff proceeded with immediate evaluation and treatment at the bedside along with determination for most appropriate LOC. The ERT members included an ICU physician and nurse; respiratory therapist; and nursing supervisor/bed flow coordinator. During bedside evaluation, if the ERT or primary team agreed with a sepsis diagnosis, the ERT or primary team used the sepsis order set to ensure standardized procedures. Stat orders generated through the sepsis order set pathway conveyed a sense of urgency and need for immediate order verification and administration of antibiotics.

In addition to clinical care process improvement, accurate documentation also was emphasized in the EWSS. When a sepsis alert was called, a clinician from the primary team was expected to complete a standardized progress note, which communicated clinical findings, a treatment plan, and captured severity of illness (eAppendix 4 available at doi:10.12788/fp.0194). It included sections for subjective, objective, assessment, and plan. In addition, data objects were created for vital signs and common laboratory findings that retrieved important clinical data from VistA and inserted it into the CPRS note.²¹

Nursing staff on the wards were expected to communicate results with the primary team for clinical decision making when a patient had a MEWS + SRS of 3 to 4. A sepsis alert may have been called at the discretion of clinical team members but was not required if the score was < 5. Additionally, vital signs were expected to be checked by the nursing staff on the wards at least every 4 hours for closer monitoring.

Sepsis Review Meetings

Weekly meetings were scheduled to review sepsis cases to assess diagnosis, treatment, and documentation entered in the patient record. The team conducting sepsis reviews comprised the chief of staff, chief of quality management, director of patient safety, physician utilization management advisor, chief resident in quality and patient safety (CRQS), and inpatient pharmacy supervisor. In addition, ad hoc physicians and nurses from different specialty areas, such as infectious diseases, hospitalist section, ICU, and the ED participated on request for subject matter expertise when needed. At the conclusion of weekly sepsis meetings, sepsis team members provided feedback to the clinical staff for continuous improvement purposes.

Results

Before implementation of an EWSS at NF/SGVHS, a plan was devised to increase awareness and educate staff on sepsis-related mortality in late FY 2017. Awareness and education about sepsis-related mortality was organized at physician, nursing, and pharmacy leadership clinical staff meetings. Posters about early warning signs of sepsis also were displayed on the nursing units for educational purposes and to convey the importance of early recognition/treatment of sepsis. In addition, the CRQS was the quality leader for house staff and led sepsis campaign change efforts for residents/fellows. An immediate improvement in unadjusted mortality at MRVAMC was noted with initial sepsis awareness and education. From FY 2017, quarter 3 to FY 2018, quarter 1, the number of acute LOC inpatient deaths decreased from 48 to 28, a 42% reduction in unadjusted mortality at MRVAMC (Figure 1). Additionally, the acute LOC SMR improved from 1.20 during FY 2017, quarter 3 down to as low as 0.71 during FY 2018, quarter 1 (Figure 2).

The number of MRVAMC inpatient deaths increased from 28 in FY 2018, quarter 1 to 45 in FY 2018, quarter 3. While acute LOC showed improvement in unadjusted mortality after sepsis education/awareness, it was felt continuous improvement could not be sustained with education alone. An EWSS was designed and implemented within the EHR system in FY 2018. Following implementation of EWSS and reeducating staff on early recognition and treatment of sepsis, acute LOC inpatient deaths decreased from 45 in FY 2018, quarter 3 through FY 2019 where unadjusted mortality was as low as 27 during FY 2019, quarter 4. The MRVAMC acute LOC SMR was consistently < 1.0 from FY 2018, quarter 4 through FY 2019, quarter 4.

In addition to the observed decrease in acute LOC inpatient deaths and improved SMR, the number of ERT alerts and sepsis alerts on the inpatient wards were monitored from FY 2017 through FY 2019. ERT alerts listed in Table 3 were nonspecific and initiated by nursing staff on the wards where a patient’s clinical status was identified as worsening while sepsis alerts were specific ERT alerts called by the ward nursing staff due to concerns for sepsis. The inpatient wards included inpatient medicine, surgery, and psychiatry acute care and the intermediate level of care unit while outpatient clinical areas of treatment, intensive care units, stroke alerts, and STEMI alerts were excluded.

Discussion

Implementation of the EWSS was associated with improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC. Although variation exists with application of EWSS at other medical centers, there was similarity with improved sepsis outcomes reported at other health care systems after EWSS implementation.^7-16

Improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC was likely due to multiple contributing factors. First, during design and implementation of the EWSS, project work was interdisciplinary with input from physicians, nurses, and pharmacists from multiple specialties (ie, ED, ICU, and the medicine service); quality management and data analysis specialists; and clinical informatics. Second, facility commitment to improving early recognition and treatment of sepsis from leadership level down to front-line staff was evident. Weekly sepsis meetings with the NF/SGVHS chief of staff helped to sustain EWSS efforts and to identify additional improvement opportunities. Third, integrated informatics solutions within the EHR helped identify early sepsis and minimized human error as well as assisted with coordination of sepsis care across services. Fourth, the focus was on both early identification and treatment of sepsis in the ED and hospital wards. Although it cannot be deduced whether there was causation between reduced inpatient mortality and an increased number of nonspecific ERT alerts+ sepsis alerts on the inpatient wards after EWSS implementation, inpatient deaths decreased and SMR improved. Finally, the EWSS emphasized both the importance of evidence-based clinical care of sepsis and standardized documentation to appropriately capture clinical severity of illness.

Limitations

This program has limitations. The EWSS was studied at a single VHA facility. Veteran demographics and local epidemiology may limit conclusion of outcomes to an individual VHA facility located in a specific geographical region. Additional research is necessary to demonstrate reproducibility and determine whether applicable to other VHA facilities and community care settings.

SMR is a risk-adjusted formula developed by the VHA Inpatient Evaluation Center, which included numerous factors such as diagnosis, comorbid conditions, age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, ICU stays, immunosuppressive status, and a COVID-19 positive indicator (added after this study). Further research is needed to evaluate sepsis-related outcomes using the EWSS during the COVID-19 pandemic.

EWSS in the literature have demonstrated various approaches to early identification and treatment of sepsis and have used different sepsis screening tools.²² Evidence suggests that the MEWS + SRS sepsis screening tool may result in false-positive screenings.^23-27 Additional research into the specificity of this sepsis screening tool is needed. Ward nursing staff were encouraged to initiate automatic sepsis alerts when MEWS + SRS was ≥ 5; however, this still depended on human factors. Because sepsis alerts are software-specific and others were incompatible with the VHA EHR, it was necessary to design our own EWSS.

Despite improvement with MRVAMC acute LOC unadjusted and adjusted mortality with our EWSS, we did not identify any actual improvement in earlier antibiotic administration times once sepsis was recognized. While accurate documentation regarding degree of sepsis improved, a MRVAMC clinical documentation improvement program was expanded in FY 2018. Therefore, it is difficult to demonstrate causation related to improved sepsis documentation with template changes alone. While sepsis alerts on the inpatient wards were variable since EWSS implementation, nonspecific ERT alerts increased. It is unclear whether some sepsis alerts were called as nonspecific ERT alerts, making it impossible to know the true number of sepsis alerts.

MRVAMC experienced an increase in nurse turnover during FY 2018 and as a teaching hospital had frequent rotating residents and fellows new to processes/protocols. These factors may have contributed to variations in unadjusted mortality. Also the decrease in inpatient mortality and improvement in SMR on acute LOC could have been the result of factors other than the EWSS and the effect of education alone may have been at least as good as that of the EWSS intervention.

Conclusions

Education along with the possible implementation of an EWSS at NF/SGVHS was associated with a decrease in the number of inpatient deaths on MRVAMC’s acute LOC wards from as high as 48 in FY 2017, quarter 3 to as low as 27 in FY 2019, quarter 4 resulting in as large of an improvement as a 44% reduction in unadjusted mortality from FY 2017 to FY 2019. In addition, MRVAMC’s acute LOC SMR improved from > 1.0 to < 1.0, demonstrating fewer inpatient mortalities than predicted from FY 2017 to FY 2019.

This multifaceted interventional strategy may be effectively applied at other VHA health care facilities that use the same EHR system. Next steps may include determining the specificity of MEWS + SRS as a sepsis screening tool; studying outcomes of MRVAMC’s EWSS during the COVID-19 era; and conducting a multicentered study on this EWSS across multiple VHA facilities.

In 1997, Elizabeth McGlynn wrote, “Measuring quality is no simple task.”¹ We are reminded of this seminal Health Affairs article at a very pertinent point—as health care practice progresses, measuring the impact of performance improvement initiatives on clinical care delivery remains integral to monitoring overall effectiveness of quality. Mortality outcomes are a major focus of quality.

Inpatient mortality within the Veterans Health Administration (VHA) was measured as actual number of deaths (unadjusted mortality), and adjusted mortality was calculated using the standardized mortality ratio (SMR). SMR included actual number of deaths during hospitalization or within 1 day of hospital discharge divided by predicted number of deaths using a risk-adjusted formula and was calculated separately for acute level of care (LOC) and the intensive care unit (ICU). Using risk-adjusted SMR, if an observed/expected ratio was > 1.0, there were more inpatient deaths than expected; if < 1.0, fewer inpatient deaths occurred than predicted; and if 1.0, observed number of inpatient deaths was equivalent to expected number of deaths.²

Mortality reduction is a complex area of performance improvement. Health care facilities often focus their efforts on the biggest mortality contributors. According to Dantes and Epstein, sepsis results in about 265,000 deaths annually in the United States.³ Reinhart and colleagues demonstrated that sepsis is a worldwide issue resulting in approximately 30 million cases and 6 million deaths annually.⁴ Furthermore, Kumar and colleagues have noted that when sepsis progresses to septic shock, survival decreases by almost 8% for each hour delay in sepsis identification and treatment.⁵

Improvements in sepsis management have been multifaceted. The Surviving Sepsis Campaign guidelines created sepsis treatment bundles to guide early diagnosis/treatment of sepsis.⁶ In addition to awareness and sepsis care bundles, a plethora of informatics solutions within electronic health record (EHR) systems have demonstrated improved sepsis care.^7-16 Various approaches to early diagnosis and management of sepsis have been collectively referred to as an early warning sepsis system (EWSS).

An EWSS typically contains automated decision support tools that are integrated in the EHR and meant to assist health care professionals with clinical workflow decision-making. Automated decision support tools within the EHR have a variety of functions, such as clinical care reminders and alerts.¹⁷

Sepsis screening tools function as a form of automated decision support and may be incorporated into the EHR to support the EWSS. Although sepsis screening tools vary, they frequently include a combination of data involving vital signs, laboratory values and/or physical examination findings, such as mental status evaluation.The Modified Early Warning Signs (MEWS) + Sepsis Recognition Score (SRS) is one example of a sepsis screening tool.^7,16

At Malcom Randall Veterans Affairs Medical Center (MRVAMC) in Gainesville, Florida, we identified a quality improvement project opportunity to improve sepsis care in the emergency department (ED) and inpatient wards using the VHA EHR system, the Computerized Patient Record System (CPRS), which is supported by the Veterans Information Systems and Technology Architecture (VistA).¹⁸ A VistA/CPRS EWSS was developed using Lean Six Sigma DMAIC (define, measure, analyze, improve, and control) methodology.¹⁹ During the improve stage, informatics solutions were applied and included a combination of EHR interventions, such as template design, an order set, and clinical reminders. Clinical reminders have a wide variety of use, such as reminders for clinical tasks and as automated decision support within clinical workflows using Boolean logic.

To the best of our knowledge, there has been no published application of an EWSS within VistA/CPRS. In this study, we outline the strategic development of an EWSS in VistA/CPRS that assisted clinical staff with identification and treatment of sepsis; improved documentation of sepsis when present; and associated with improvement in unadjusted and adjusted inpatient mortality.

Methods 

According to policy activities that constitute research at MRVAMC, no institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

The North Florida/South Georgia Veterans Health System (NF/SGVHS) includes MRVAMC, a large academic hospital with rotating residents/fellows and multiple specialty care services. MRVAMC comprised 144 beds on the medicine/surgery wards; 48 beds in the psychiatry unit; 18 intermediate LOC beds; and 27 ICU beds. The MRVAMC SMR was identified as an improvement opportunity during fiscal year (FY) 2017 (Table 1). Its adjusted mortality for acute LOC demonstrated an observed/expected ratio of > 1.0 suggesting more inpatient deaths were observed than expected. The number of deaths (unadjusted mortality) on acute LOC at MRVAMC was noted to be rising during the first 3 quarters of FY 2017. A deeper examination of data by Pyramid Analytics (www.pyramidanalytics.com) discovered that sepsis was the primary driver for inpatient mortality on acute LOC at MRVAMC. Our goal was to reduce inpatient sepsis-related mortality via development of an EWSS that leveraged VistA/CPRS to improve early identification and treatment of sepsis in the ED and inpatient wards.

Emergency Department

Given the importance of recognizing sepsis early, the sepsis team focused on improvement opportunities at the initial point of patient contact: ED triage. The goal was to incorporate automated VistA/CPRS decision support to assist clinicians with identifying sepsis in triage using MEWS, which was chosen to optimize immediate hospital-wide buy-in. Clinical staff were already familiar with MEWS, which was in use on the inpatient wards.

Flow through the ED and availability of resources differed from the wards. Hence, modification to MEWS on the wards was necessary to fit clinical workflow in the ED. Temperature, heart rate (HR), respiratory rate (RR), systolic blood pressure (SBP), mental status, and white blood cell count (WBC) factored into a MEWS + SRS score on the wards (Table 2). For the ED, MEWS included temperature, HR, RR and SBP, but excluded mental status and WBC. Mental status assessment was excluded due to technical infeasibility (while vital signs could be automatically calculated in real time for a MEWS score, that was not possible for mental status changes). WBC was excluded from the ED as laboratory test results would not be available in triage.

MEWS + SRS scores were calculated in VistA by using clinical reminders. Clinical reminder logic included a series of conditional statements based on various combinations of MEWS + SRS clinical data entered in the EHR. When ED triage vital signs data were entered in CPRS, clinical data were stored and processed according to clinical reminder logic in VistA and displayed to the user in CPRS. While MEWS of ≥ 5 triggered a sepsis alert on the wards, the ≥ 4 threshold was used in the ED given mental status and WBC were excluded from calculations in triage (eAppendix 1 available at doi:10.12788/fp.0194).

Once a sepsis alert was triggered in triage for MEWS ≥ 4, ED nursing staff prioritized bed location and expedited staffing with an ED attending physician for early assessment. The ED attending then performed an assessment to confirm whether sepsis was present and direct early treatment. Although every patient who triggered a sepsis alert in triage did not meet clinical findings of sepsis, patients with MEWS ≥ 4 were frequently ill and required timely intervention.

If an ED attending physician agreed with a sepsis diagnosis, the physician had access to a sepsis workup and treatment order set in CPRS (eAppendix 2 available at doi:10.12788/fp.0194). The sepsis order set incorporated recommendations from the Surviving Sepsis Campaign guidelines and included orders for 2 large-bore peripheral IV lines; aggressive fluid resuscitation (30 mL/kg) for patients with clinical findings of hypoperfusion; broad-spectrum antibiotics; and frequent ordering of laboratory tests and imaging during initial sepsis workup.⁶ Vancomycin and cefepime were selected as routine broad-spectrum antibiotics in the order set when sepsis was suspected based on local antimicrobial stewardship and safety-efficacy profiles. For example, Luther and colleagues demonstrated that cefepime has lower rates of acute kidney injury when combined with vancomycin vs vancomycin + piperacillin-tazobactam.²⁰ If a β-lactam antibiotic could not be used due to a patient’s drug allergy history, aztreonam was available as an alternative option.

The design of the order set also functioned as a communication interface with clinical pharmacists. Given the large volume of antibiotics ordered in the ED, it was difficult for pharmacists to prioritize antibiotic order verification. While stat orders convey high priority, they often lack specificity. When antibiotic orders were selected from the sepsis order set, comments were already included that stated: “STAT. First dose for sepsis protocol” (eAppendix 3 available at doi:10.12788/fp.0194). This standardized communication conveyed a sense of urgency and a collective understanding that patients with suspected sepsis required timely order verification and administration of antibiotics.

Hospital Ward

Mental status and WBC were included on the wards to monitor for possible signs of sepsis, using MEWS + SRS, which was routinely monitored by nursing every 4 to 8 hours. When MEWS + SRS was ≥ 5 points, ward nursing staff called a sepsis alert.^7,16 Early response team (ERT) members received telephone notifications of the alert. ERT staff proceeded with immediate evaluation and treatment at the bedside along with determination for most appropriate LOC. The ERT members included an ICU physician and nurse; respiratory therapist; and nursing supervisor/bed flow coordinator. During bedside evaluation, if the ERT or primary team agreed with a sepsis diagnosis, the ERT or primary team used the sepsis order set to ensure standardized procedures. Stat orders generated through the sepsis order set pathway conveyed a sense of urgency and need for immediate order verification and administration of antibiotics.

In addition to clinical care process improvement, accurate documentation also was emphasized in the EWSS. When a sepsis alert was called, a clinician from the primary team was expected to complete a standardized progress note, which communicated clinical findings, a treatment plan, and captured severity of illness (eAppendix 4 available at doi:10.12788/fp.0194). It included sections for subjective, objective, assessment, and plan. In addition, data objects were created for vital signs and common laboratory findings that retrieved important clinical data from VistA and inserted it into the CPRS note.²¹

Nursing staff on the wards were expected to communicate results with the primary team for clinical decision making when a patient had a MEWS + SRS of 3 to 4. A sepsis alert may have been called at the discretion of clinical team members but was not required if the score was < 5. Additionally, vital signs were expected to be checked by the nursing staff on the wards at least every 4 hours for closer monitoring.

Sepsis Review Meetings

Weekly meetings were scheduled to review sepsis cases to assess diagnosis, treatment, and documentation entered in the patient record. The team conducting sepsis reviews comprised the chief of staff, chief of quality management, director of patient safety, physician utilization management advisor, chief resident in quality and patient safety (CRQS), and inpatient pharmacy supervisor. In addition, ad hoc physicians and nurses from different specialty areas, such as infectious diseases, hospitalist section, ICU, and the ED participated on request for subject matter expertise when needed. At the conclusion of weekly sepsis meetings, sepsis team members provided feedback to the clinical staff for continuous improvement purposes.

Results

Before implementation of an EWSS at NF/SGVHS, a plan was devised to increase awareness and educate staff on sepsis-related mortality in late FY 2017. Awareness and education about sepsis-related mortality was organized at physician, nursing, and pharmacy leadership clinical staff meetings. Posters about early warning signs of sepsis also were displayed on the nursing units for educational purposes and to convey the importance of early recognition/treatment of sepsis. In addition, the CRQS was the quality leader for house staff and led sepsis campaign change efforts for residents/fellows. An immediate improvement in unadjusted mortality at MRVAMC was noted with initial sepsis awareness and education. From FY 2017, quarter 3 to FY 2018, quarter 1, the number of acute LOC inpatient deaths decreased from 48 to 28, a 42% reduction in unadjusted mortality at MRVAMC (Figure 1). Additionally, the acute LOC SMR improved from 1.20 during FY 2017, quarter 3 down to as low as 0.71 during FY 2018, quarter 1 (Figure 2).

The number of MRVAMC inpatient deaths increased from 28 in FY 2018, quarter 1 to 45 in FY 2018, quarter 3. While acute LOC showed improvement in unadjusted mortality after sepsis education/awareness, it was felt continuous improvement could not be sustained with education alone. An EWSS was designed and implemented within the EHR system in FY 2018. Following implementation of EWSS and reeducating staff on early recognition and treatment of sepsis, acute LOC inpatient deaths decreased from 45 in FY 2018, quarter 3 through FY 2019 where unadjusted mortality was as low as 27 during FY 2019, quarter 4. The MRVAMC acute LOC SMR was consistently < 1.0 from FY 2018, quarter 4 through FY 2019, quarter 4.

In addition to the observed decrease in acute LOC inpatient deaths and improved SMR, the number of ERT alerts and sepsis alerts on the inpatient wards were monitored from FY 2017 through FY 2019. ERT alerts listed in Table 3 were nonspecific and initiated by nursing staff on the wards where a patient’s clinical status was identified as worsening while sepsis alerts were specific ERT alerts called by the ward nursing staff due to concerns for sepsis. The inpatient wards included inpatient medicine, surgery, and psychiatry acute care and the intermediate level of care unit while outpatient clinical areas of treatment, intensive care units, stroke alerts, and STEMI alerts were excluded.

Discussion

Implementation of the EWSS was associated with improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC. Although variation exists with application of EWSS at other medical centers, there was similarity with improved sepsis outcomes reported at other health care systems after EWSS implementation.^7-16

Improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC was likely due to multiple contributing factors. First, during design and implementation of the EWSS, project work was interdisciplinary with input from physicians, nurses, and pharmacists from multiple specialties (ie, ED, ICU, and the medicine service); quality management and data analysis specialists; and clinical informatics. Second, facility commitment to improving early recognition and treatment of sepsis from leadership level down to front-line staff was evident. Weekly sepsis meetings with the NF/SGVHS chief of staff helped to sustain EWSS efforts and to identify additional improvement opportunities. Third, integrated informatics solutions within the EHR helped identify early sepsis and minimized human error as well as assisted with coordination of sepsis care across services. Fourth, the focus was on both early identification and treatment of sepsis in the ED and hospital wards. Although it cannot be deduced whether there was causation between reduced inpatient mortality and an increased number of nonspecific ERT alerts+ sepsis alerts on the inpatient wards after EWSS implementation, inpatient deaths decreased and SMR improved. Finally, the EWSS emphasized both the importance of evidence-based clinical care of sepsis and standardized documentation to appropriately capture clinical severity of illness.

Limitations

This program has limitations. The EWSS was studied at a single VHA facility. Veteran demographics and local epidemiology may limit conclusion of outcomes to an individual VHA facility located in a specific geographical region. Additional research is necessary to demonstrate reproducibility and determine whether applicable to other VHA facilities and community care settings.

SMR is a risk-adjusted formula developed by the VHA Inpatient Evaluation Center, which included numerous factors such as diagnosis, comorbid conditions, age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, ICU stays, immunosuppressive status, and a COVID-19 positive indicator (added after this study). Further research is needed to evaluate sepsis-related outcomes using the EWSS during the COVID-19 pandemic.

EWSS in the literature have demonstrated various approaches to early identification and treatment of sepsis and have used different sepsis screening tools.²² Evidence suggests that the MEWS + SRS sepsis screening tool may result in false-positive screenings.^23-27 Additional research into the specificity of this sepsis screening tool is needed. Ward nursing staff were encouraged to initiate automatic sepsis alerts when MEWS + SRS was ≥ 5; however, this still depended on human factors. Because sepsis alerts are software-specific and others were incompatible with the VHA EHR, it was necessary to design our own EWSS.

Despite improvement with MRVAMC acute LOC unadjusted and adjusted mortality with our EWSS, we did not identify any actual improvement in earlier antibiotic administration times once sepsis was recognized. While accurate documentation regarding degree of sepsis improved, a MRVAMC clinical documentation improvement program was expanded in FY 2018. Therefore, it is difficult to demonstrate causation related to improved sepsis documentation with template changes alone. While sepsis alerts on the inpatient wards were variable since EWSS implementation, nonspecific ERT alerts increased. It is unclear whether some sepsis alerts were called as nonspecific ERT alerts, making it impossible to know the true number of sepsis alerts.

MRVAMC experienced an increase in nurse turnover during FY 2018 and as a teaching hospital had frequent rotating residents and fellows new to processes/protocols. These factors may have contributed to variations in unadjusted mortality. Also the decrease in inpatient mortality and improvement in SMR on acute LOC could have been the result of factors other than the EWSS and the effect of education alone may have been at least as good as that of the EWSS intervention.

Conclusions

Education along with the possible implementation of an EWSS at NF/SGVHS was associated with a decrease in the number of inpatient deaths on MRVAMC’s acute LOC wards from as high as 48 in FY 2017, quarter 3 to as low as 27 in FY 2019, quarter 4 resulting in as large of an improvement as a 44% reduction in unadjusted mortality from FY 2017 to FY 2019. In addition, MRVAMC’s acute LOC SMR improved from > 1.0 to < 1.0, demonstrating fewer inpatient mortalities than predicted from FY 2017 to FY 2019.

This multifaceted interventional strategy may be effectively applied at other VHA health care facilities that use the same EHR system. Next steps may include determining the specificity of MEWS + SRS as a sepsis screening tool; studying outcomes of MRVAMC’s EWSS during the COVID-19 era; and conducting a multicentered study on this EWSS across multiple VHA facilities.

In 1997, Elizabeth McGlynn wrote, “Measuring quality is no simple task.”¹ We are reminded of this seminal Health Affairs article at a very pertinent point—as health care practice progresses, measuring the impact of performance improvement initiatives on clinical care delivery remains integral to monitoring overall effectiveness of quality. Mortality outcomes are a major focus of quality.

Inpatient mortality within the Veterans Health Administration (VHA) was measured as actual number of deaths (unadjusted mortality), and adjusted mortality was calculated using the standardized mortality ratio (SMR). SMR included actual number of deaths during hospitalization or within 1 day of hospital discharge divided by predicted number of deaths using a risk-adjusted formula and was calculated separately for acute level of care (LOC) and the intensive care unit (ICU). Using risk-adjusted SMR, if an observed/expected ratio was > 1.0, there were more inpatient deaths than expected; if < 1.0, fewer inpatient deaths occurred than predicted; and if 1.0, observed number of inpatient deaths was equivalent to expected number of deaths.²

Mortality reduction is a complex area of performance improvement. Health care facilities often focus their efforts on the biggest mortality contributors. According to Dantes and Epstein, sepsis results in about 265,000 deaths annually in the United States.³ Reinhart and colleagues demonstrated that sepsis is a worldwide issue resulting in approximately 30 million cases and 6 million deaths annually.⁴ Furthermore, Kumar and colleagues have noted that when sepsis progresses to septic shock, survival decreases by almost 8% for each hour delay in sepsis identification and treatment.⁵

Improvements in sepsis management have been multifaceted. The Surviving Sepsis Campaign guidelines created sepsis treatment bundles to guide early diagnosis/treatment of sepsis.⁶ In addition to awareness and sepsis care bundles, a plethora of informatics solutions within electronic health record (EHR) systems have demonstrated improved sepsis care.^7-16 Various approaches to early diagnosis and management of sepsis have been collectively referred to as an early warning sepsis system (EWSS).

An EWSS typically contains automated decision support tools that are integrated in the EHR and meant to assist health care professionals with clinical workflow decision-making. Automated decision support tools within the EHR have a variety of functions, such as clinical care reminders and alerts.¹⁷

Sepsis screening tools function as a form of automated decision support and may be incorporated into the EHR to support the EWSS. Although sepsis screening tools vary, they frequently include a combination of data involving vital signs, laboratory values and/or physical examination findings, such as mental status evaluation.The Modified Early Warning Signs (MEWS) + Sepsis Recognition Score (SRS) is one example of a sepsis screening tool.^7,16

At Malcom Randall Veterans Affairs Medical Center (MRVAMC) in Gainesville, Florida, we identified a quality improvement project opportunity to improve sepsis care in the emergency department (ED) and inpatient wards using the VHA EHR system, the Computerized Patient Record System (CPRS), which is supported by the Veterans Information Systems and Technology Architecture (VistA).¹⁸ A VistA/CPRS EWSS was developed using Lean Six Sigma DMAIC (define, measure, analyze, improve, and control) methodology.¹⁹ During the improve stage, informatics solutions were applied and included a combination of EHR interventions, such as template design, an order set, and clinical reminders. Clinical reminders have a wide variety of use, such as reminders for clinical tasks and as automated decision support within clinical workflows using Boolean logic.

To the best of our knowledge, there has been no published application of an EWSS within VistA/CPRS. In this study, we outline the strategic development of an EWSS in VistA/CPRS that assisted clinical staff with identification and treatment of sepsis; improved documentation of sepsis when present; and associated with improvement in unadjusted and adjusted inpatient mortality.

Methods 

According to policy activities that constitute research at MRVAMC, no institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

The North Florida/South Georgia Veterans Health System (NF/SGVHS) includes MRVAMC, a large academic hospital with rotating residents/fellows and multiple specialty care services. MRVAMC comprised 144 beds on the medicine/surgery wards; 48 beds in the psychiatry unit; 18 intermediate LOC beds; and 27 ICU beds. The MRVAMC SMR was identified as an improvement opportunity during fiscal year (FY) 2017 (Table 1). Its adjusted mortality for acute LOC demonstrated an observed/expected ratio of > 1.0 suggesting more inpatient deaths were observed than expected. The number of deaths (unadjusted mortality) on acute LOC at MRVAMC was noted to be rising during the first 3 quarters of FY 2017. A deeper examination of data by Pyramid Analytics (www.pyramidanalytics.com) discovered that sepsis was the primary driver for inpatient mortality on acute LOC at MRVAMC. Our goal was to reduce inpatient sepsis-related mortality via development of an EWSS that leveraged VistA/CPRS to improve early identification and treatment of sepsis in the ED and inpatient wards.

Emergency Department

Given the importance of recognizing sepsis early, the sepsis team focused on improvement opportunities at the initial point of patient contact: ED triage. The goal was to incorporate automated VistA/CPRS decision support to assist clinicians with identifying sepsis in triage using MEWS, which was chosen to optimize immediate hospital-wide buy-in. Clinical staff were already familiar with MEWS, which was in use on the inpatient wards.

Flow through the ED and availability of resources differed from the wards. Hence, modification to MEWS on the wards was necessary to fit clinical workflow in the ED. Temperature, heart rate (HR), respiratory rate (RR), systolic blood pressure (SBP), mental status, and white blood cell count (WBC) factored into a MEWS + SRS score on the wards (Table 2). For the ED, MEWS included temperature, HR, RR and SBP, but excluded mental status and WBC. Mental status assessment was excluded due to technical infeasibility (while vital signs could be automatically calculated in real time for a MEWS score, that was not possible for mental status changes). WBC was excluded from the ED as laboratory test results would not be available in triage.

MEWS + SRS scores were calculated in VistA by using clinical reminders. Clinical reminder logic included a series of conditional statements based on various combinations of MEWS + SRS clinical data entered in the EHR. When ED triage vital signs data were entered in CPRS, clinical data were stored and processed according to clinical reminder logic in VistA and displayed to the user in CPRS. While MEWS of ≥ 5 triggered a sepsis alert on the wards, the ≥ 4 threshold was used in the ED given mental status and WBC were excluded from calculations in triage (eAppendix 1 available at doi:10.12788/fp.0194).

Once a sepsis alert was triggered in triage for MEWS ≥ 4, ED nursing staff prioritized bed location and expedited staffing with an ED attending physician for early assessment. The ED attending then performed an assessment to confirm whether sepsis was present and direct early treatment. Although every patient who triggered a sepsis alert in triage did not meet clinical findings of sepsis, patients with MEWS ≥ 4 were frequently ill and required timely intervention.

If an ED attending physician agreed with a sepsis diagnosis, the physician had access to a sepsis workup and treatment order set in CPRS (eAppendix 2 available at doi:10.12788/fp.0194). The sepsis order set incorporated recommendations from the Surviving Sepsis Campaign guidelines and included orders for 2 large-bore peripheral IV lines; aggressive fluid resuscitation (30 mL/kg) for patients with clinical findings of hypoperfusion; broad-spectrum antibiotics; and frequent ordering of laboratory tests and imaging during initial sepsis workup.⁶ Vancomycin and cefepime were selected as routine broad-spectrum antibiotics in the order set when sepsis was suspected based on local antimicrobial stewardship and safety-efficacy profiles. For example, Luther and colleagues demonstrated that cefepime has lower rates of acute kidney injury when combined with vancomycin vs vancomycin + piperacillin-tazobactam.²⁰ If a β-lactam antibiotic could not be used due to a patient’s drug allergy history, aztreonam was available as an alternative option.

The design of the order set also functioned as a communication interface with clinical pharmacists. Given the large volume of antibiotics ordered in the ED, it was difficult for pharmacists to prioritize antibiotic order verification. While stat orders convey high priority, they often lack specificity. When antibiotic orders were selected from the sepsis order set, comments were already included that stated: “STAT. First dose for sepsis protocol” (eAppendix 3 available at doi:10.12788/fp.0194). This standardized communication conveyed a sense of urgency and a collective understanding that patients with suspected sepsis required timely order verification and administration of antibiotics.

Hospital Ward

Mental status and WBC were included on the wards to monitor for possible signs of sepsis, using MEWS + SRS, which was routinely monitored by nursing every 4 to 8 hours. When MEWS + SRS was ≥ 5 points, ward nursing staff called a sepsis alert.^7,16 Early response team (ERT) members received telephone notifications of the alert. ERT staff proceeded with immediate evaluation and treatment at the bedside along with determination for most appropriate LOC. The ERT members included an ICU physician and nurse; respiratory therapist; and nursing supervisor/bed flow coordinator. During bedside evaluation, if the ERT or primary team agreed with a sepsis diagnosis, the ERT or primary team used the sepsis order set to ensure standardized procedures. Stat orders generated through the sepsis order set pathway conveyed a sense of urgency and need for immediate order verification and administration of antibiotics.

In addition to clinical care process improvement, accurate documentation also was emphasized in the EWSS. When a sepsis alert was called, a clinician from the primary team was expected to complete a standardized progress note, which communicated clinical findings, a treatment plan, and captured severity of illness (eAppendix 4 available at doi:10.12788/fp.0194). It included sections for subjective, objective, assessment, and plan. In addition, data objects were created for vital signs and common laboratory findings that retrieved important clinical data from VistA and inserted it into the CPRS note.²¹

Nursing staff on the wards were expected to communicate results with the primary team for clinical decision making when a patient had a MEWS + SRS of 3 to 4. A sepsis alert may have been called at the discretion of clinical team members but was not required if the score was < 5. Additionally, vital signs were expected to be checked by the nursing staff on the wards at least every 4 hours for closer monitoring.

Sepsis Review Meetings

Weekly meetings were scheduled to review sepsis cases to assess diagnosis, treatment, and documentation entered in the patient record. The team conducting sepsis reviews comprised the chief of staff, chief of quality management, director of patient safety, physician utilization management advisor, chief resident in quality and patient safety (CRQS), and inpatient pharmacy supervisor. In addition, ad hoc physicians and nurses from different specialty areas, such as infectious diseases, hospitalist section, ICU, and the ED participated on request for subject matter expertise when needed. At the conclusion of weekly sepsis meetings, sepsis team members provided feedback to the clinical staff for continuous improvement purposes.

Results

Before implementation of an EWSS at NF/SGVHS, a plan was devised to increase awareness and educate staff on sepsis-related mortality in late FY 2017. Awareness and education about sepsis-related mortality was organized at physician, nursing, and pharmacy leadership clinical staff meetings. Posters about early warning signs of sepsis also were displayed on the nursing units for educational purposes and to convey the importance of early recognition/treatment of sepsis. In addition, the CRQS was the quality leader for house staff and led sepsis campaign change efforts for residents/fellows. An immediate improvement in unadjusted mortality at MRVAMC was noted with initial sepsis awareness and education. From FY 2017, quarter 3 to FY 2018, quarter 1, the number of acute LOC inpatient deaths decreased from 48 to 28, a 42% reduction in unadjusted mortality at MRVAMC (Figure 1). Additionally, the acute LOC SMR improved from 1.20 during FY 2017, quarter 3 down to as low as 0.71 during FY 2018, quarter 1 (Figure 2).

The number of MRVAMC inpatient deaths increased from 28 in FY 2018, quarter 1 to 45 in FY 2018, quarter 3. While acute LOC showed improvement in unadjusted mortality after sepsis education/awareness, it was felt continuous improvement could not be sustained with education alone. An EWSS was designed and implemented within the EHR system in FY 2018. Following implementation of EWSS and reeducating staff on early recognition and treatment of sepsis, acute LOC inpatient deaths decreased from 45 in FY 2018, quarter 3 through FY 2019 where unadjusted mortality was as low as 27 during FY 2019, quarter 4. The MRVAMC acute LOC SMR was consistently < 1.0 from FY 2018, quarter 4 through FY 2019, quarter 4.

In addition to the observed decrease in acute LOC inpatient deaths and improved SMR, the number of ERT alerts and sepsis alerts on the inpatient wards were monitored from FY 2017 through FY 2019. ERT alerts listed in Table 3 were nonspecific and initiated by nursing staff on the wards where a patient’s clinical status was identified as worsening while sepsis alerts were specific ERT alerts called by the ward nursing staff due to concerns for sepsis. The inpatient wards included inpatient medicine, surgery, and psychiatry acute care and the intermediate level of care unit while outpatient clinical areas of treatment, intensive care units, stroke alerts, and STEMI alerts were excluded.

Discussion

Implementation of the EWSS was associated with improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC. Although variation exists with application of EWSS at other medical centers, there was similarity with improved sepsis outcomes reported at other health care systems after EWSS implementation.^7-16

Improved unadjusted mortality and adjusted mortality for acute LOC at MRVAMC was likely due to multiple contributing factors. First, during design and implementation of the EWSS, project work was interdisciplinary with input from physicians, nurses, and pharmacists from multiple specialties (ie, ED, ICU, and the medicine service); quality management and data analysis specialists; and clinical informatics. Second, facility commitment to improving early recognition and treatment of sepsis from leadership level down to front-line staff was evident. Weekly sepsis meetings with the NF/SGVHS chief of staff helped to sustain EWSS efforts and to identify additional improvement opportunities. Third, integrated informatics solutions within the EHR helped identify early sepsis and minimized human error as well as assisted with coordination of sepsis care across services. Fourth, the focus was on both early identification and treatment of sepsis in the ED and hospital wards. Although it cannot be deduced whether there was causation between reduced inpatient mortality and an increased number of nonspecific ERT alerts+ sepsis alerts on the inpatient wards after EWSS implementation, inpatient deaths decreased and SMR improved. Finally, the EWSS emphasized both the importance of evidence-based clinical care of sepsis and standardized documentation to appropriately capture clinical severity of illness.

Limitations

This program has limitations. The EWSS was studied at a single VHA facility. Veteran demographics and local epidemiology may limit conclusion of outcomes to an individual VHA facility located in a specific geographical region. Additional research is necessary to demonstrate reproducibility and determine whether applicable to other VHA facilities and community care settings.

SMR is a risk-adjusted formula developed by the VHA Inpatient Evaluation Center, which included numerous factors such as diagnosis, comorbid conditions, age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, ICU stays, immunosuppressive status, and a COVID-19 positive indicator (added after this study). Further research is needed to evaluate sepsis-related outcomes using the EWSS during the COVID-19 pandemic.

EWSS in the literature have demonstrated various approaches to early identification and treatment of sepsis and have used different sepsis screening tools.²² Evidence suggests that the MEWS + SRS sepsis screening tool may result in false-positive screenings.^23-27 Additional research into the specificity of this sepsis screening tool is needed. Ward nursing staff were encouraged to initiate automatic sepsis alerts when MEWS + SRS was ≥ 5; however, this still depended on human factors. Because sepsis alerts are software-specific and others were incompatible with the VHA EHR, it was necessary to design our own EWSS.

Despite improvement with MRVAMC acute LOC unadjusted and adjusted mortality with our EWSS, we did not identify any actual improvement in earlier antibiotic administration times once sepsis was recognized. While accurate documentation regarding degree of sepsis improved, a MRVAMC clinical documentation improvement program was expanded in FY 2018. Therefore, it is difficult to demonstrate causation related to improved sepsis documentation with template changes alone. While sepsis alerts on the inpatient wards were variable since EWSS implementation, nonspecific ERT alerts increased. It is unclear whether some sepsis alerts were called as nonspecific ERT alerts, making it impossible to know the true number of sepsis alerts.

MRVAMC experienced an increase in nurse turnover during FY 2018 and as a teaching hospital had frequent rotating residents and fellows new to processes/protocols. These factors may have contributed to variations in unadjusted mortality. Also the decrease in inpatient mortality and improvement in SMR on acute LOC could have been the result of factors other than the EWSS and the effect of education alone may have been at least as good as that of the EWSS intervention.

Conclusions

Education along with the possible implementation of an EWSS at NF/SGVHS was associated with a decrease in the number of inpatient deaths on MRVAMC’s acute LOC wards from as high as 48 in FY 2017, quarter 3 to as low as 27 in FY 2019, quarter 4 resulting in as large of an improvement as a 44% reduction in unadjusted mortality from FY 2017 to FY 2019. In addition, MRVAMC’s acute LOC SMR improved from > 1.0 to < 1.0, demonstrating fewer inpatient mortalities than predicted from FY 2017 to FY 2019.

This multifaceted interventional strategy may be effectively applied at other VHA health care facilities that use the same EHR system. Next steps may include determining the specificity of MEWS + SRS as a sepsis screening tool; studying outcomes of MRVAMC’s EWSS during the COVID-19 era; and conducting a multicentered study on this EWSS across multiple VHA facilities.

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).

Standardized template design is a useful tool to improve clinical documentation and reliable reporting of health care outcomes when constructed with clear objectives and with collaboration of key stakeholders. A standardized template should not only capture accurate diagnostic information, but also inform quality improvement (QI) measures and best practices.

Kang and colleagues showed that a correlation exists between organizational satisfaction and improved quality outcomes.¹ A new initiative should have a well-defined purpose reinforced by collaborative workgroups and engaged employees who understand their clinical care role with electronic health record (EHR) modifications.

Several studies have shown how the usefulness of templates achieve multipurpose goals, such as accurate documentation and improved care. Valluru and colleagues showed a significant increase in vaccination rates for patients with inflammatory bowel disease after implementing a standardized template.² By using a standardized template, Thaker and colleagues showed improved documentation regarding obesity and increased nutritional and physical activity counseling.³ Furthermore, Grogan and colleagues showed that templates are useful for house staff education on International Classification of Diseases (ICD) terminology and demonstrated improved documentation in the postintervention group.^4,5

This article discusses the US Department of Veterans Affairs (VA) North Florida/South Georgia Veterans Health System (NF/SGVHS) integrated informatics solutions within template design in the Veterans Health Administration (VHA) EHR system that was associated with an increase in its case severity index (CSI) through improved clinical documentation capture.

Methods

According to policy activities that constitute research at NF/SGVHS, institutional review board approval was not required as this work met the criteria for operational improvement activities exempt from ethics review.

NF/SGVHS includes 2 hospitals: Malcom Randall VA Medical Center (MRVAMC) in Gainesville, Florida, and Lake City VA Medical Center (LCVAMC) in Lake City, Florida. MRVAMC is a large, 1a, academic VA facility composed of rotating residents and fellows and includes multiple specialty care services. LCVAMC is a smaller, nonteaching facility.

Template Design Impact

CSI is a risk-adjusted formula developed by the Inpatient Evaluation Center within VHA. CSI is incorporated into the VHA quality metrics reporting system, Strategic Analytics for Improvement and Learning (SAIL). CSI risk-adjusts metrics such as length of stay and mortality before releasing SAIL reports. CSI is calculated separately for acute level of care (LOC) and for the intensive care unit (ICU). In fiscal year (FY) 2017, acute LOC preimplementation data for CSI at NF/SGVHS were 0.76 for MRVAMC and 0.81 for LCVAMC, which was significantly below the national VHA average of 0.96 (Table).

A below-average CSI conveys a less complicated case mix compared with most other VA facilities. Although smaller VA facilities may have a less complicated case mix, it is unusual for large, tertiary care 1a VA facilities to have a low CSI. This low CSI is usually due to inadequate documentation, which affects not only risk-adjusted quality metrics outcomes, but also potential reimbursement.⁶

An interdisciplinary team composed of attendings, residents, and a clinical document improvement specialist identified the below-average acute LOC CSI for MRVAMC and LCVAMC compared with that of the national VHA average. Further analysis by chart reviews showed inconsistencies with standardized documentation despite prior health care provider education on ICD terminology and specific groups of common comorbidities analyzed in administrative data reviews for risk-adjustment purposes, known as Elixhauser comorbidities.^5,7

A chart review showed lack of clarity regarding primary reason(s) for admission and chronic comorbidities within NF/SGVHS. Using Pareto chart analysis, the template team designed a standardized history and physical (H&P) medicine template based on NF/SGVHS common medicine admissions (Figure 1). A Pareto chart is a valuable QI tool that assists with identifying majority contributors to a problem(s) being analyzed when evaluating a large set of data points. Subsequently, this tool helps focus direction on QI efforts.⁸

In addition, data objects were created within the template that permitted the ability to retrieve information from the VHA EHR and insert specific data points of interest in the template; for example, body mass index to assess degree of obesity and estimated glomerular filtration rate to determine the stage of chronic kidney disease. This allowed users to easily reference data in one template in lieu of searching for data in multiple places in the EHR.⁹

Results

The standardized H&P medicine template was implemented at MRVAMC and LCVAMC in June 2018 (the final month of the third quarter of FY 2018). As clinical providers throughout NF/SGVHS used the standardized template, acute LOC postimplementation data for CSI significantly improved. Although the national VHA average slightly decreased from 0.96 in the first quarter of FY 2017 to 0.89, in the first quarter of FY 2019, MRVAMC acute LOC CSI improved from 0.76 to 0.97, and LCVAMC acute LOC CSI improved from 0.81 to 1.07 during the same period.

In addition, compliance also was monitored within MRVAMC and LCVAMC for about 1 year after standardized H&P medicine template implementation. Compliance was determined by how often the standardized H&P medicine template was used for inpatient medicine admissions to the acute care wards vs other H&P notes used (such as personalized templates).

Methodology for compliance analysis included acquisition of completed H&P medicine notes from June 18, 2018 to June 30, 2019, within the VHA Veterans Information Systems and Technology Architecture (VistA) clinical and business information system using the search strings: “H&P admission history and physical” and “history of present illness.”¹⁰

A review identified 10,845 completed medicine H&P notes. Nine hundred eighteen notes were excluded as their search function yielded a location not corresponding to MRVAMC or LCVAMC. Of the 9,927 notes remaining, 8,025 of these were completed medicine H&P notes at MRVAMC and 1,902 were completed medicine H&P notes at LCVAMC (Figure 2).

Discussion

Template design with clear objectives, strategic collaboration, and integrated informatics solutions has the potential to increase accuracy of documentation. As shown, the NF/SGVHS template design was associated with significant improvement in acute LOC CSI for both MRVAMC and LCVAMC due to more accurate documentation using the standardized H&P medicine template.

Numerous factors contributed to the success of this template design. First, a clear vision for application of the template was communicated with key stakeholders. In addition, the template design team was focused on specific goals rather than a one size fits all approach, which was crucial for sustainable execution. Although interdisciplinary teamwork has the potential to result in innovative practices, large multidisciplinary teams also may have difficulty establishing a shared vision that can result in barriers to achieving project goals.

Balancing standardization and customization was essential for user buy-in. As noted by Gardner and Pearce, inviting clinical providers to participate in template design and allowing for customization has the potential to increase acceptance and use of templates.¹¹ Although the original design for the standardized H&P medicine template started with the medicine service at NF/SGVHS, the design framework is applicable to numerous services where various clinical care elements can be customized.

Explaining the informatics tools built into the template allowed clinicians to see opportunities to improve clinical documentation and the impact it has on reporting health care outcomes. When improvement work involves integrating clinical care delivery and administrative expectations, it is essential that health care systems understand and strategically execute project initiatives at this critical juncture.

Finally, incorporation of a sustainability plan when process improvement strategies are implemented is vital. In addition to collaboration with the clinical providers during design and implementation of the standardized template, leadership buy-in was key. Compliance with standardized H&P medicine template use was monitored monthly and reviewed by the NF/SGVHS Chief of Staff.

As noted, LCVAMC postimplementation acute LOC CSI was higher than that of MRVAMC despite being a smaller facility. This might be due to the MRVAMC designation as a teaching institution. Medicine is the only inpatient service at LCVAMC staffed by hospitalists with limited specialists available for consultation, whereas MRVAMC is a tertiary care teaching facility with numerous inpatient services and subspecialties. As LCVAMC has more continuity, house staff rotating at MRVAMC require continued training/education on new templates and process changes.

Limitations

Although standardized template design was successful at NF/SGVHS, limitations should be noted. Our clinical documentation improvement (CDI) program also was expanded about the same time as the new templates were released. The expansion of the CDI program in addition to new template design likely had a synergistic effect on acute LOC CSI.

CSI is a complex, risk-adjusted model that includes numerous factors, including but not limited to diagnosis and comorbid conditions. Other factors include age, marital status, procedures, source of admission, specific laboratory values, medical or surgical diagnosis-related group, intensive care unit stays, and immunosuppressive status. CSI also includes operative and nonoperative components that average into an overall CSI. As the majority of CSI is composed of nonoperative constituents within NF/SGVHS, we do not believe this had any substantial impact on reporting of CSI improvements.

In addition, template entry into VHA EHR requires a location selection (such as a clinic name or ward name following an inpatient admission). Of the 10,845 completed H&P medicine notes identified in VistA, 918 notes were excluded from analysis as their search function yielded a location not corresponding to MRVAMC or LCVAMC. For the 918 notes excluded, we believe this was likely due to user error where locations not related to MRVAMC or LCVAMC were selected during standardized H&P medicine template entry.

Conclusions

After the NF/SGVHS implementation of a uniquely designed template embedded with informatics solutions within the VHA EHR, the CSI increased due to more accurate documentation.

Next steps include determining the impact of the NF/SGVHS template design on potential reimbursement and expanding template design into the outpatient setting where there are additional opportunities to improve clinical documentation and reliable reporting of health care outcomes.

Acknowledgments

The authors thank the following individuals for their experience and contribution: Beverley White is the Clinical Documentation Improvement Coordinator at North Florida/South Georgia Veterans Health System and provided expertise on documentation requirements. Russell Jacobitz and Susan Rozelle provided technical expertise on electronic health record system enhancements and implemented the template design. Jess Delaune, MD, and Robert Carroll, MD, provided additional physician input during template design. We also acknowledge the Inpatient Evaluation Center (IPEC) within the Veterans Health Administration (VHA). IPEC developed the case severity index, a risk-adjusted formula incorporated into the VHA quality metric reporting system, Strategic Analytics for Improvement and Learning (SAIL).

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.

In the current health care environment, hospitals are constantly challenged to improve quality metrics and deliver better health care outcomes. One means to achieving quality improvement is through the use of order sets, groups of related orders that a health care provider (HCP) can place with either a few keystrokes or mouse clicks.¹

Historically, design of order sets has largely focused on clicking checkboxes containing evidence-based practices. According to Bates and colleagues and the Institute for Safe Medication Practices, incorporating evidence-based medicine (EBM) into order sets is not by itself sufficient.^2,3Execution of proper design coupled with simplicity and provider efficiency is paramount to HCP buy-in, increased likelihood of order set adherence, and to potentially better outcomes.

In this article, we outline advancements in order set design. These improvements increase provider efficiency and ease of use; incorporate human factors engineering (HFE); apply failure mode and effects analysis; and include EBM.

Methods

An inpatient nicotine replacement therapy (NRT) order was developed as part of a multifaceted solution to improve tobacco cessation care at the James A. Haley Veterans’ Hospital (JAHVH) in Tampa, Florida, a complexity level 1a facility. This NRT order set used the 4-step order set design framework the authors’ developed (for additional information about the NRT order set, contact the authors). We distinguish order set design technique between 2 different inpatient NRT order sets. The first order set in the comparison (Figure 1) is an inpatient NRT order set of unknown origin—it is common for US Department of Veterans Affairs (VA) medical facilities to share order sets and other resources. The second order set (Figure 2) is an inpatient NRT order set we designed using our 4-step process for comparison in this article. No institutional review board approval was required as this work met criteria for operational improvement activities exempt from ethics review.

Justin Iannello, DO, MBA, was the team leader and developer of the 4-step order set design technique. The intervention team consisted of 4 internal medicine physicians with expertise in quality improvement and patient safety: 1 certified professional in patient safety and certified as a Lean Six Sigma Black Belt; 2 physicians certified as Lean Six Sigma Black Belts; and 1 physician certified as a Lean Six Sigma Green Belt. Two inpatient clinical pharmacists and 1 quality management specialist also were involved in its development.

Development of a new NRT order set was felt to be an integral part of the tobacco cessation care delivery process. An NRT order set perceived by users as value-added required a solution that merged EBM with standardization and applied quality improvement principles. The result was an approach to order set design that focused on 4 key questions: Is the order set efficient and easy to use/navigate? Is human factors engineering incorporated? Is failure mode and effects analysis applied? Are evidence-based practices included?

Ease of Use and Navigation

Implementing an order set that is efficient and easy to use or navigate seems straightforward but can be difficult to execute. Figure 1 shows many detailed options consisting of different combinations of nicotine patches, lozenges, and gum. Also included are oral tobacco cessation options (bupropion and varenicline). Although more options may seem better, confusion about appropriate medication selection can occur.

According to Heath and Heath, too many options can result in lack of action.⁴ For example, Heath and Heath discuss a food store that offered 6 free samples of different jams on one day and 24 jams the following day. The customers who sampled 6 different types of jam were 10 times more likely to buy jam. The authors concluded that the more options available, the more difficulty a potential buyer has in deciding on a course of action.⁴

In clinical situations where a HCP is using an order set, the number of options can mean the difference between use vs avoidance if the choices are overwhelming. HCPs process layers of detail every day when creating differential diagnoses and treatment plans. While that level of detail is necessary clinically, that same level of detail included in orders sets can create challenges for HCPs.

Figure 2 advances the order set in Figure 1 by providing a simpler and cleaner design, so HCPs can more easily review and process the information. This order set design minimizes the number of options available to help users make the right decision, focusing on value for the appropriate setting and audience. In other words, order sets should not be a “one size fits all” approach.

Order sets should be tailored to the appropriate clinical setting (eg, inpatient acute care, outpatient clinic setting, etc) and HCP (eg, hospitalist, tobacco cessation specialist, etc). We are comparing NRT order sets designed for HCPs who do not routinely prescribe oral tobacco cessation products in the inpatient setting. When possible, autogenerated bundle orders should also be used according to evidence-based recommendations (such as nicotine patch tapers) for ease of use and further simplification of order sets.

Finally, usability testing known as “evaluating a product or service by testing it with representative users” helps further refine an order set.⁵Usability testing should be applied during all phases of order set development with end user(s) as it helps identify problems with order set design prior to implementation. By applying usability testing, the order set becomes more meaningful and valued by the user.

Human Factors Engineering

HFE is “the study of all the factors that make it easier to do the work in the right way.”⁶ HFE seeks to identify, align, and apply processes for people and the world within which they live and work to promote safe and efficient practices, especially in relation to the technology and physical design features in their work environment.⁶

The average American adult makes about 35,000 decisions per day.⁷ Thus, there is potential for error at any moment. Design that does not take HFE into account can be dangerous. For example, when tube feed and IV line connectors look similar and are compatible, patients may inadvertently receive food administered directly into their bloodstream.⁸

HFE can and should be applied to order sets. Everything from the look, feel, and verbiage of an order set affects potential outcomes. For example, consider the impact even seemingly minor modifications can have on outcomes simply by guiding users in a different way: Figure 1 provides NRT options based on cigarette use per day, whereas Figure 2 conveys pack use per day in relation to the equivalent number of cigarettes used daily. These differences may seem small; however, it helps guide users to the right choice when considering that health care providers have been historically trained on social history gathering that emphasizes packs per day and pack-years.

Failure Mode and Effects Analysis

Failure mode and effects analysis (FMEA) is “a structured way to identify and address potential problems, or failures and their resulting effects on the system or process before an adverse event occurs.”⁹ The benefit of an order set must be weighed against the risk during development. FMEA should be applied during order set design to assess and limit risk just as with any other clinical care process.

FMEA examines both level of risk and frequency of risk occurrence associated with a new proposed process. For example, let’s evaluate an order set designed for pain control after surgery that consists of multiple high-risk opioids along with antihistamine medications for as-needed itch relief (a non-life-threatening adverse event (AE) of opioids well known by the medical community). An interdisciplinary FMEA team consisting of subject matter experts may examine how the process should flow in step-by-step detail and then discuss the benefit of a process and risk for potential error. A FMEA team would then analyze what could go wrong with each part of the process and assign a level of risk and risk frequency for various steps in the process, and then decide that certain steps should be modified or eliminated. Perhaps after FMEA, a facility might conclude that the risk of serious complications is high when you combine opioid use with antihistamine medications. The facility could decide to remove antihistamine medications from an order set if it is determined that risks outweigh benefits. While a root cause analysis might identify the cause of an AE after order set use, these situations can be prevented with FMEA.

When applying FMEA to Figure 1, while bupropion is known as an evidence-based oral tobacco cessation option, there is the possibility that bupropion could be inadvertently prescribed from the order set in a hospitalized patient with alcohol withdrawal and withdrawal seizure history. These potentially dangerous situations can be avoided with FMEA. Thus, although bupropion may be evidence-based for NRT, decisions regarding order set design using EBM alone are insufficient.

The practitioner must consider possible unintended consequences within order sets and target treatment options to the appropriate setting and audience. Although Figure 1 may appear to be more inclusive, the interdisciplinary committee designing the inpatient NRT order set felt there was heightened risk with introducing bupropion in Figure 1 and decided the risk would be lowered by removing bupropion from the redesigned NRT order set (Figure 2). In addition to the goal of balancing availability of NRT options with acceptable risk, Figure 2 also focused on building an NRT order set most applicable to the inpatient setting.

Including Evidence-Based Practices

EBM has become a routine part of clinical decision making. Therefore, including EBM in order set design is vital. EBM for NRT has demonstrated that combination therapy is more effective than is monotherapy to help tobacco users quit. Incremental doses of NRT are recommended for patients who use tobacco more frequently.¹⁰

As shown in Figures 1 and 2, both order set designs incorporate EBM for NRT. Although the importance of implementing EBM is evident, critical factors, such as HFE and FMEA make a difference with well-designed order sets.

Results

The 4-step order set design technique was used during development of an inpatient NRT order set at the JAHVH. Results for the inpatient Joint Commission Tobacco Treatment Measures were obtained from the Veterans Health Administration quality metric reporting system known as Strategic Analytics for Improvement and Learning (SAIL). SAIL performance measure outcomes, which include the inpatient Joint Commission Tobacco Treatment Measures, are derived from chart reviews conducted by the External Peer Review Program. Outcomes demonstrated that TOB-2 and TOB-3 (2 inpatient Joint Commission Tobacco Treatment Measures) known as tob20 and tob40, respectively, within SAIL improved by more than 300% after development of an NRT order set using the 4-step order set design framework along with implementation of a multifaceted tobacco cessation care delivery system at JAHVH.

Discussion

While the overall tobacco cessation care delivery system contributed to improved outcomes with the inpatient Joint Commission Tobacco Treatment Measures at JAHVH, the NRT order set was a cornerstone of the design. Although using our order set design technique does not necessarily guarantee successful outcomes, we believe using the 4-step order set design process increases the value of order sets and has potential to improve quality outcomes.

Limitations

Although improved outcomes following implementation of our NRT order set suggest correlation, causation cannot be proven. Also while the NRT order set is believed to have helped tremendously with outcomes, the entire tobacco cessation care delivery system at JAHVH contributed to the results. In addition, the inpatient Joint Commission Tobacco Treatment Measures help improve processes for tobacco cessation care. However, we are uncertain whether the results of our improvement efforts helped patients stop tobacco use. Further studies are needed to determine impact on population health. Finally, our results were based on improvement work done at a single center. Further studies are necessary to see whether results are reproducible.

Conclusion

There was significant improvement with the inpatient Joint Commission Tobacco Treatment Measures outcomes following development of a tobacco cessation care delivery system that included design of an inpatient NRT order set using a 4-step process we developed. This 4-step structure includes emphasis on efficiency and ease of use; human factors engineering; failure mode and effects analysis; and incorporation of evidence-based medicine (Box.) Postimplementation results showed improvement of the inpatient Joint Commission Tobacco Treatment Measures by greater than 3-fold at a single hospital.

The next steps for this initiative include testing the 4-step order set design process in multiple clinical settings to determine the effectiveness of this approach in other areas of clinical care.