Hospital Medicine

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; [email protected].

Financial disclosures: None.

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; [email protected].

Financial disclosures: None.

From the University of Toronto (Dr. Basuita, Corey L. Kamen, and Dr. Soong) and Sinai Health System (Corey L. Kamen, Cheryl Ethier, and Dr. Soong), Toronto, Ontario, Canada. Co-first authors are Manpreet Basuita, MD, and Corey L. Kamen, BSc.

Abstract

• Objective: Routine laboratory testing is common among medical inpatients; however, when ordered inappropriately testing can represent low-value care. We examined the impact of an evidence-based intervention bundle on utilization.
• Participants/setting: This prospective cohort study took place at a tertiary academic medical center and included 6424 patients admitted to the general internal medicine service between April 2016 and March 2018.
• Intervention: An intervention bundle, whose first components were implemented in July 2016, included computer order entry restrictions on repetitive laboratory testing, education, and audit-feedback.
• Measures: Data were extracted from the hospital electronic health record. The primary outcome was the number of routine blood tests (complete blood count, creatinine, and electrolytes) ordered per inpatient day.
• Analysis: Descriptive statistics were calculated for demographic variables. We used statistical process control charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome.
• Results: The mean number of combined routine laboratory tests ordered per inpatient day decreased from 1.19 (SD, 0.21) tests to 1.11 (SD, 0.05), a relative reduction of 6.7% (P < 0.0001). Mean cost per case related to laboratory tests decreased from $17.24 in the pre-intervention period to $16.17 in the post-intervention period (relative reduction of 6.2%). This resulted in savings of $26,851 in the intervention year.
• Conclusion: A laboratory intervention bundle was associated with small reductions in testing and costs. A routine test performed less than once per inpatient day may not be clinically appropriate or possible.

Keywords: utilization; clinical costs; quality improvement; QI intervention; internal medicine; inpatient.

Routine laboratory blood testing is a commonly used diagnostic tool that physicians rely on to provide patient care. Although routine blood testing represents less than 5% of most hospital budgets, routine use and over-reliance on testing among physicians makes it a target of cost-reduction efforts.^1-3 A variety of interventions have been proposed to reduce inappropriate laboratory tests, with varying results.^1,4-6 Successful interventions include providing physicians with fee data associated with ordered laboratory tests, unbundling panels of tests, and multicomponent interventions.⁶ We conducted a multifaceted quality improvement study to promote and develop interventions to adopt appropriate blood test ordering practices.

Methods

Setting

This prospective cohort study took place at Mount Sinai Hospital, a 443-bed academic hospital affiliated with the University of Toronto, where more than 2400 learners rotate through annually. The study was approved by the Mount Sinai Hospital Research Ethics Board.

Participants

We included all inpatient admissions to the general internal medicine service between April 2016 and March 2018. Exclusion criteria included a length of stay (LOS) longer than 365 days and admission to a critical care unit. Patients with more than 1 admission were counted as separate hospital inpatient visits.

Intervention

Based on internal data, we targeted the top 3 most frequently ordered routine blood tests: complete blood count (CBC), creatinine, and electrolytes. Trainee interviews revealed that habit, bundled order sets, and fear of “missing something” contributed to inappropriate routine blood test ordering. Based on these root causes, we used the Model for Improvement to iteratively develop a multimodal intervention that began in July 2016.^7,8 This included a change to the computerized provider order entry (CPOE) to nudge clinicians to a restrictive ordering strategy by substituting the “Daily x3” frequency of blood test ordering with a “Daily x1” option on a pick list of order options. Clinicians could still order daily routine blood tests for any specified duration, but would have to do so by manually changing the default setting within the CPOE.

From July 2017 to March 2018, the research team educated residents on appropriate laboratory test ordering and provided audit and feedback data to the clinicians. Diagnostic uncertainty was addressed in teaching sessions. Attending physicians were surveyed on appropriate indications for daily laboratory testing for each of CBC, electrolytes, and creatinine. Appropriate indications (Figure 1) were displayed in visible clinical areas and incorporated into teaching sessions.⁹

Clinician teams received real-time performance data on their routine blood test ordering patterns compared with an institutional benchmark. Bar graphs of blood work ordering rates (sum of CBCs, creatinine, and electrolytes ordered for all patients on a given team divided by the total LOS for all patients) were distributed to each internal medicine team via email every 2 weeks (Figure 2).^1,10-12

Data Collection and Analysis

Data were extracted from the hospital electronic health record (EHR). The primary outcome was the number of routine blood tests (CBC, creatinine, and electrolytes) ordered per inpatient day. Descriptive statistics were calculated for demographic variables. We used statistical process control (SPC) charts to compare the baseline period (April 2016-June 2017) and the intervention period (July 2017-March 2018) for the primary outcome. SPC charts display process changes over time. Data are plotted in chronological order, with the central line representing the outcome mean, an upper line representing the upper control limit, and a lower line representing the lower control limit. The upper and lower limits were set at 3δ, which correspond to 3 standard deviations above and below the mean. Six successive points above or beyond the mean suggests “special cause variation,” indicating that observed results are unlikely due to secular trends. SPC charts are commonly used quality tools for process improvement as well as research.^13-16 These charts were created using QI Macros SPC software for Excel V. 2012.07 (KnowWare International, Denver, CO).

The direct cost of each laboratory test was acquired from the hospital laboratory department. The cost of each laboratory test (CBC = $7.54/test, electrolytes = $2.04/test, creatinine = $1.28/test, in Canadian dollars) was subsequently added together and multiplied by the pre- and post-intervention difference of total blood tests saved per inpatient day and then multiplied by 365 to arrive at an estimated cost savings per year.

Results

Over the study period, there were 6424 unique patient admissions on the general internal medicine service, with a median LOS of 3.5 days (Table).

The majority of inpatient visits had at least 1 test of CBC (80%; mean, 3.6 tests/visit), creatinine (79.3%; mean, 3.5 tests/visit), or electrolytes (81.6%; mean, 3.9 tests/visit) completed. In total, 56,767 laboratory tests were ordered.

Following the intervention, there was a reduction in both rates of routine blood test orders and their associated costs, with a shift below the mean. The mean number of tests ordered (combined CBC, creatinine, and electrolytes) per inpatient day decreased from 1.19 (SD, 0.21) in the pre-intervention period to 1.11 (SD, 0.05) in the post-intervention period (P < 0.0001), representing a 6.7% relative reduction (Figure 3). We observed a 6.2% relative reduction in costs per inpatient day, translating to a total savings of $26,851 over 1 year for the intervention period.

Discussion

Our study suggests that a multimodal intervention, including CPOE restrictions, resident education with posters, and audit and feedback strategies, can reduce lab test ordering on general internal medicine wards. This finding is similar to those of previous studies using a similar intervention, although different laboratory tests were targeted.^{1,2,5,6,10,17}

Our study found lower test result reductions than those reported by a previous study, which reported a relative reduction of 17% to 30%,¹⁸ and by another investigation that was conducted recently in a similar setting.¹⁷ In the latter study, reductions in laboratory testing were mostly found in nonroutine tests, and no significant improvements were noted in CBC, electrolytes, and creatine, the 3 tests we studied over the same duration.¹⁷ This may represent a ceiling effect to reducing laboratory testing, and efforts to reduce CBC, electrolytes, and creatinine testing beyond 0.3 to 0.4 tests per inpatient day (or combined 1.16 tests per inpatient day) may not be clinically appropriate or possible. This information can guide institutions to include other areas of overuse based on rates of utilization in order to maximize the benefits from a resource intensive intervention.

There are a number of limitations that merit discussion. First, observational studies do not demonstrate causation; however, to our knowledge, there were no other co-interventions that were being conducted during the study duration. One important note is that our project’s intervention began in July, at which point there are new internal medicine residents beginning their training. As the concept of resource allocation becomes more important, medical schools are spending more time educating students about Choosing Wisely, and, therefore, newer cohorts of residents may be more cognizant of appropriate blood testing. Second, this is a single-center study, limiting generalizability; however, we note that many other centers have reported similar findings. Another limitation is that we do not know whether there were any adverse clinical events associated with blood work ordering that was too restrictive, although informal tracking of STAT laboratory testing remained stable throughout the study period. It is important to ensure that blood work is ordered in moderation and tailored to patients using one’s clinical judgment.

Future Directions

We observed modest reductions in the quantity and costs associated with a quality improvement intervention aimed at reducing routine blood testing. A baseline rate of laboratory testing of less than 1 test per inpatient day may require including other target tests to drive down absolute utilization.

Corresponding author: Christine Soong, MD, MSc, 433-600 University Avenue, Toronto, Ontario, Canada M5G 1X5; [email protected].

Financial disclosures: None.

From the Senior Health Program, Providence Health & Services, Oregon, Portland, OR.

Abstract

Background: Approximately 51 million adults in the United States are 65 years of age or older, yet few geriatric-trained primary care providers (PCP) serve this population. The Age-Friendly Health System framework, consisting of evidence-based 4M care (Mobility, Medication, Mentation, and what Matters), encourages all PCPs to assess mobility in older adults.

Objective: To improve PCP knowledge, confidence, and clinical practice in assessing and managing fall risk.

Methods: A 1-week educational session focusing on mobility (part of a 4-week Geriatric Mini-Fellowship) for 6 selected PCPs from a large health care system was conducted to increase knowledge and ability to address fall risk in older adults. The week included learning and practicing a Fall Risk Management Plan (FRMP) algorithm, including planning for their own practice changes. Pre- and post-test surveys assessed changes in knowledge and confidence. Patient data were compared 12 months before and after training to evaluate PCP adoption of FRMP components.

Results: The training increased provider knowledge and confidence. The trained PCPs were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to assess orthostatic blood pressure in their 65+ patients after the mini-fellowship. In high-risk patients, they were 4.1 times more likely to discuss fall risk and 6.3 times more likely to assess orthostatic blood pressure than their nontrained peers. Changes in physical therapy referral rates were not observed.

Conclusions: In-depth, skills-based geriatric educational sessions improved PCPs’ knowledge and confidence and also improved their fall risk management practices for their older patients.

Keywords: geriatrics; guidelines; Age-Friendly Health System; 4M; workforce training; practice change; fellowship.

The US population is aging rapidly. People aged 85 years and older are the largest-growing segment of the US population, and this segment is expected to increase by 123% by 2040.¹ Caregiving needs increase with age as older adults develop more chronic conditions, such as hypertension, heart disease, arthritis, and dementia. However, even with increasing morbidity and dependence, a majority of older adults still live in the community rather than in institutional settings.² These older adults seek medical care more frequently than younger people, with about 22% of patients 75 years and older having 10 or more health care visits in the previous 12 months. By 2040, nearly a quarter of the US population is expected to be 65 or older, with many of these older adults seeking regular primary care from providers who do not have formal training in the care of a population with multiple complex, chronic health conditions and increased caregiving needs.¹

Despite this growing demand for health care professionals trained in the care of older adults, access to these types of clinicians is limited. In 2018, there were roughly 7000 certified geriatricians, with only 3600 of them practicing full-time.^3,4 Similarly, of 290,000 certified nurse practitioners (NPs), about 9% of them have geriatric certification.⁵ Geriatricians, medical doctors trained in the care of older adults, and geriatric-trained NPs are part of a cadre of a geriatric-trained workforce that provides unique expertise in caring for older adults with chronic and advanced illness. They know how to manage multiple, complex geriatric syndromes like falls, dementia, and polypharmacy; understand and maximize team-based care; and focus on caring for an older person with a goal-centered versus a disease-centered approach.⁶

Broadly, geriatric care includes a spectrum of adults, from those who are aging healthfully to those who are the frailest. Research has suggested that approximately 30% of older adults need care by a geriatric-trained clinician, with the oldest and frailest patients needing more clinician time for assessment and treatment, care coordination, and coaching of caregivers.⁷ With this assumption in mind, it is projected that by 2025, there will be a national shortage of 26,980 geriatricians, with the western United States disproportionately affected by this shortage.⁴Rather than lamenting this shortage, Tinetti recommends a new path forward: “Our mission should not be to train enough geriatricians to provide direct care, but rather to ensure that every clinician caring for older adults is competent in geriatric principles and practices.”⁸ Sometimes called ”geriatricizing,” the idea is to use existing geriatric providers as a small elite training force to infuse geriatric principles and skills across their colleagues in primary care and other disciplines.^8,9 Efforts of the American Geriatrics Society (AGS), with support from the John A. Hartford Foundation (JAHF), have been successful in developing geriatric training across multiple specialties, including surgery, orthopedics, and emergency medicine (www.americangeriatrics.org/programs/geriatrics-specialists-initiative).

The Age-Friendly Health System and 4M Model

To help augment this idea of equipping health care systems and their clinicians with more readily available geriatric knowledge, skills, and tools, the JAHF, along with the Institute for Healthcare Improvement (IHI), created the Age-Friendly Health System (AFHS) paradigm in 2015.¹⁰ Using the 4M model, the AFHS initiative established a set of evidence-based geriatric priorities and interventions meant to improve the care of older adults, reduce harm and duplication, and provide a framework for engaging leadership, clinical teams, and operational systems across inpatient and ambulatory settings.¹¹ Mobility, including fall risk screening and intervention, is 1 of the 4M foundational elements of the Age-Friendly model. In addition to Mobility, the 4M model also includes 3 other key geriatric domains: Mentation (dementia, depression, and delirium), Medication (high-risk medications, polypharmacy, and deprescribing), and What Matters (goals of care conversations and understanding quality of life for older patients).¹¹ The 4M initiative encourages adoption of a geriatric lens that looks across chronic conditions and accounts for the interplay among geriatric syndromes, such as falls, cognitive impairment, and frailty, in order to provide care better tailored to what the patient needs and desires.¹² IHI and JAHF have targeted the adoption of the 4M model by 20% of US health care systems by 2020.¹¹

Mini-Fellowship and Mobility Week

To bolster geriatric skills among community-based primary care providers (PCPs), we initiated a Geriatric Mini-Fellowship, a 4-week condensed curriculum taught over 6 months. Each week focuses on 1 of the age-friendly 4Ms, with the goal of increasing the knowledge, self-efficacy, skills, and competencies of the participating PCPs (called “fellow” hereafter) and at the same time, equipping each to become a champion of geriatric practice. This article focuses on the Mobility week, the second week of the mini-fellowship, and the effect of the week on the fellows’ practice changes.

To construct the Mobility week’s curriculum with a focus on the ambulatory setting, we relied upon national evidence-based work in fall risk management. The Centers for Disease Control and Prevention (CDC) has made fall risk screening and management in primary care a high priority. Using the clinical practice guidelines for managing fall risk developed by the American and British Geriatrics Societies (AGS/BGS), the CDC developed the Stopping Elderly Accidents, Deaths, and Injuries (STEADI) toolkit.¹³ Foundational to the toolkit is the validated 12-item Stay Independent falls screening questionnaire (STEADI questionnaire).¹⁴ Patients who score 4 or higher (out of a total score of 14) on the questionnaire are considered at increased risk of falling. The CDC has developed a clinical algorithm that guides clinical teams through screening and assessment to help identify appropriate interventions to target specific risk factors. Research has clearly established that a multifactorial approach to fall risk intervention can be successful in reducing fall risk by as much as 25%.^15-17

The significant morbidity and mortality caused by falls make training nongeriatrician clinicians on how to better address fall risk imperative. More than 25% of older adults fall each year.¹⁸ These falls contribute to rising rates of fall-related deaths,¹⁹ emergency department (ED) visits,²⁰ and hospital readmissions.²¹ Initiatives like the AFHS focus on mobility and the CDC’s development of supporting clinical materials²² aim to improve primary care adoption of fall risk screening and intervention practices.^23,24 The epidemic of falls must compel all PCPs, not just those practicing geriatrics, to make discussing and addressing fall risk and falls a priority.

Methods

Setting

This project took place as part of a regional primary care effort in Oregon. Providence Health & Services-Oregon is part of a multi-state integrated health care system in the western United States whose PCPs serve more than 80,000 patients aged 65 years and older per year; these patients comprise 38% of the system’s office visits each year. Regionally, there are 47 family and internal medicine clinics employing roughly 290 providers (physicians, NPs, and physician assistants). The organization has only 4 PCPs trained in geriatrics and does not offer any geriatric clinical consultation services. Six PCPs from different clinics, representing both rural and urban settings, are chosen to participate in the geriatric mini-fellowship each year.

This project was conducted as a quality improvement initiative within the organization and did not constitute human subjects research. It was not conducted under the oversight of the Institutional Review Board.

Intervention

The mini-fellowship was taught in 4 1-week blocks between April and October 2018, with a curriculum designed to be interactive and practical. The faculty was intentionally interdisciplinary to teach and model team-based practice. Each week participants were excused from their clinical practice. Approximately 160 hours of continuing medical education credits were awarded for the full mini-fellowship. As part of each weekly session, a performance improvement project (PIP) focused on that week’s topic (1 of the 4Ms) was developed by the fellow and their team members to incorporate the mini-fellowship learnings into their clinic workflows. Fellows also had 2 hours per week of dedicated administration time for a year, outside the fellowship, to work on their PIP and 4M practice changes within their clinic.

Provider Education

The week for mobility training comprised 4 daylong sessions. The first 2 days were spent learning about the epidemiology of falls; risk factors for falling; how to conduct a thorough history and assessment of fall risk; and how to create a prioritized Fall Risk Management Plan (FRMP) to decrease a patient’s individual fall risk through tailored interventions. The FRMP was adapted from the CDC STEADI toolkit.¹³ Core faculty were 2 geriatric-trained providers (NP and physician) and a physical therapist (PT) specializing in fall prevention.

On the third day, fellows took part in a simulated fall risk clinic, in which older adults volunteered to be patient partners, providing an opportunity to apply learnings from days 1 and 2. The clinic included the fellow observing a PT complete a mobility assessment and a pharmacist conduct a high-risk medication review. The fellow synthesized the findings of the mobility assessment and medication review, as well as their own history and assessment, to create a summary of fall risk recommendations to discuss with their volunteer patient partner. The fellows were observed and evaluated in their skills by their patient partner, course faculty, and another fellow. The patient partners, and their assigned fellow, also participated in a 45-minute fall risk presentation, led by a nurse.

On the fourth day, the fellows were joined by select clinic partners, including nurses, pharmacists, and/or medical assistants. The session included discussions among each fellow’s clinical team regarding the current state of fall risk efforts at their clinic, an analysis of barriers, and identification of opportunities to improve workflows and screening rates. Each fellow took with them an action plan tailored to their clinic to improve fall risk management practices, starting with the fellow’s own practice.

Fall Risk Management Plan

The educational sessions introduced the fellows to the FRMP. The FRMP, adapted from the STEADI toolkit, includes a process for fall risk screening (Figure 1) and stratifying a patient’s risk based on their STEADI score in order to promote 3 priority assessments (gait evaluation with PT referral if appropriate; orthostatic blood pressure; and high-risk medication review; Figure 2). Initial actions based on these priority assessments were followed over time, with additional fall risk interventions added as clinically indicated.²⁵ The FRMP is intended to be used during routine office visits, Medicare annual wellness visits, or office visits focused on fall risk or related medical disorders (ie, fall risk visits.)

Providers and their teams were encouraged to spread out fall-related conversations with their patients over multiple visits, since many patients have multiple fall risk factors at play, in addition to other chronic medical issues, and since many interventions often require behavior changes on the part of the patient. Providers also had access to fall-related electronic health record (EHR) templates as well as a comprehensive, internal fall risk management website that included assessment tools, evidence-based resources, and patient handouts.

Assessment and Measurements

We assessed provider knowledge and comfort in their fall risk evaluation and management skills before and after the educational intervention using an 11-item multiple-choice questionnaire and a 4-item confidence questionnaire. The confidence questions used a 7-point Likert scale, with 0 indicating “no confidence” and 7 indicating ”lots of confidence.” The questions were administered via a paper survey. Qualitative comments were derived from evaluations completed at the end of the week.

The fellows’ practice of fall risk screening and management was studied from May 2018, at the completion of Mobility week, to May 2019 for the post-intervention period. A 1-year timeframe before May 2018 was used as the pre-intervention period. Eligible visit types, during which we assumed fall risk was discussed, were any office visits for patients 65+ completed by the patients’ PCPs that used fall risk as a reason for the visit or had a fall-related diagnosis code. Fall risk visits performed by other clinic providers were not counted.

Of those patients who had fall risk screenings completed and were determined to be high risk (STEADI score ≥ 4), data were analyzed to determine whether these patients had any fall-related follow-up visits to their PCP within 60 days of the STEADI screening. For these high-risk patients, data were studied to understand whether orthostatic blood pressure measurements were performed (as documented in a flowsheet) and whether a PT referral was placed. These data were compared with those from providers who practiced in clinics within the same system but who did not participate in the mini-fellowship. Data were obtained from the organization’s EHR. Additional data were measured to evaluate patterns of deprescribing of select high-risk medications, but these data are not included in this analysis.

Analysis

A paired-samples t test was used to measure changes in provider confidence levels. Data were aggregated across fellows, resulting in a mean. A chi-square test of independence was performed to examine the relationship between rates of FRMP adoption by select provider groups. Analysis included a pre- and post-intervention assessment of the fellows’ adoption of FRMP practices, as well as a comparison between the fellows’ practice patterns and those of a control group of PCPs in the organization’s other clinics who did not participate in the mini-fellowship (nontrained control group). Excluded from the control group were providers from the same clinic as the fellows; providers in clinics with a geriatric-trained provider on staff; and clinics outside of the Portland metro and Medford service areas. We used an alpha level of 0.05 for all statistical tests.

Data from 5 providers were included in the analysis of the FRMP adoption. The sixth provider changed practice settings from the clinic to the ED after completing the fellowship; her patient data were not included in the FRMP part of the analysis. EHR data included data on all visits of patients 65+, as well as data for just those 65+ patients who had been identified as being at high risk to fall based on a STEADI score of 4 or higher.

Results

Provider Questionnaire

All 6 providers responded to the pre-intervention and post-intervention tests. For the knowledge questions, fellows, as a composite, correctly answered 57% of the questions before the intervention and 79% after the intervention. Provider confidence level in delivering fall risk care was measured prior to the training (mean, 4.12 [SD, 0.62]) and at the end of the training (mean, 6.47 [SD, 0.45]), demonstrating a significant increase in confidence (t (5) = –10.46, P < 0.001).

Qualitative Comments

Providers also had the opportunity to provide comments on their experience during the Mobility week and at the end of 1 year. In general, the simulated interdisciplinary fall risk clinic was highly rated (“the highlight of the week”) as a practical strategy to embed learning principles. One fellow commented, “Putting the learning into practice helps solidify it in my brain.” Fellows also appreciated the opportunity to learn and meet with their clinic colleagues to begin work on a fall-risk focused PIP and to “have a framework for what to do for people who screen positive [for fall risk].”

FRMP Adoption

A comparison of the care the fellows provided to their patients 65+ in the 12 months pre- and post-training shows the fellows demonstrated significant changes in practice patterns. The fellows were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to check orthostatic blood pressure than prior to the mini-fellowship (Table 1).

The control providers also demonstrated significant increases in fall risk screening and discussion of fall risk between the pre- and post-intervention periods; however, the relative risk (RR) was between 1.10 and 1.13 for this group. For the control group, checking orthostatic blood pressure did not significantly change. In the 12 months after training (Table 2), the fellows were 4.2 times more likely to discuss fall risk and almost 5 times more likely to check orthostatic blood pressure than their nontrained peers for all of their patients 65+, regardless of their risk to fall.

As shown in Table 3, for those patients determined to be at high risk of falling (STEADI score ≥ 4), fellows showed statistically significant increases in fall risk visits (RR, 3.02) and assessment of orthostatic blood pressure (RR, 10.68) before and after the mini-fellowship. The control providers did not show any changes in practice patterns between the pre- and post-period among patients at high risk to fall.

Neither the fellows nor the control group showed changes in patterns of referral to PT. In comparing the 2 groups in the 12 months after training (Table 4), for their patients at risk of falling, the fellows were 4 times more likely to complete fall risk visits and over 6 times more likely to assess orthostatic blood pressure than their nontrained peers. Subgroup analysis of the 75+ population revealed similar trends and significance, but these results are not included here.

Discussion

This study aimed to improve not only providers’ knowledge and confidence in caring for older adults at increased risk to fall, but also their clinical practice in assessing and managing fall risk. In addition to improved knowledge and confidence, we found that the fellows increased their discussion of fall risk (through fall risk visits) and their assessment of orthostatic blood pressure for all of their patients, not just for those identified at increased risk to fall. This improvement held true for the fellows themselves before and after the intervention, but also as compared to their nontrained peers. These practice improvements for all of their 65+ patients, not just those identified as being at high risk to fall, are especially important, since studies indicate that early screening and intervention can help identify people at risk and prevent future falls.¹⁵

We were surprised that there were no significant differences in PT referrals made by the trained fellows, but this finding may have been confounded by the fact that the data included all PT referrals, regardless of diagnosis, not just those referrals that were fall-related. Furthermore, our baseline PT referral rates, at 39% for the intervention group and 42% for the control group, are higher than national data when looking at rehabilitation use by older adults.²⁶

In comparison to a study evaluating the occurrence of fall risk–related clinical practice in primary care before any fall-related educational intervention, orthostatics were checked less frequently in our study (10% versus 30%) and there were fewer PT referrals (42%–44% versus 53%).²⁷ However, the Phelan study took place in patients who had actually had a fall, rather than just having a higher risk for a fall, and was based on detailed chart review. Other studies^23,24 found higher rates of fall risk interventions, but did not break out PT referrals specifically.

In terms of the educational intervention itself, most studies of geriatric education interventions have measured changes in knowledge, confidence, or self-efficacy as they relate to geriatric competence,^28-30 and do not measure practice change as an outcome outside of intent to change or self-reported practice change.^31,32 In general, practice change or longer-term health care–related outcomes have not been studied. Additionally, a range of dosages of educational interventions has been studied, from 1-hour lunchtime presentations^23,32 to half-day²⁹ or several half-day workshops,²⁸ up to 160 hours over 10 months³⁰ or 5 weekends over 6 months.³¹ The duration of our entire intervention at 160 hours over 6 months would be considered on the upper end of dosing relative to these studies, with our Mobility week intervention comprising 32 hours during 1 week. In the Warshaw study, despite 107 1-hour sessions being taught to over 60 physicians in 16 practices over 4 years, only 2 practices ultimately initiated any practice change projects.³² We believe that only curricula that embed practice change skills and opportunities, at a significant enough dose, can actually impact practice change in a sustainable manner.

Knowledge and skill acquisition among individual providers does not take place to a sufficient degree in the current health care arena, which is focused on productivity and short visit times. Consistent with other studies, we included interdisciplinary members of the primary care team for part of the mini-fellowship, although other studies used models that train across disciplines for the entirety of the learning experience.^28-30,33 Our educational model was strengthened by including other professionals to provide some of the education and model the ideal geriatric team, including PT, occupational therapy, and pharmacy, for the week on mobility.

Most studies exploring interventions through geriatric educational initiatives are conducted within academic institutions, with a primary focus on physician faculty and, by extension, their teaching of residents and others.^34,35 We believe our integrated model, which is steeped in community-based primary care practices like Lam’s,³¹ offers the greatest outreach to large community-based care systems and their patients. Training providers to work with their teams to change their own practices first gives skills and expertise that help further establish them as geriatric champions within their practices, laying the groundwork for more widespread practice change at their clinics.

Limitations

In addition to the limitations described above relating to the capture of PT referrals, other limitations included the relatively short time period for follow-up data as well as the small size of the intervention group. However, we found value in the instructional depth that the small group size allowed.

While the nontrained providers did show some improvement during the same period, we believe the relative risk was not clinically significant. We suspect that the larger health system efforts to standardize screening of patients 65+ across all clinics as a core quality metric confounded these results. The data analysis also included only fall-related patient visits that occurred with a provider who was that patient’s PCP, which could have missed visits done by other PCP colleagues, RNs, or pharmacists in the same clinic, thus undercounting the true number of fall-related visits. Furthermore, counting of fall-related interventions relied upon providers documenting consistently in the EHR, which could also lead to under-represention of fall risk clinical efforts.

The data presented, while encouraging, do not reflect clinic-wide practice change patterns and are considered only proximate outcomes rather than more long-term or cost-related outcomes, as would be captured by fall-related utilization measures like emergency room visits and hospitalizations. We expect to evaluate the broader impact and these value-based outcomes in the future. All providers and teams were from the same health care system, which may not allow our results to transfer to other organizations or regions of clinical practice.

Summary

This study demonstrates that an intensive mini-fellowship model of geriatrics training improved both knowledge and confidence in the realm of fall risk assessment and intervention among PCPs who had not been formally trained in geriatrics. More importantly, the training improved the fall-related care of their patients at increased risk to fall, but also of all of their older patients, with improvements in care measured up to a year after the mini-fellowship. Although this article only describes the work done as part of the Mobility aim of the 4M AFHS model, we believe the entire mini-fellowship curriculum offers the opportunity to “geriatricize” clinicians and their teams in learning geriatric principles and skills that they can translate into their practice in a sustainable way, as Tinetti encourages.⁸ Future study to evaluate other process outcomes more precisely, such as PT, as well as cost- and value-based outcomes, and the influence of trained providers on their clinic partners, will further establish the value proposition of targeted, disseminated, intensive geriatrics training of primary care clinicians as a strategy of age-friendly health systems as they work to improve the care of their older adults.

Acknowledgment: We are grateful for the dedication and hard work of the 2018 Geriatric Mini-Fellowship fellows at Providence Health & Services-Oregon who made this article possible. Thanks to Drs. Stephanie Cha, Emily Puukka-Clark, Laurie Dutkiewicz, Cara Ellis, Deb Frost, Jordan Roth, and Subhechchha Shah for promoting the AFHS work within their Providence Medical Group clinics and to PMG leadership and the fellows’ clinical teams for supporting the fellows, the AFHS work, and their older patients.

Corresponding author: Colleen M. Casey, PhD, ANP-BC, Providence Health & Services, Senior Health Program, 4400 NE Halsey, 5th Floor, Portland, OR 97213; [email protected].

Financial disclosures: None.

From the Senior Health Program, Providence Health & Services, Oregon, Portland, OR.

Abstract

Background: Approximately 51 million adults in the United States are 65 years of age or older, yet few geriatric-trained primary care providers (PCP) serve this population. The Age-Friendly Health System framework, consisting of evidence-based 4M care (Mobility, Medication, Mentation, and what Matters), encourages all PCPs to assess mobility in older adults.

Objective: To improve PCP knowledge, confidence, and clinical practice in assessing and managing fall risk.

Methods: A 1-week educational session focusing on mobility (part of a 4-week Geriatric Mini-Fellowship) for 6 selected PCPs from a large health care system was conducted to increase knowledge and ability to address fall risk in older adults. The week included learning and practicing a Fall Risk Management Plan (FRMP) algorithm, including planning for their own practice changes. Pre- and post-test surveys assessed changes in knowledge and confidence. Patient data were compared 12 months before and after training to evaluate PCP adoption of FRMP components.

Results: The training increased provider knowledge and confidence. The trained PCPs were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to assess orthostatic blood pressure in their 65+ patients after the mini-fellowship. In high-risk patients, they were 4.1 times more likely to discuss fall risk and 6.3 times more likely to assess orthostatic blood pressure than their nontrained peers. Changes in physical therapy referral rates were not observed.

Conclusions: In-depth, skills-based geriatric educational sessions improved PCPs’ knowledge and confidence and also improved their fall risk management practices for their older patients.

Keywords: geriatrics; guidelines; Age-Friendly Health System; 4M; workforce training; practice change; fellowship.

The US population is aging rapidly. People aged 85 years and older are the largest-growing segment of the US population, and this segment is expected to increase by 123% by 2040.¹ Caregiving needs increase with age as older adults develop more chronic conditions, such as hypertension, heart disease, arthritis, and dementia. However, even with increasing morbidity and dependence, a majority of older adults still live in the community rather than in institutional settings.² These older adults seek medical care more frequently than younger people, with about 22% of patients 75 years and older having 10 or more health care visits in the previous 12 months. By 2040, nearly a quarter of the US population is expected to be 65 or older, with many of these older adults seeking regular primary care from providers who do not have formal training in the care of a population with multiple complex, chronic health conditions and increased caregiving needs.¹

Despite this growing demand for health care professionals trained in the care of older adults, access to these types of clinicians is limited. In 2018, there were roughly 7000 certified geriatricians, with only 3600 of them practicing full-time.^3,4 Similarly, of 290,000 certified nurse practitioners (NPs), about 9% of them have geriatric certification.⁵ Geriatricians, medical doctors trained in the care of older adults, and geriatric-trained NPs are part of a cadre of a geriatric-trained workforce that provides unique expertise in caring for older adults with chronic and advanced illness. They know how to manage multiple, complex geriatric syndromes like falls, dementia, and polypharmacy; understand and maximize team-based care; and focus on caring for an older person with a goal-centered versus a disease-centered approach.⁶

Broadly, geriatric care includes a spectrum of adults, from those who are aging healthfully to those who are the frailest. Research has suggested that approximately 30% of older adults need care by a geriatric-trained clinician, with the oldest and frailest patients needing more clinician time for assessment and treatment, care coordination, and coaching of caregivers.⁷ With this assumption in mind, it is projected that by 2025, there will be a national shortage of 26,980 geriatricians, with the western United States disproportionately affected by this shortage.⁴Rather than lamenting this shortage, Tinetti recommends a new path forward: “Our mission should not be to train enough geriatricians to provide direct care, but rather to ensure that every clinician caring for older adults is competent in geriatric principles and practices.”⁸ Sometimes called ”geriatricizing,” the idea is to use existing geriatric providers as a small elite training force to infuse geriatric principles and skills across their colleagues in primary care and other disciplines.^8,9 Efforts of the American Geriatrics Society (AGS), with support from the John A. Hartford Foundation (JAHF), have been successful in developing geriatric training across multiple specialties, including surgery, orthopedics, and emergency medicine (www.americangeriatrics.org/programs/geriatrics-specialists-initiative).

The Age-Friendly Health System and 4M Model

To help augment this idea of equipping health care systems and their clinicians with more readily available geriatric knowledge, skills, and tools, the JAHF, along with the Institute for Healthcare Improvement (IHI), created the Age-Friendly Health System (AFHS) paradigm in 2015.¹⁰ Using the 4M model, the AFHS initiative established a set of evidence-based geriatric priorities and interventions meant to improve the care of older adults, reduce harm and duplication, and provide a framework for engaging leadership, clinical teams, and operational systems across inpatient and ambulatory settings.¹¹ Mobility, including fall risk screening and intervention, is 1 of the 4M foundational elements of the Age-Friendly model. In addition to Mobility, the 4M model also includes 3 other key geriatric domains: Mentation (dementia, depression, and delirium), Medication (high-risk medications, polypharmacy, and deprescribing), and What Matters (goals of care conversations and understanding quality of life for older patients).¹¹ The 4M initiative encourages adoption of a geriatric lens that looks across chronic conditions and accounts for the interplay among geriatric syndromes, such as falls, cognitive impairment, and frailty, in order to provide care better tailored to what the patient needs and desires.¹² IHI and JAHF have targeted the adoption of the 4M model by 20% of US health care systems by 2020.¹¹

Mini-Fellowship and Mobility Week

To bolster geriatric skills among community-based primary care providers (PCPs), we initiated a Geriatric Mini-Fellowship, a 4-week condensed curriculum taught over 6 months. Each week focuses on 1 of the age-friendly 4Ms, with the goal of increasing the knowledge, self-efficacy, skills, and competencies of the participating PCPs (called “fellow” hereafter) and at the same time, equipping each to become a champion of geriatric practice. This article focuses on the Mobility week, the second week of the mini-fellowship, and the effect of the week on the fellows’ practice changes.

To construct the Mobility week’s curriculum with a focus on the ambulatory setting, we relied upon national evidence-based work in fall risk management. The Centers for Disease Control and Prevention (CDC) has made fall risk screening and management in primary care a high priority. Using the clinical practice guidelines for managing fall risk developed by the American and British Geriatrics Societies (AGS/BGS), the CDC developed the Stopping Elderly Accidents, Deaths, and Injuries (STEADI) toolkit.¹³ Foundational to the toolkit is the validated 12-item Stay Independent falls screening questionnaire (STEADI questionnaire).¹⁴ Patients who score 4 or higher (out of a total score of 14) on the questionnaire are considered at increased risk of falling. The CDC has developed a clinical algorithm that guides clinical teams through screening and assessment to help identify appropriate interventions to target specific risk factors. Research has clearly established that a multifactorial approach to fall risk intervention can be successful in reducing fall risk by as much as 25%.^15-17

The significant morbidity and mortality caused by falls make training nongeriatrician clinicians on how to better address fall risk imperative. More than 25% of older adults fall each year.¹⁸ These falls contribute to rising rates of fall-related deaths,¹⁹ emergency department (ED) visits,²⁰ and hospital readmissions.²¹ Initiatives like the AFHS focus on mobility and the CDC’s development of supporting clinical materials²² aim to improve primary care adoption of fall risk screening and intervention practices.^23,24 The epidemic of falls must compel all PCPs, not just those practicing geriatrics, to make discussing and addressing fall risk and falls a priority.

Methods

Setting

This project took place as part of a regional primary care effort in Oregon. Providence Health & Services-Oregon is part of a multi-state integrated health care system in the western United States whose PCPs serve more than 80,000 patients aged 65 years and older per year; these patients comprise 38% of the system’s office visits each year. Regionally, there are 47 family and internal medicine clinics employing roughly 290 providers (physicians, NPs, and physician assistants). The organization has only 4 PCPs trained in geriatrics and does not offer any geriatric clinical consultation services. Six PCPs from different clinics, representing both rural and urban settings, are chosen to participate in the geriatric mini-fellowship each year.

This project was conducted as a quality improvement initiative within the organization and did not constitute human subjects research. It was not conducted under the oversight of the Institutional Review Board.

Intervention

The mini-fellowship was taught in 4 1-week blocks between April and October 2018, with a curriculum designed to be interactive and practical. The faculty was intentionally interdisciplinary to teach and model team-based practice. Each week participants were excused from their clinical practice. Approximately 160 hours of continuing medical education credits were awarded for the full mini-fellowship. As part of each weekly session, a performance improvement project (PIP) focused on that week’s topic (1 of the 4Ms) was developed by the fellow and their team members to incorporate the mini-fellowship learnings into their clinic workflows. Fellows also had 2 hours per week of dedicated administration time for a year, outside the fellowship, to work on their PIP and 4M practice changes within their clinic.

Provider Education

The week for mobility training comprised 4 daylong sessions. The first 2 days were spent learning about the epidemiology of falls; risk factors for falling; how to conduct a thorough history and assessment of fall risk; and how to create a prioritized Fall Risk Management Plan (FRMP) to decrease a patient’s individual fall risk through tailored interventions. The FRMP was adapted from the CDC STEADI toolkit.¹³ Core faculty were 2 geriatric-trained providers (NP and physician) and a physical therapist (PT) specializing in fall prevention.

On the third day, fellows took part in a simulated fall risk clinic, in which older adults volunteered to be patient partners, providing an opportunity to apply learnings from days 1 and 2. The clinic included the fellow observing a PT complete a mobility assessment and a pharmacist conduct a high-risk medication review. The fellow synthesized the findings of the mobility assessment and medication review, as well as their own history and assessment, to create a summary of fall risk recommendations to discuss with their volunteer patient partner. The fellows were observed and evaluated in their skills by their patient partner, course faculty, and another fellow. The patient partners, and their assigned fellow, also participated in a 45-minute fall risk presentation, led by a nurse.

On the fourth day, the fellows were joined by select clinic partners, including nurses, pharmacists, and/or medical assistants. The session included discussions among each fellow’s clinical team regarding the current state of fall risk efforts at their clinic, an analysis of barriers, and identification of opportunities to improve workflows and screening rates. Each fellow took with them an action plan tailored to their clinic to improve fall risk management practices, starting with the fellow’s own practice.

Fall Risk Management Plan

The educational sessions introduced the fellows to the FRMP. The FRMP, adapted from the STEADI toolkit, includes a process for fall risk screening (Figure 1) and stratifying a patient’s risk based on their STEADI score in order to promote 3 priority assessments (gait evaluation with PT referral if appropriate; orthostatic blood pressure; and high-risk medication review; Figure 2). Initial actions based on these priority assessments were followed over time, with additional fall risk interventions added as clinically indicated.²⁵ The FRMP is intended to be used during routine office visits, Medicare annual wellness visits, or office visits focused on fall risk or related medical disorders (ie, fall risk visits.)

Providers and their teams were encouraged to spread out fall-related conversations with their patients over multiple visits, since many patients have multiple fall risk factors at play, in addition to other chronic medical issues, and since many interventions often require behavior changes on the part of the patient. Providers also had access to fall-related electronic health record (EHR) templates as well as a comprehensive, internal fall risk management website that included assessment tools, evidence-based resources, and patient handouts.

Assessment and Measurements

We assessed provider knowledge and comfort in their fall risk evaluation and management skills before and after the educational intervention using an 11-item multiple-choice questionnaire and a 4-item confidence questionnaire. The confidence questions used a 7-point Likert scale, with 0 indicating “no confidence” and 7 indicating ”lots of confidence.” The questions were administered via a paper survey. Qualitative comments were derived from evaluations completed at the end of the week.

The fellows’ practice of fall risk screening and management was studied from May 2018, at the completion of Mobility week, to May 2019 for the post-intervention period. A 1-year timeframe before May 2018 was used as the pre-intervention period. Eligible visit types, during which we assumed fall risk was discussed, were any office visits for patients 65+ completed by the patients’ PCPs that used fall risk as a reason for the visit or had a fall-related diagnosis code. Fall risk visits performed by other clinic providers were not counted.

Of those patients who had fall risk screenings completed and were determined to be high risk (STEADI score ≥ 4), data were analyzed to determine whether these patients had any fall-related follow-up visits to their PCP within 60 days of the STEADI screening. For these high-risk patients, data were studied to understand whether orthostatic blood pressure measurements were performed (as documented in a flowsheet) and whether a PT referral was placed. These data were compared with those from providers who practiced in clinics within the same system but who did not participate in the mini-fellowship. Data were obtained from the organization’s EHR. Additional data were measured to evaluate patterns of deprescribing of select high-risk medications, but these data are not included in this analysis.

Analysis

A paired-samples t test was used to measure changes in provider confidence levels. Data were aggregated across fellows, resulting in a mean. A chi-square test of independence was performed to examine the relationship between rates of FRMP adoption by select provider groups. Analysis included a pre- and post-intervention assessment of the fellows’ adoption of FRMP practices, as well as a comparison between the fellows’ practice patterns and those of a control group of PCPs in the organization’s other clinics who did not participate in the mini-fellowship (nontrained control group). Excluded from the control group were providers from the same clinic as the fellows; providers in clinics with a geriatric-trained provider on staff; and clinics outside of the Portland metro and Medford service areas. We used an alpha level of 0.05 for all statistical tests.

Data from 5 providers were included in the analysis of the FRMP adoption. The sixth provider changed practice settings from the clinic to the ED after completing the fellowship; her patient data were not included in the FRMP part of the analysis. EHR data included data on all visits of patients 65+, as well as data for just those 65+ patients who had been identified as being at high risk to fall based on a STEADI score of 4 or higher.

Results

Provider Questionnaire

All 6 providers responded to the pre-intervention and post-intervention tests. For the knowledge questions, fellows, as a composite, correctly answered 57% of the questions before the intervention and 79% after the intervention. Provider confidence level in delivering fall risk care was measured prior to the training (mean, 4.12 [SD, 0.62]) and at the end of the training (mean, 6.47 [SD, 0.45]), demonstrating a significant increase in confidence (t (5) = –10.46, P < 0.001).

Qualitative Comments

Providers also had the opportunity to provide comments on their experience during the Mobility week and at the end of 1 year. In general, the simulated interdisciplinary fall risk clinic was highly rated (“the highlight of the week”) as a practical strategy to embed learning principles. One fellow commented, “Putting the learning into practice helps solidify it in my brain.” Fellows also appreciated the opportunity to learn and meet with their clinic colleagues to begin work on a fall-risk focused PIP and to “have a framework for what to do for people who screen positive [for fall risk].”

FRMP Adoption

A comparison of the care the fellows provided to their patients 65+ in the 12 months pre- and post-training shows the fellows demonstrated significant changes in practice patterns. The fellows were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to check orthostatic blood pressure than prior to the mini-fellowship (Table 1).

The control providers also demonstrated significant increases in fall risk screening and discussion of fall risk between the pre- and post-intervention periods; however, the relative risk (RR) was between 1.10 and 1.13 for this group. For the control group, checking orthostatic blood pressure did not significantly change. In the 12 months after training (Table 2), the fellows were 4.2 times more likely to discuss fall risk and almost 5 times more likely to check orthostatic blood pressure than their nontrained peers for all of their patients 65+, regardless of their risk to fall.

As shown in Table 3, for those patients determined to be at high risk of falling (STEADI score ≥ 4), fellows showed statistically significant increases in fall risk visits (RR, 3.02) and assessment of orthostatic blood pressure (RR, 10.68) before and after the mini-fellowship. The control providers did not show any changes in practice patterns between the pre- and post-period among patients at high risk to fall.

Neither the fellows nor the control group showed changes in patterns of referral to PT. In comparing the 2 groups in the 12 months after training (Table 4), for their patients at risk of falling, the fellows were 4 times more likely to complete fall risk visits and over 6 times more likely to assess orthostatic blood pressure than their nontrained peers. Subgroup analysis of the 75+ population revealed similar trends and significance, but these results are not included here.

Discussion

This study aimed to improve not only providers’ knowledge and confidence in caring for older adults at increased risk to fall, but also their clinical practice in assessing and managing fall risk. In addition to improved knowledge and confidence, we found that the fellows increased their discussion of fall risk (through fall risk visits) and their assessment of orthostatic blood pressure for all of their patients, not just for those identified at increased risk to fall. This improvement held true for the fellows themselves before and after the intervention, but also as compared to their nontrained peers. These practice improvements for all of their 65+ patients, not just those identified as being at high risk to fall, are especially important, since studies indicate that early screening and intervention can help identify people at risk and prevent future falls.¹⁵

We were surprised that there were no significant differences in PT referrals made by the trained fellows, but this finding may have been confounded by the fact that the data included all PT referrals, regardless of diagnosis, not just those referrals that were fall-related. Furthermore, our baseline PT referral rates, at 39% for the intervention group and 42% for the control group, are higher than national data when looking at rehabilitation use by older adults.²⁶

In comparison to a study evaluating the occurrence of fall risk–related clinical practice in primary care before any fall-related educational intervention, orthostatics were checked less frequently in our study (10% versus 30%) and there were fewer PT referrals (42%–44% versus 53%).²⁷ However, the Phelan study took place in patients who had actually had a fall, rather than just having a higher risk for a fall, and was based on detailed chart review. Other studies^23,24 found higher rates of fall risk interventions, but did not break out PT referrals specifically.

In terms of the educational intervention itself, most studies of geriatric education interventions have measured changes in knowledge, confidence, or self-efficacy as they relate to geriatric competence,^28-30 and do not measure practice change as an outcome outside of intent to change or self-reported practice change.^31,32 In general, practice change or longer-term health care–related outcomes have not been studied. Additionally, a range of dosages of educational interventions has been studied, from 1-hour lunchtime presentations^23,32 to half-day²⁹ or several half-day workshops,²⁸ up to 160 hours over 10 months³⁰ or 5 weekends over 6 months.³¹ The duration of our entire intervention at 160 hours over 6 months would be considered on the upper end of dosing relative to these studies, with our Mobility week intervention comprising 32 hours during 1 week. In the Warshaw study, despite 107 1-hour sessions being taught to over 60 physicians in 16 practices over 4 years, only 2 practices ultimately initiated any practice change projects.³² We believe that only curricula that embed practice change skills and opportunities, at a significant enough dose, can actually impact practice change in a sustainable manner.

Knowledge and skill acquisition among individual providers does not take place to a sufficient degree in the current health care arena, which is focused on productivity and short visit times. Consistent with other studies, we included interdisciplinary members of the primary care team for part of the mini-fellowship, although other studies used models that train across disciplines for the entirety of the learning experience.^28-30,33 Our educational model was strengthened by including other professionals to provide some of the education and model the ideal geriatric team, including PT, occupational therapy, and pharmacy, for the week on mobility.

Most studies exploring interventions through geriatric educational initiatives are conducted within academic institutions, with a primary focus on physician faculty and, by extension, their teaching of residents and others.^34,35 We believe our integrated model, which is steeped in community-based primary care practices like Lam’s,³¹ offers the greatest outreach to large community-based care systems and their patients. Training providers to work with their teams to change their own practices first gives skills and expertise that help further establish them as geriatric champions within their practices, laying the groundwork for more widespread practice change at their clinics.

Limitations

In addition to the limitations described above relating to the capture of PT referrals, other limitations included the relatively short time period for follow-up data as well as the small size of the intervention group. However, we found value in the instructional depth that the small group size allowed.

While the nontrained providers did show some improvement during the same period, we believe the relative risk was not clinically significant. We suspect that the larger health system efforts to standardize screening of patients 65+ across all clinics as a core quality metric confounded these results. The data analysis also included only fall-related patient visits that occurred with a provider who was that patient’s PCP, which could have missed visits done by other PCP colleagues, RNs, or pharmacists in the same clinic, thus undercounting the true number of fall-related visits. Furthermore, counting of fall-related interventions relied upon providers documenting consistently in the EHR, which could also lead to under-represention of fall risk clinical efforts.

The data presented, while encouraging, do not reflect clinic-wide practice change patterns and are considered only proximate outcomes rather than more long-term or cost-related outcomes, as would be captured by fall-related utilization measures like emergency room visits and hospitalizations. We expect to evaluate the broader impact and these value-based outcomes in the future. All providers and teams were from the same health care system, which may not allow our results to transfer to other organizations or regions of clinical practice.

Summary

This study demonstrates that an intensive mini-fellowship model of geriatrics training improved both knowledge and confidence in the realm of fall risk assessment and intervention among PCPs who had not been formally trained in geriatrics. More importantly, the training improved the fall-related care of their patients at increased risk to fall, but also of all of their older patients, with improvements in care measured up to a year after the mini-fellowship. Although this article only describes the work done as part of the Mobility aim of the 4M AFHS model, we believe the entire mini-fellowship curriculum offers the opportunity to “geriatricize” clinicians and their teams in learning geriatric principles and skills that they can translate into their practice in a sustainable way, as Tinetti encourages.⁸ Future study to evaluate other process outcomes more precisely, such as PT, as well as cost- and value-based outcomes, and the influence of trained providers on their clinic partners, will further establish the value proposition of targeted, disseminated, intensive geriatrics training of primary care clinicians as a strategy of age-friendly health systems as they work to improve the care of their older adults.

Acknowledgment: We are grateful for the dedication and hard work of the 2018 Geriatric Mini-Fellowship fellows at Providence Health & Services-Oregon who made this article possible. Thanks to Drs. Stephanie Cha, Emily Puukka-Clark, Laurie Dutkiewicz, Cara Ellis, Deb Frost, Jordan Roth, and Subhechchha Shah for promoting the AFHS work within their Providence Medical Group clinics and to PMG leadership and the fellows’ clinical teams for supporting the fellows, the AFHS work, and their older patients.

Corresponding author: Colleen M. Casey, PhD, ANP-BC, Providence Health & Services, Senior Health Program, 4400 NE Halsey, 5th Floor, Portland, OR 97213; [email protected].

Financial disclosures: None.

From the Senior Health Program, Providence Health & Services, Oregon, Portland, OR.

Abstract

Background: Approximately 51 million adults in the United States are 65 years of age or older, yet few geriatric-trained primary care providers (PCP) serve this population. The Age-Friendly Health System framework, consisting of evidence-based 4M care (Mobility, Medication, Mentation, and what Matters), encourages all PCPs to assess mobility in older adults.

Objective: To improve PCP knowledge, confidence, and clinical practice in assessing and managing fall risk.

Methods: A 1-week educational session focusing on mobility (part of a 4-week Geriatric Mini-Fellowship) for 6 selected PCPs from a large health care system was conducted to increase knowledge and ability to address fall risk in older adults. The week included learning and practicing a Fall Risk Management Plan (FRMP) algorithm, including planning for their own practice changes. Pre- and post-test surveys assessed changes in knowledge and confidence. Patient data were compared 12 months before and after training to evaluate PCP adoption of FRMP components.

Results: The training increased provider knowledge and confidence. The trained PCPs were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to assess orthostatic blood pressure in their 65+ patients after the mini-fellowship. In high-risk patients, they were 4.1 times more likely to discuss fall risk and 6.3 times more likely to assess orthostatic blood pressure than their nontrained peers. Changes in physical therapy referral rates were not observed.

Conclusions: In-depth, skills-based geriatric educational sessions improved PCPs’ knowledge and confidence and also improved their fall risk management practices for their older patients.

Keywords: geriatrics; guidelines; Age-Friendly Health System; 4M; workforce training; practice change; fellowship.

The US population is aging rapidly. People aged 85 years and older are the largest-growing segment of the US population, and this segment is expected to increase by 123% by 2040.¹ Caregiving needs increase with age as older adults develop more chronic conditions, such as hypertension, heart disease, arthritis, and dementia. However, even with increasing morbidity and dependence, a majority of older adults still live in the community rather than in institutional settings.² These older adults seek medical care more frequently than younger people, with about 22% of patients 75 years and older having 10 or more health care visits in the previous 12 months. By 2040, nearly a quarter of the US population is expected to be 65 or older, with many of these older adults seeking regular primary care from providers who do not have formal training in the care of a population with multiple complex, chronic health conditions and increased caregiving needs.¹

Despite this growing demand for health care professionals trained in the care of older adults, access to these types of clinicians is limited. In 2018, there were roughly 7000 certified geriatricians, with only 3600 of them practicing full-time.^3,4 Similarly, of 290,000 certified nurse practitioners (NPs), about 9% of them have geriatric certification.⁵ Geriatricians, medical doctors trained in the care of older adults, and geriatric-trained NPs are part of a cadre of a geriatric-trained workforce that provides unique expertise in caring for older adults with chronic and advanced illness. They know how to manage multiple, complex geriatric syndromes like falls, dementia, and polypharmacy; understand and maximize team-based care; and focus on caring for an older person with a goal-centered versus a disease-centered approach.⁶

Broadly, geriatric care includes a spectrum of adults, from those who are aging healthfully to those who are the frailest. Research has suggested that approximately 30% of older adults need care by a geriatric-trained clinician, with the oldest and frailest patients needing more clinician time for assessment and treatment, care coordination, and coaching of caregivers.⁷ With this assumption in mind, it is projected that by 2025, there will be a national shortage of 26,980 geriatricians, with the western United States disproportionately affected by this shortage.⁴Rather than lamenting this shortage, Tinetti recommends a new path forward: “Our mission should not be to train enough geriatricians to provide direct care, but rather to ensure that every clinician caring for older adults is competent in geriatric principles and practices.”⁸ Sometimes called ”geriatricizing,” the idea is to use existing geriatric providers as a small elite training force to infuse geriatric principles and skills across their colleagues in primary care and other disciplines.^8,9 Efforts of the American Geriatrics Society (AGS), with support from the John A. Hartford Foundation (JAHF), have been successful in developing geriatric training across multiple specialties, including surgery, orthopedics, and emergency medicine (www.americangeriatrics.org/programs/geriatrics-specialists-initiative).

The Age-Friendly Health System and 4M Model

To help augment this idea of equipping health care systems and their clinicians with more readily available geriatric knowledge, skills, and tools, the JAHF, along with the Institute for Healthcare Improvement (IHI), created the Age-Friendly Health System (AFHS) paradigm in 2015.¹⁰ Using the 4M model, the AFHS initiative established a set of evidence-based geriatric priorities and interventions meant to improve the care of older adults, reduce harm and duplication, and provide a framework for engaging leadership, clinical teams, and operational systems across inpatient and ambulatory settings.¹¹ Mobility, including fall risk screening and intervention, is 1 of the 4M foundational elements of the Age-Friendly model. In addition to Mobility, the 4M model also includes 3 other key geriatric domains: Mentation (dementia, depression, and delirium), Medication (high-risk medications, polypharmacy, and deprescribing), and What Matters (goals of care conversations and understanding quality of life for older patients).¹¹ The 4M initiative encourages adoption of a geriatric lens that looks across chronic conditions and accounts for the interplay among geriatric syndromes, such as falls, cognitive impairment, and frailty, in order to provide care better tailored to what the patient needs and desires.¹² IHI and JAHF have targeted the adoption of the 4M model by 20% of US health care systems by 2020.¹¹

Mini-Fellowship and Mobility Week

To bolster geriatric skills among community-based primary care providers (PCPs), we initiated a Geriatric Mini-Fellowship, a 4-week condensed curriculum taught over 6 months. Each week focuses on 1 of the age-friendly 4Ms, with the goal of increasing the knowledge, self-efficacy, skills, and competencies of the participating PCPs (called “fellow” hereafter) and at the same time, equipping each to become a champion of geriatric practice. This article focuses on the Mobility week, the second week of the mini-fellowship, and the effect of the week on the fellows’ practice changes.

To construct the Mobility week’s curriculum with a focus on the ambulatory setting, we relied upon national evidence-based work in fall risk management. The Centers for Disease Control and Prevention (CDC) has made fall risk screening and management in primary care a high priority. Using the clinical practice guidelines for managing fall risk developed by the American and British Geriatrics Societies (AGS/BGS), the CDC developed the Stopping Elderly Accidents, Deaths, and Injuries (STEADI) toolkit.¹³ Foundational to the toolkit is the validated 12-item Stay Independent falls screening questionnaire (STEADI questionnaire).¹⁴ Patients who score 4 or higher (out of a total score of 14) on the questionnaire are considered at increased risk of falling. The CDC has developed a clinical algorithm that guides clinical teams through screening and assessment to help identify appropriate interventions to target specific risk factors. Research has clearly established that a multifactorial approach to fall risk intervention can be successful in reducing fall risk by as much as 25%.^15-17

The significant morbidity and mortality caused by falls make training nongeriatrician clinicians on how to better address fall risk imperative. More than 25% of older adults fall each year.¹⁸ These falls contribute to rising rates of fall-related deaths,¹⁹ emergency department (ED) visits,²⁰ and hospital readmissions.²¹ Initiatives like the AFHS focus on mobility and the CDC’s development of supporting clinical materials²² aim to improve primary care adoption of fall risk screening and intervention practices.^23,24 The epidemic of falls must compel all PCPs, not just those practicing geriatrics, to make discussing and addressing fall risk and falls a priority.

Methods

Setting

This project took place as part of a regional primary care effort in Oregon. Providence Health & Services-Oregon is part of a multi-state integrated health care system in the western United States whose PCPs serve more than 80,000 patients aged 65 years and older per year; these patients comprise 38% of the system’s office visits each year. Regionally, there are 47 family and internal medicine clinics employing roughly 290 providers (physicians, NPs, and physician assistants). The organization has only 4 PCPs trained in geriatrics and does not offer any geriatric clinical consultation services. Six PCPs from different clinics, representing both rural and urban settings, are chosen to participate in the geriatric mini-fellowship each year.

This project was conducted as a quality improvement initiative within the organization and did not constitute human subjects research. It was not conducted under the oversight of the Institutional Review Board.

Intervention

The mini-fellowship was taught in 4 1-week blocks between April and October 2018, with a curriculum designed to be interactive and practical. The faculty was intentionally interdisciplinary to teach and model team-based practice. Each week participants were excused from their clinical practice. Approximately 160 hours of continuing medical education credits were awarded for the full mini-fellowship. As part of each weekly session, a performance improvement project (PIP) focused on that week’s topic (1 of the 4Ms) was developed by the fellow and their team members to incorporate the mini-fellowship learnings into their clinic workflows. Fellows also had 2 hours per week of dedicated administration time for a year, outside the fellowship, to work on their PIP and 4M practice changes within their clinic.

Provider Education

The week for mobility training comprised 4 daylong sessions. The first 2 days were spent learning about the epidemiology of falls; risk factors for falling; how to conduct a thorough history and assessment of fall risk; and how to create a prioritized Fall Risk Management Plan (FRMP) to decrease a patient’s individual fall risk through tailored interventions. The FRMP was adapted from the CDC STEADI toolkit.¹³ Core faculty were 2 geriatric-trained providers (NP and physician) and a physical therapist (PT) specializing in fall prevention.

On the third day, fellows took part in a simulated fall risk clinic, in which older adults volunteered to be patient partners, providing an opportunity to apply learnings from days 1 and 2. The clinic included the fellow observing a PT complete a mobility assessment and a pharmacist conduct a high-risk medication review. The fellow synthesized the findings of the mobility assessment and medication review, as well as their own history and assessment, to create a summary of fall risk recommendations to discuss with their volunteer patient partner. The fellows were observed and evaluated in their skills by their patient partner, course faculty, and another fellow. The patient partners, and their assigned fellow, also participated in a 45-minute fall risk presentation, led by a nurse.

On the fourth day, the fellows were joined by select clinic partners, including nurses, pharmacists, and/or medical assistants. The session included discussions among each fellow’s clinical team regarding the current state of fall risk efforts at their clinic, an analysis of barriers, and identification of opportunities to improve workflows and screening rates. Each fellow took with them an action plan tailored to their clinic to improve fall risk management practices, starting with the fellow’s own practice.

Fall Risk Management Plan

The educational sessions introduced the fellows to the FRMP. The FRMP, adapted from the STEADI toolkit, includes a process for fall risk screening (Figure 1) and stratifying a patient’s risk based on their STEADI score in order to promote 3 priority assessments (gait evaluation with PT referral if appropriate; orthostatic blood pressure; and high-risk medication review; Figure 2). Initial actions based on these priority assessments were followed over time, with additional fall risk interventions added as clinically indicated.²⁵ The FRMP is intended to be used during routine office visits, Medicare annual wellness visits, or office visits focused on fall risk or related medical disorders (ie, fall risk visits.)

Providers and their teams were encouraged to spread out fall-related conversations with their patients over multiple visits, since many patients have multiple fall risk factors at play, in addition to other chronic medical issues, and since many interventions often require behavior changes on the part of the patient. Providers also had access to fall-related electronic health record (EHR) templates as well as a comprehensive, internal fall risk management website that included assessment tools, evidence-based resources, and patient handouts.

Assessment and Measurements

We assessed provider knowledge and comfort in their fall risk evaluation and management skills before and after the educational intervention using an 11-item multiple-choice questionnaire and a 4-item confidence questionnaire. The confidence questions used a 7-point Likert scale, with 0 indicating “no confidence” and 7 indicating ”lots of confidence.” The questions were administered via a paper survey. Qualitative comments were derived from evaluations completed at the end of the week.

The fellows’ practice of fall risk screening and management was studied from May 2018, at the completion of Mobility week, to May 2019 for the post-intervention period. A 1-year timeframe before May 2018 was used as the pre-intervention period. Eligible visit types, during which we assumed fall risk was discussed, were any office visits for patients 65+ completed by the patients’ PCPs that used fall risk as a reason for the visit or had a fall-related diagnosis code. Fall risk visits performed by other clinic providers were not counted.

Of those patients who had fall risk screenings completed and were determined to be high risk (STEADI score ≥ 4), data were analyzed to determine whether these patients had any fall-related follow-up visits to their PCP within 60 days of the STEADI screening. For these high-risk patients, data were studied to understand whether orthostatic blood pressure measurements were performed (as documented in a flowsheet) and whether a PT referral was placed. These data were compared with those from providers who practiced in clinics within the same system but who did not participate in the mini-fellowship. Data were obtained from the organization’s EHR. Additional data were measured to evaluate patterns of deprescribing of select high-risk medications, but these data are not included in this analysis.

Analysis

A paired-samples t test was used to measure changes in provider confidence levels. Data were aggregated across fellows, resulting in a mean. A chi-square test of independence was performed to examine the relationship between rates of FRMP adoption by select provider groups. Analysis included a pre- and post-intervention assessment of the fellows’ adoption of FRMP practices, as well as a comparison between the fellows’ practice patterns and those of a control group of PCPs in the organization’s other clinics who did not participate in the mini-fellowship (nontrained control group). Excluded from the control group were providers from the same clinic as the fellows; providers in clinics with a geriatric-trained provider on staff; and clinics outside of the Portland metro and Medford service areas. We used an alpha level of 0.05 for all statistical tests.

Data from 5 providers were included in the analysis of the FRMP adoption. The sixth provider changed practice settings from the clinic to the ED after completing the fellowship; her patient data were not included in the FRMP part of the analysis. EHR data included data on all visits of patients 65+, as well as data for just those 65+ patients who had been identified as being at high risk to fall based on a STEADI score of 4 or higher.

Results

Provider Questionnaire

All 6 providers responded to the pre-intervention and post-intervention tests. For the knowledge questions, fellows, as a composite, correctly answered 57% of the questions before the intervention and 79% after the intervention. Provider confidence level in delivering fall risk care was measured prior to the training (mean, 4.12 [SD, 0.62]) and at the end of the training (mean, 6.47 [SD, 0.45]), demonstrating a significant increase in confidence (t (5) = –10.46, P < 0.001).

Qualitative Comments

Providers also had the opportunity to provide comments on their experience during the Mobility week and at the end of 1 year. In general, the simulated interdisciplinary fall risk clinic was highly rated (“the highlight of the week”) as a practical strategy to embed learning principles. One fellow commented, “Putting the learning into practice helps solidify it in my brain.” Fellows also appreciated the opportunity to learn and meet with their clinic colleagues to begin work on a fall-risk focused PIP and to “have a framework for what to do for people who screen positive [for fall risk].”

FRMP Adoption

A comparison of the care the fellows provided to their patients 65+ in the 12 months pre- and post-training shows the fellows demonstrated significant changes in practice patterns. The fellows were 1.7 times more likely to screen for fall risk; 3.6 times more likely to discuss fall risk; and 5.8 times more likely to check orthostatic blood pressure than prior to the mini-fellowship (Table 1).

The control providers also demonstrated significant increases in fall risk screening and discussion of fall risk between the pre- and post-intervention periods; however, the relative risk (RR) was between 1.10 and 1.13 for this group. For the control group, checking orthostatic blood pressure did not significantly change. In the 12 months after training (Table 2), the fellows were 4.2 times more likely to discuss fall risk and almost 5 times more likely to check orthostatic blood pressure than their nontrained peers for all of their patients 65+, regardless of their risk to fall.

As shown in Table 3, for those patients determined to be at high risk of falling (STEADI score ≥ 4), fellows showed statistically significant increases in fall risk visits (RR, 3.02) and assessment of orthostatic blood pressure (RR, 10.68) before and after the mini-fellowship. The control providers did not show any changes in practice patterns between the pre- and post-period among patients at high risk to fall.

Neither the fellows nor the control group showed changes in patterns of referral to PT. In comparing the 2 groups in the 12 months after training (Table 4), for their patients at risk of falling, the fellows were 4 times more likely to complete fall risk visits and over 6 times more likely to assess orthostatic blood pressure than their nontrained peers. Subgroup analysis of the 75+ population revealed similar trends and significance, but these results are not included here.

Discussion

This study aimed to improve not only providers’ knowledge and confidence in caring for older adults at increased risk to fall, but also their clinical practice in assessing and managing fall risk. In addition to improved knowledge and confidence, we found that the fellows increased their discussion of fall risk (through fall risk visits) and their assessment of orthostatic blood pressure for all of their patients, not just for those identified at increased risk to fall. This improvement held true for the fellows themselves before and after the intervention, but also as compared to their nontrained peers. These practice improvements for all of their 65+ patients, not just those identified as being at high risk to fall, are especially important, since studies indicate that early screening and intervention can help identify people at risk and prevent future falls.¹⁵

We were surprised that there were no significant differences in PT referrals made by the trained fellows, but this finding may have been confounded by the fact that the data included all PT referrals, regardless of diagnosis, not just those referrals that were fall-related. Furthermore, our baseline PT referral rates, at 39% for the intervention group and 42% for the control group, are higher than national data when looking at rehabilitation use by older adults.²⁶

In comparison to a study evaluating the occurrence of fall risk–related clinical practice in primary care before any fall-related educational intervention, orthostatics were checked less frequently in our study (10% versus 30%) and there were fewer PT referrals (42%–44% versus 53%).²⁷ However, the Phelan study took place in patients who had actually had a fall, rather than just having a higher risk for a fall, and was based on detailed chart review. Other studies^23,24 found higher rates of fall risk interventions, but did not break out PT referrals specifically.

In terms of the educational intervention itself, most studies of geriatric education interventions have measured changes in knowledge, confidence, or self-efficacy as they relate to geriatric competence,^28-30 and do not measure practice change as an outcome outside of intent to change or self-reported practice change.^31,32 In general, practice change or longer-term health care–related outcomes have not been studied. Additionally, a range of dosages of educational interventions has been studied, from 1-hour lunchtime presentations^23,32 to half-day²⁹ or several half-day workshops,²⁸ up to 160 hours over 10 months³⁰ or 5 weekends over 6 months.³¹ The duration of our entire intervention at 160 hours over 6 months would be considered on the upper end of dosing relative to these studies, with our Mobility week intervention comprising 32 hours during 1 week. In the Warshaw study, despite 107 1-hour sessions being taught to over 60 physicians in 16 practices over 4 years, only 2 practices ultimately initiated any practice change projects.³² We believe that only curricula that embed practice change skills and opportunities, at a significant enough dose, can actually impact practice change in a sustainable manner.

Knowledge and skill acquisition among individual providers does not take place to a sufficient degree in the current health care arena, which is focused on productivity and short visit times. Consistent with other studies, we included interdisciplinary members of the primary care team for part of the mini-fellowship, although other studies used models that train across disciplines for the entirety of the learning experience.^28-30,33 Our educational model was strengthened by including other professionals to provide some of the education and model the ideal geriatric team, including PT, occupational therapy, and pharmacy, for the week on mobility.

Most studies exploring interventions through geriatric educational initiatives are conducted within academic institutions, with a primary focus on physician faculty and, by extension, their teaching of residents and others.^34,35 We believe our integrated model, which is steeped in community-based primary care practices like Lam’s,³¹ offers the greatest outreach to large community-based care systems and their patients. Training providers to work with their teams to change their own practices first gives skills and expertise that help further establish them as geriatric champions within their practices, laying the groundwork for more widespread practice change at their clinics.

Limitations

In addition to the limitations described above relating to the capture of PT referrals, other limitations included the relatively short time period for follow-up data as well as the small size of the intervention group. However, we found value in the instructional depth that the small group size allowed.

While the nontrained providers did show some improvement during the same period, we believe the relative risk was not clinically significant. We suspect that the larger health system efforts to standardize screening of patients 65+ across all clinics as a core quality metric confounded these results. The data analysis also included only fall-related patient visits that occurred with a provider who was that patient’s PCP, which could have missed visits done by other PCP colleagues, RNs, or pharmacists in the same clinic, thus undercounting the true number of fall-related visits. Furthermore, counting of fall-related interventions relied upon providers documenting consistently in the EHR, which could also lead to under-represention of fall risk clinical efforts.

The data presented, while encouraging, do not reflect clinic-wide practice change patterns and are considered only proximate outcomes rather than more long-term or cost-related outcomes, as would be captured by fall-related utilization measures like emergency room visits and hospitalizations. We expect to evaluate the broader impact and these value-based outcomes in the future. All providers and teams were from the same health care system, which may not allow our results to transfer to other organizations or regions of clinical practice.

Summary

This study demonstrates that an intensive mini-fellowship model of geriatrics training improved both knowledge and confidence in the realm of fall risk assessment and intervention among PCPs who had not been formally trained in geriatrics. More importantly, the training improved the fall-related care of their patients at increased risk to fall, but also of all of their older patients, with improvements in care measured up to a year after the mini-fellowship. Although this article only describes the work done as part of the Mobility aim of the 4M AFHS model, we believe the entire mini-fellowship curriculum offers the opportunity to “geriatricize” clinicians and their teams in learning geriatric principles and skills that they can translate into their practice in a sustainable way, as Tinetti encourages.⁸ Future study to evaluate other process outcomes more precisely, such as PT, as well as cost- and value-based outcomes, and the influence of trained providers on their clinic partners, will further establish the value proposition of targeted, disseminated, intensive geriatrics training of primary care clinicians as a strategy of age-friendly health systems as they work to improve the care of their older adults.

Acknowledgment: We are grateful for the dedication and hard work of the 2018 Geriatric Mini-Fellowship fellows at Providence Health & Services-Oregon who made this article possible. Thanks to Drs. Stephanie Cha, Emily Puukka-Clark, Laurie Dutkiewicz, Cara Ellis, Deb Frost, Jordan Roth, and Subhechchha Shah for promoting the AFHS work within their Providence Medical Group clinics and to PMG leadership and the fellows’ clinical teams for supporting the fellows, the AFHS work, and their older patients.

Corresponding author: Colleen M. Casey, PhD, ANP-BC, Providence Health & Services, Senior Health Program, 4400 NE Halsey, 5th Floor, Portland, OR 97213; [email protected].

Financial disclosures: None.

Early in the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the World Health Organization issued guidance for coronavirus disease 2019 (COVID-19) management.¹ Based on a high intubation rate among 12 subjects with Middle Eastern respiratory syndrome, noninvasive ventilation (NIV) was discouraged.² While high-flow nasal oxygen (HFNO) was recognized as a reasonable strategy to avoid endotracheal intubation,¹ uncertainty regarding the potential of both therapies to aerosolize SARS-CoV-2 and reports of rapid, unexpected respiratory decompensations were deterrents to use.³ As hospitals prepared for a surge of patients, reports of SARS-CoV-2 transmission to healthcare personnel also emerged. Together, these issues led many institutions to recommend lower than usual thresholds for intubation. This well-intentioned guidance was based on limited historical data, a rapidly evolving literature that frequently appeared on preprint servers before peer review, or as anecdotes on social media.

As COVID-19 caseloads increased, clinicians were immediately faced with patients who rapidly reached the planned intubation threshold, but also looked very comfortable with minimal to no use of accessory muscles of respiration. In addition, the pace of respiratory decompensation among those who ultimately required intubation was slower than expected. Moreover, intensive care unit (ICU) capacity was stretched thin, raising concern for an imminent need for ventilator rationing. Lastly, the risk of SARS-CoV-2 transmission to healthcare workers appeared well-controlled with the use of personal protective equipment.⁴

In light of this accumulating experience, sites worldwide evolved quickly from their initial management strategies for COVID-19 respiratory failure. However, the deliberate process described by Soares et al in this issue of the Journal of Hospital Medicine is notable.⁵ Their transition towards the beginning of the pandemic from a conservative early intubation approach to a new strategy that encouraged use of NIV, HFNO, and self-proning is described. They were motivated by reports of good outcomes using these interventions, high mortality in intubated patients, and reassurance that aerosolization of respiratory secretions during NIV and HFNO was comparable to regular nasal cannula or face mask oxygen.³ The new protocol was defined and rapidly deployed over 4 days using multipronged communication from project and institutional leaders via in person and electronic means (email, Whatsapp, GoogleDrive). To facilitate implementation, COVID-19 patients requiring respiratory support were placed in dedicated units with bedside flowsheets for guidance. An immediate impact was demonstrated over the next 2 weeks by a significant decrease in use of mechanical ventilation in COVID-19 patients from 25.2% to 10.7%. In-hospital mortality, the primary outcome, did not change, ICU admissions decreased, as did hospital length of stay (10 vs 8.4 days, though not statistically significant), all providing supportive evidence for relative safety of the new protocol.

Soares et al exemplify a nimble system that recognized planned strategies to be problematic, and then achieved rapid implementation of a new protocol across a four-hospital system. Changes in medical practice are typically much slower, with some studies suggesting this process may take a decade or more. Implementation science focuses on translating research evidence into clinical practice using strategies tailored to particular contexts. The current study harnessed important implementation principles to quickly translate evidence into practice using effective engagement and education of key stakeholders across specialties (eg, emergency medicine, hospitalists, critical care, and respiratory therapy), the identification of pathways that mitigated barriers, frequent re-evaluation of a rapidly evolving literature, and an open-mindedness to the value of change.⁶ As the pandemic continues, traditional research and implementation science are critical not only to define optimal treatments and management strategies, but also to learn how best to implement successful interventions in an accelerated manner.⁷

The authors reported no conflicts of interest.

Dr Hochberg is supported by a National Institutes of Health training grant (T32HL007534).

Early in the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the World Health Organization issued guidance for coronavirus disease 2019 (COVID-19) management.¹ Based on a high intubation rate among 12 subjects with Middle Eastern respiratory syndrome, noninvasive ventilation (NIV) was discouraged.² While high-flow nasal oxygen (HFNO) was recognized as a reasonable strategy to avoid endotracheal intubation,¹ uncertainty regarding the potential of both therapies to aerosolize SARS-CoV-2 and reports of rapid, unexpected respiratory decompensations were deterrents to use.³ As hospitals prepared for a surge of patients, reports of SARS-CoV-2 transmission to healthcare personnel also emerged. Together, these issues led many institutions to recommend lower than usual thresholds for intubation. This well-intentioned guidance was based on limited historical data, a rapidly evolving literature that frequently appeared on preprint servers before peer review, or as anecdotes on social media.

As COVID-19 caseloads increased, clinicians were immediately faced with patients who rapidly reached the planned intubation threshold, but also looked very comfortable with minimal to no use of accessory muscles of respiration. In addition, the pace of respiratory decompensation among those who ultimately required intubation was slower than expected. Moreover, intensive care unit (ICU) capacity was stretched thin, raising concern for an imminent need for ventilator rationing. Lastly, the risk of SARS-CoV-2 transmission to healthcare workers appeared well-controlled with the use of personal protective equipment.⁴

In light of this accumulating experience, sites worldwide evolved quickly from their initial management strategies for COVID-19 respiratory failure. However, the deliberate process described by Soares et al in this issue of the Journal of Hospital Medicine is notable.⁵ Their transition towards the beginning of the pandemic from a conservative early intubation approach to a new strategy that encouraged use of NIV, HFNO, and self-proning is described. They were motivated by reports of good outcomes using these interventions, high mortality in intubated patients, and reassurance that aerosolization of respiratory secretions during NIV and HFNO was comparable to regular nasal cannula or face mask oxygen.³ The new protocol was defined and rapidly deployed over 4 days using multipronged communication from project and institutional leaders via in person and electronic means (email, Whatsapp, GoogleDrive). To facilitate implementation, COVID-19 patients requiring respiratory support were placed in dedicated units with bedside flowsheets for guidance. An immediate impact was demonstrated over the next 2 weeks by a significant decrease in use of mechanical ventilation in COVID-19 patients from 25.2% to 10.7%. In-hospital mortality, the primary outcome, did not change, ICU admissions decreased, as did hospital length of stay (10 vs 8.4 days, though not statistically significant), all providing supportive evidence for relative safety of the new protocol.

Soares et al exemplify a nimble system that recognized planned strategies to be problematic, and then achieved rapid implementation of a new protocol across a four-hospital system. Changes in medical practice are typically much slower, with some studies suggesting this process may take a decade or more. Implementation science focuses on translating research evidence into clinical practice using strategies tailored to particular contexts. The current study harnessed important implementation principles to quickly translate evidence into practice using effective engagement and education of key stakeholders across specialties (eg, emergency medicine, hospitalists, critical care, and respiratory therapy), the identification of pathways that mitigated barriers, frequent re-evaluation of a rapidly evolving literature, and an open-mindedness to the value of change.⁶ As the pandemic continues, traditional research and implementation science are critical not only to define optimal treatments and management strategies, but also to learn how best to implement successful interventions in an accelerated manner.⁷

The authors reported no conflicts of interest.

Dr Hochberg is supported by a National Institutes of Health training grant (T32HL007534).

Early in the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the World Health Organization issued guidance for coronavirus disease 2019 (COVID-19) management.¹ Based on a high intubation rate among 12 subjects with Middle Eastern respiratory syndrome, noninvasive ventilation (NIV) was discouraged.² While high-flow nasal oxygen (HFNO) was recognized as a reasonable strategy to avoid endotracheal intubation,¹ uncertainty regarding the potential of both therapies to aerosolize SARS-CoV-2 and reports of rapid, unexpected respiratory decompensations were deterrents to use.³ As hospitals prepared for a surge of patients, reports of SARS-CoV-2 transmission to healthcare personnel also emerged. Together, these issues led many institutions to recommend lower than usual thresholds for intubation. This well-intentioned guidance was based on limited historical data, a rapidly evolving literature that frequently appeared on preprint servers before peer review, or as anecdotes on social media.

As COVID-19 caseloads increased, clinicians were immediately faced with patients who rapidly reached the planned intubation threshold, but also looked very comfortable with minimal to no use of accessory muscles of respiration. In addition, the pace of respiratory decompensation among those who ultimately required intubation was slower than expected. Moreover, intensive care unit (ICU) capacity was stretched thin, raising concern for an imminent need for ventilator rationing. Lastly, the risk of SARS-CoV-2 transmission to healthcare workers appeared well-controlled with the use of personal protective equipment.⁴

In light of this accumulating experience, sites worldwide evolved quickly from their initial management strategies for COVID-19 respiratory failure. However, the deliberate process described by Soares et al in this issue of the Journal of Hospital Medicine is notable.⁵ Their transition towards the beginning of the pandemic from a conservative early intubation approach to a new strategy that encouraged use of NIV, HFNO, and self-proning is described. They were motivated by reports of good outcomes using these interventions, high mortality in intubated patients, and reassurance that aerosolization of respiratory secretions during NIV and HFNO was comparable to regular nasal cannula or face mask oxygen.³ The new protocol was defined and rapidly deployed over 4 days using multipronged communication from project and institutional leaders via in person and electronic means (email, Whatsapp, GoogleDrive). To facilitate implementation, COVID-19 patients requiring respiratory support were placed in dedicated units with bedside flowsheets for guidance. An immediate impact was demonstrated over the next 2 weeks by a significant decrease in use of mechanical ventilation in COVID-19 patients from 25.2% to 10.7%. In-hospital mortality, the primary outcome, did not change, ICU admissions decreased, as did hospital length of stay (10 vs 8.4 days, though not statistically significant), all providing supportive evidence for relative safety of the new protocol.

Soares et al exemplify a nimble system that recognized planned strategies to be problematic, and then achieved rapid implementation of a new protocol across a four-hospital system. Changes in medical practice are typically much slower, with some studies suggesting this process may take a decade or more. Implementation science focuses on translating research evidence into clinical practice using strategies tailored to particular contexts. The current study harnessed important implementation principles to quickly translate evidence into practice using effective engagement and education of key stakeholders across specialties (eg, emergency medicine, hospitalists, critical care, and respiratory therapy), the identification of pathways that mitigated barriers, frequent re-evaluation of a rapidly evolving literature, and an open-mindedness to the value of change.⁶ As the pandemic continues, traditional research and implementation science are critical not only to define optimal treatments and management strategies, but also to learn how best to implement successful interventions in an accelerated manner.⁷

The authors reported no conflicts of interest.

Dr Hochberg is supported by a National Institutes of Health training grant (T32HL007534).

Since 2012, when the Centers for Medicare & Medicaid Services (CMS) began linking financial penalties to hospitals with excessive readmissions for adult patients, researchers have questioned the extent to which pediatric readmissions can be used as a reliable quality measure. Compared with readmissions among adult patients, readmissions among pediatric patients are relatively uncommon. Furthermore, few (approximately 2%) qualify as potentially preventable, and pediatric readmission rates remain largely unchanged despite targeted attempts to prevent reutilization.^1,2 Nonetheless, state Medicaid agencies have continued to reduce reimbursement for hospitals based on available readmissions metrics, most commonly the Potentially Preventable Readmissions (PPR) algorithm.¹

In this issue of the Journal of Hospital Medicine, Auger et al³ performed a retrospective study to explore four existing metrics of pediatric hospital readmissions for their ability to identify preventable and unplanned readmissions. Investigators examined 30-day readmissions (n = 1,125) from 2014-2016 across multiple subspecialties, and classified readmissions by their preventability and unplanned status with use of a validated chart abstraction tool. Using the results of chart abstraction as the gold standard, investigators calculated the sensitivity and specificity, as well as estimated the positive and negative predictive values, of each readmissions metric. Auger and colleagues found that none of the four readmissions metrics could reliably assess preventability, and that only one metric reliably predicted unplanned hospital readmissions. Specifically, the commonly used PPR algorithm was estimated to have a positive predictive value of 13.0%-35.5% across a prevalence range of 10%-30%. This means that in a hospital where 10% of readmissions are truly preventable, the PPR will be wrong approximately 87% of the time. Tying payments to this metric is difficult to justify.

The authors highlighted the policy implications of the PPR falling short in its ability to identify preventable and unplanned pediatric readmissions. A good quality measure should be consistently reliable, and neither the PPR nor other measures studied meets this benchmark. Yet the findings lead to a broader conclusion: if most pediatric readmissions are not preventable, if there is no reliable way of measuring preventability, and if we have not demonstrated the ability to change patient trajectories away from reutilization, then perhaps the sun has set on using readmissions as a comprehensive quality measure for hospital-based care.

So how, then, should the hospital-to-home transition be evaluated? The paradigm of pediatric value of care is shifting to incorporate family-centered perspectives into consideration of quality measures.² There has to be a balance between healthcare costs and outcomes that affect families; measures should take into account issues such as patient and caregiver anxiety and time away from work.² Moreover, because social determinants of health and medical complexity strongly influence readmission rates,^4,5 focus should be placed on redirecting resources toward patients and families with significant medical, social, and financial needs as they transition home from the hospital. While measures of healthcare equity are currently lacking, the overall quality and equity of pediatric care transitions could be enhanced by looking beyond the narrow lens of readmission rates to incorporate actual needs assessments of families.

In summary, Auger and colleagues identified deficits in existing readmission metrics—but creating a solution that is meaningful to all stakeholders will be more complex than simply identifying a better metric. Family-centered quality metrics show promise in creating value in pediatric care within an equitable health system, but long-term evaluation of these metrics is necessary.

The authors have nothing to disclose.

Since 2012, when the Centers for Medicare & Medicaid Services (CMS) began linking financial penalties to hospitals with excessive readmissions for adult patients, researchers have questioned the extent to which pediatric readmissions can be used as a reliable quality measure. Compared with readmissions among adult patients, readmissions among pediatric patients are relatively uncommon. Furthermore, few (approximately 2%) qualify as potentially preventable, and pediatric readmission rates remain largely unchanged despite targeted attempts to prevent reutilization.^1,2 Nonetheless, state Medicaid agencies have continued to reduce reimbursement for hospitals based on available readmissions metrics, most commonly the Potentially Preventable Readmissions (PPR) algorithm.¹

In this issue of the Journal of Hospital Medicine, Auger et al³ performed a retrospective study to explore four existing metrics of pediatric hospital readmissions for their ability to identify preventable and unplanned readmissions. Investigators examined 30-day readmissions (n = 1,125) from 2014-2016 across multiple subspecialties, and classified readmissions by their preventability and unplanned status with use of a validated chart abstraction tool. Using the results of chart abstraction as the gold standard, investigators calculated the sensitivity and specificity, as well as estimated the positive and negative predictive values, of each readmissions metric. Auger and colleagues found that none of the four readmissions metrics could reliably assess preventability, and that only one metric reliably predicted unplanned hospital readmissions. Specifically, the commonly used PPR algorithm was estimated to have a positive predictive value of 13.0%-35.5% across a prevalence range of 10%-30%. This means that in a hospital where 10% of readmissions are truly preventable, the PPR will be wrong approximately 87% of the time. Tying payments to this metric is difficult to justify.

The authors highlighted the policy implications of the PPR falling short in its ability to identify preventable and unplanned pediatric readmissions. A good quality measure should be consistently reliable, and neither the PPR nor other measures studied meets this benchmark. Yet the findings lead to a broader conclusion: if most pediatric readmissions are not preventable, if there is no reliable way of measuring preventability, and if we have not demonstrated the ability to change patient trajectories away from reutilization, then perhaps the sun has set on using readmissions as a comprehensive quality measure for hospital-based care.

So how, then, should the hospital-to-home transition be evaluated? The paradigm of pediatric value of care is shifting to incorporate family-centered perspectives into consideration of quality measures.² There has to be a balance between healthcare costs and outcomes that affect families; measures should take into account issues such as patient and caregiver anxiety and time away from work.² Moreover, because social determinants of health and medical complexity strongly influence readmission rates,^4,5 focus should be placed on redirecting resources toward patients and families with significant medical, social, and financial needs as they transition home from the hospital. While measures of healthcare equity are currently lacking, the overall quality and equity of pediatric care transitions could be enhanced by looking beyond the narrow lens of readmission rates to incorporate actual needs assessments of families.

In summary, Auger and colleagues identified deficits in existing readmission metrics—but creating a solution that is meaningful to all stakeholders will be more complex than simply identifying a better metric. Family-centered quality metrics show promise in creating value in pediatric care within an equitable health system, but long-term evaluation of these metrics is necessary.

The authors have nothing to disclose.

Since 2012, when the Centers for Medicare & Medicaid Services (CMS) began linking financial penalties to hospitals with excessive readmissions for adult patients, researchers have questioned the extent to which pediatric readmissions can be used as a reliable quality measure. Compared with readmissions among adult patients, readmissions among pediatric patients are relatively uncommon. Furthermore, few (approximately 2%) qualify as potentially preventable, and pediatric readmission rates remain largely unchanged despite targeted attempts to prevent reutilization.^1,2 Nonetheless, state Medicaid agencies have continued to reduce reimbursement for hospitals based on available readmissions metrics, most commonly the Potentially Preventable Readmissions (PPR) algorithm.¹

In this issue of the Journal of Hospital Medicine, Auger et al³ performed a retrospective study to explore four existing metrics of pediatric hospital readmissions for their ability to identify preventable and unplanned readmissions. Investigators examined 30-day readmissions (n = 1,125) from 2014-2016 across multiple subspecialties, and classified readmissions by their preventability and unplanned status with use of a validated chart abstraction tool. Using the results of chart abstraction as the gold standard, investigators calculated the sensitivity and specificity, as well as estimated the positive and negative predictive values, of each readmissions metric. Auger and colleagues found that none of the four readmissions metrics could reliably assess preventability, and that only one metric reliably predicted unplanned hospital readmissions. Specifically, the commonly used PPR algorithm was estimated to have a positive predictive value of 13.0%-35.5% across a prevalence range of 10%-30%. This means that in a hospital where 10% of readmissions are truly preventable, the PPR will be wrong approximately 87% of the time. Tying payments to this metric is difficult to justify.

The authors highlighted the policy implications of the PPR falling short in its ability to identify preventable and unplanned pediatric readmissions. A good quality measure should be consistently reliable, and neither the PPR nor other measures studied meets this benchmark. Yet the findings lead to a broader conclusion: if most pediatric readmissions are not preventable, if there is no reliable way of measuring preventability, and if we have not demonstrated the ability to change patient trajectories away from reutilization, then perhaps the sun has set on using readmissions as a comprehensive quality measure for hospital-based care.

So how, then, should the hospital-to-home transition be evaluated? The paradigm of pediatric value of care is shifting to incorporate family-centered perspectives into consideration of quality measures.² There has to be a balance between healthcare costs and outcomes that affect families; measures should take into account issues such as patient and caregiver anxiety and time away from work.² Moreover, because social determinants of health and medical complexity strongly influence readmission rates,^4,5 focus should be placed on redirecting resources toward patients and families with significant medical, social, and financial needs as they transition home from the hospital. While measures of healthcare equity are currently lacking, the overall quality and equity of pediatric care transitions could be enhanced by looking beyond the narrow lens of readmission rates to incorporate actual needs assessments of families.

In summary, Auger and colleagues identified deficits in existing readmission metrics—but creating a solution that is meaningful to all stakeholders will be more complex than simply identifying a better metric. Family-centered quality metrics show promise in creating value in pediatric care within an equitable health system, but long-term evaluation of these metrics is necessary.

The authors have nothing to disclose.

In this edition of the Journal of Hospital Medicine, Fenster and colleagues evaluate the trend of postdischarge intravenous (IV) antibiotic therapy for children with osteomyelitis, complicated pneumonia, and complicated appendicitis.¹ Children requiring prolonged antibiotic therapy were historically discharged home with a peripherally inserted central catheter (PICC) for IV antibiotics. Recent studies suggest that treatment failure occurs uncommonly, and that oral antibiotics are as effective as those administered intravenously.^2-4 Oral antibiotics also avoid the additional risk of PICC-related complications, such as line malfunction, infections, and thrombi, which all lead to increased re-visits to hospital.

New research seldom leads to rapid change in clinical practice.⁵ This is particularly the case when new evidence favors the abandonment of accepted medical practices or supports the deimplementation of low-value care. The mounting body of evidence suggests that postdischarge IV antibiotic therapy is low-value care for children with osteomyelitis, complicated pneumonia, and complicated appendicitis, and that overuse is associated with unnecessary harm. Fenster and colleagues sought to evaluate the extent to which the management of these conditions has changed over time in the United States. They conducted a retrospective cohort study of children discharged from hospitals contributing data to the Pediatric Health Information System (PHIS) database. Validated algorithms using discharge diagnosis and procedure codes were used to identify children with the three conditions who were discharged home with IV antibiotic therapy.

Between January 2000 and December 2018 and across 52 hospitals, there were 24,753 hospitalizations for osteomyelitis, 13,700 for complicated pneumonia, and 60,575 for complicated appendicitis. Rates of postdischarge IV antibiotic therapy decreased over time for all conditions, from 61% to 22% for osteomyelitis, from 29% to 19% for complicated pneumonia, and from 13% to 2% for complicated appendicitis. Rather than a gradual reduction over time, the authors used piecewise linear regression to identify an inflection point when the decrease started: the inflection points for all three occurred around 2009 or 2010. Despite the observed decrease over time, there was significant variation in practice patterns among hospitals in 2018. For example, while the median rate of postdischarge IV antibiotic therapy for osteomyelitis was 18%, the interquartile ranged from 9% to 40%.

The authors conducted several sensitivity analyses, with the exclusion of hospitals that provided data only for certain years, which supported the robustness of the findings. Yet there are important limitations, most notably the lack of data on outcomes related to overuse and efficiency: type of antibiotics used (narrow vs broad spectrum) and total duration of antibiotics or variation in length of stay. The validated algorithms were also based on older ICD-9 codes and may perform less well with ICD-10 or from 2015 onwards. Lastly, the findings are limited to children’s hospitals and may not apply to general hospitals that care for many children.

The authors suggest that the deimplementation of postdischarge IV antibiotic therapy for the three conditions occurred spontaneously. Yet it is worth considering the different levels of agents of change that may have influenced these observations, such as research evidence, national condition guidelines, national efforts at reducing overuse and improving safety, local hospital efforts, and shared decision-making.

Postdischarge antibiotic therapy options for osteomyelitis, complicated pneumonia, and complicated appendicitis are supported by weak research evidence. Oral and parenteral therapy are equally effective but based on observational data; a randomized controlled trial is unlikely to ever be conducted because of uncommon outcomes, such as treatment failures. For these scenarios, greater emphasis should be placed on factors other than effectiveness, such as harms, availability of alternative options, and cost.⁶ For postdischarge IV antibiotic therapy, one potential explanation for the observed deimplementation is the greater awareness of harm, with up to 20% of cases with IV antibiotics requiring PICC removal.⁷ There is also a readily available alternative (oral antibiotics) with a favorable cost and effectiveness profile.

National condition guidelines advocating early transition to oral antibiotic therapy began to appear before and during the observed inflection point of 2009 and 2010. The 2002 British Thoracic Society guidelines for community-acquired pneumonia suggested considering oral agents after clear evidence of improvement,⁸ and the 2010 Infectious Diseases Society of America guidelines recommended oral antibiotic options for children discharged home with intra-abdominal infections.⁹ A systematic review published in 2002 also questioned the need for prolonged IV antibiotic therapy compared with early transition to oral agents in osteomyelitis.¹⁰ While no targeted national interventions to drive practice change existed, widespread national efforts at reducing overuse (eg, Choosing Wisely^®) and improving safety (eg, reducing central line complications) have increased in the past decade.¹¹

An important agent of change that Fenster and colleagues were not able to tease out was the impact of local hospital level efforts. In parallel to national efforts, there has likely been targeted hospital-level interventions that are disease specific (eg, order sets, pathways/guidelines, shared–decision-making tools) or focused on reducing adverse events (eg, reducing inappropriate PICC use). For example, between 2010 and 2012, one US children’s hospital increased the number of children with osteomyelitis discharged on oral antibiotics from a median of 0% to 100% with a bundle of quality improvement interventions, including standardized treatment protocols and shared decision-making.¹²

Despite the encouraging results, up to 22% of children were discharged from hospitals with postdischarge IV antibiotic therapy, and significant variation persists in 2018. Evidence of harm or even strong recommendations to change practice are themselves inadequate for behavior change.¹³ While it is clear that some element of deimplementation may have occurred organically over the past two decades, it is time for concerted deimplementation strategies that focus on practitioners or hospitals with “entrenched practices.”⁶

Dr Gill has received grant funding from the Canadian Paediatric Society, the Hospital for Sick Children, and the Canadian Institutes of Health Research (CIHR) in the past 5 years. He is on editorial board of BMJ Evidence-Based Medicine (EBM) and on the Institute Advisory Board for the CIHR Institute of Human Development and Child and Youth Health (IHDCYH), for which he has expenses reimbursed to attend meetings. He is a member of the EBMLive steering committee, and he has expenses reimbursed to attend the conference. Dr Mahant has received grant funding from CIHR in the past 5 years and is a Senior Deputy Editor of Journal of Hospital Medicine. The authors reported no conflicts of interest or financial relationships relevant to this manuscript.

In this edition of the Journal of Hospital Medicine, Fenster and colleagues evaluate the trend of postdischarge intravenous (IV) antibiotic therapy for children with osteomyelitis, complicated pneumonia, and complicated appendicitis.¹ Children requiring prolonged antibiotic therapy were historically discharged home with a peripherally inserted central catheter (PICC) for IV antibiotics. Recent studies suggest that treatment failure occurs uncommonly, and that oral antibiotics are as effective as those administered intravenously.^2-4 Oral antibiotics also avoid the additional risk of PICC-related complications, such as line malfunction, infections, and thrombi, which all lead to increased re-visits to hospital.

New research seldom leads to rapid change in clinical practice.⁵ This is particularly the case when new evidence favors the abandonment of accepted medical practices or supports the deimplementation of low-value care. The mounting body of evidence suggests that postdischarge IV antibiotic therapy is low-value care for children with osteomyelitis, complicated pneumonia, and complicated appendicitis, and that overuse is associated with unnecessary harm. Fenster and colleagues sought to evaluate the extent to which the management of these conditions has changed over time in the United States. They conducted a retrospective cohort study of children discharged from hospitals contributing data to the Pediatric Health Information System (PHIS) database. Validated algorithms using discharge diagnosis and procedure codes were used to identify children with the three conditions who were discharged home with IV antibiotic therapy.

Between January 2000 and December 2018 and across 52 hospitals, there were 24,753 hospitalizations for osteomyelitis, 13,700 for complicated pneumonia, and 60,575 for complicated appendicitis. Rates of postdischarge IV antibiotic therapy decreased over time for all conditions, from 61% to 22% for osteomyelitis, from 29% to 19% for complicated pneumonia, and from 13% to 2% for complicated appendicitis. Rather than a gradual reduction over time, the authors used piecewise linear regression to identify an inflection point when the decrease started: the inflection points for all three occurred around 2009 or 2010. Despite the observed decrease over time, there was significant variation in practice patterns among hospitals in 2018. For example, while the median rate of postdischarge IV antibiotic therapy for osteomyelitis was 18%, the interquartile ranged from 9% to 40%.

The authors conducted several sensitivity analyses, with the exclusion of hospitals that provided data only for certain years, which supported the robustness of the findings. Yet there are important limitations, most notably the lack of data on outcomes related to overuse and efficiency: type of antibiotics used (narrow vs broad spectrum) and total duration of antibiotics or variation in length of stay. The validated algorithms were also based on older ICD-9 codes and may perform less well with ICD-10 or from 2015 onwards. Lastly, the findings are limited to children’s hospitals and may not apply to general hospitals that care for many children.

The authors suggest that the deimplementation of postdischarge IV antibiotic therapy for the three conditions occurred spontaneously. Yet it is worth considering the different levels of agents of change that may have influenced these observations, such as research evidence, national condition guidelines, national efforts at reducing overuse and improving safety, local hospital efforts, and shared decision-making.

Postdischarge antibiotic therapy options for osteomyelitis, complicated pneumonia, and complicated appendicitis are supported by weak research evidence. Oral and parenteral therapy are equally effective but based on observational data; a randomized controlled trial is unlikely to ever be conducted because of uncommon outcomes, such as treatment failures. For these scenarios, greater emphasis should be placed on factors other than effectiveness, such as harms, availability of alternative options, and cost.⁶ For postdischarge IV antibiotic therapy, one potential explanation for the observed deimplementation is the greater awareness of harm, with up to 20% of cases with IV antibiotics requiring PICC removal.⁷ There is also a readily available alternative (oral antibiotics) with a favorable cost and effectiveness profile.

National condition guidelines advocating early transition to oral antibiotic therapy began to appear before and during the observed inflection point of 2009 and 2010. The 2002 British Thoracic Society guidelines for community-acquired pneumonia suggested considering oral agents after clear evidence of improvement,⁸ and the 2010 Infectious Diseases Society of America guidelines recommended oral antibiotic options for children discharged home with intra-abdominal infections.⁹ A systematic review published in 2002 also questioned the need for prolonged IV antibiotic therapy compared with early transition to oral agents in osteomyelitis.¹⁰ While no targeted national interventions to drive practice change existed, widespread national efforts at reducing overuse (eg, Choosing Wisely^®) and improving safety (eg, reducing central line complications) have increased in the past decade.¹¹

An important agent of change that Fenster and colleagues were not able to tease out was the impact of local hospital level efforts. In parallel to national efforts, there has likely been targeted hospital-level interventions that are disease specific (eg, order sets, pathways/guidelines, shared–decision-making tools) or focused on reducing adverse events (eg, reducing inappropriate PICC use). For example, between 2010 and 2012, one US children’s hospital increased the number of children with osteomyelitis discharged on oral antibiotics from a median of 0% to 100% with a bundle of quality improvement interventions, including standardized treatment protocols and shared decision-making.¹²

Despite the encouraging results, up to 22% of children were discharged from hospitals with postdischarge IV antibiotic therapy, and significant variation persists in 2018. Evidence of harm or even strong recommendations to change practice are themselves inadequate for behavior change.¹³ While it is clear that some element of deimplementation may have occurred organically over the past two decades, it is time for concerted deimplementation strategies that focus on practitioners or hospitals with “entrenched practices.”⁶

Dr Gill has received grant funding from the Canadian Paediatric Society, the Hospital for Sick Children, and the Canadian Institutes of Health Research (CIHR) in the past 5 years. He is on editorial board of BMJ Evidence-Based Medicine (EBM) and on the Institute Advisory Board for the CIHR Institute of Human Development and Child and Youth Health (IHDCYH), for which he has expenses reimbursed to attend meetings. He is a member of the EBMLive steering committee, and he has expenses reimbursed to attend the conference. Dr Mahant has received grant funding from CIHR in the past 5 years and is a Senior Deputy Editor of Journal of Hospital Medicine. The authors reported no conflicts of interest or financial relationships relevant to this manuscript.

In this edition of the Journal of Hospital Medicine, Fenster and colleagues evaluate the trend of postdischarge intravenous (IV) antibiotic therapy for children with osteomyelitis, complicated pneumonia, and complicated appendicitis.¹ Children requiring prolonged antibiotic therapy were historically discharged home with a peripherally inserted central catheter (PICC) for IV antibiotics. Recent studies suggest that treatment failure occurs uncommonly, and that oral antibiotics are as effective as those administered intravenously.^2-4 Oral antibiotics also avoid the additional risk of PICC-related complications, such as line malfunction, infections, and thrombi, which all lead to increased re-visits to hospital.

New research seldom leads to rapid change in clinical practice.⁵ This is particularly the case when new evidence favors the abandonment of accepted medical practices or supports the deimplementation of low-value care. The mounting body of evidence suggests that postdischarge IV antibiotic therapy is low-value care for children with osteomyelitis, complicated pneumonia, and complicated appendicitis, and that overuse is associated with unnecessary harm. Fenster and colleagues sought to evaluate the extent to which the management of these conditions has changed over time in the United States. They conducted a retrospective cohort study of children discharged from hospitals contributing data to the Pediatric Health Information System (PHIS) database. Validated algorithms using discharge diagnosis and procedure codes were used to identify children with the three conditions who were discharged home with IV antibiotic therapy.

Between January 2000 and December 2018 and across 52 hospitals, there were 24,753 hospitalizations for osteomyelitis, 13,700 for complicated pneumonia, and 60,575 for complicated appendicitis. Rates of postdischarge IV antibiotic therapy decreased over time for all conditions, from 61% to 22% for osteomyelitis, from 29% to 19% for complicated pneumonia, and from 13% to 2% for complicated appendicitis. Rather than a gradual reduction over time, the authors used piecewise linear regression to identify an inflection point when the decrease started: the inflection points for all three occurred around 2009 or 2010. Despite the observed decrease over time, there was significant variation in practice patterns among hospitals in 2018. For example, while the median rate of postdischarge IV antibiotic therapy for osteomyelitis was 18%, the interquartile ranged from 9% to 40%.

The authors conducted several sensitivity analyses, with the exclusion of hospitals that provided data only for certain years, which supported the robustness of the findings. Yet there are important limitations, most notably the lack of data on outcomes related to overuse and efficiency: type of antibiotics used (narrow vs broad spectrum) and total duration of antibiotics or variation in length of stay. The validated algorithms were also based on older ICD-9 codes and may perform less well with ICD-10 or from 2015 onwards. Lastly, the findings are limited to children’s hospitals and may not apply to general hospitals that care for many children.

The authors suggest that the deimplementation of postdischarge IV antibiotic therapy for the three conditions occurred spontaneously. Yet it is worth considering the different levels of agents of change that may have influenced these observations, such as research evidence, national condition guidelines, national efforts at reducing overuse and improving safety, local hospital efforts, and shared decision-making.

Postdischarge antibiotic therapy options for osteomyelitis, complicated pneumonia, and complicated appendicitis are supported by weak research evidence. Oral and parenteral therapy are equally effective but based on observational data; a randomized controlled trial is unlikely to ever be conducted because of uncommon outcomes, such as treatment failures. For these scenarios, greater emphasis should be placed on factors other than effectiveness, such as harms, availability of alternative options, and cost.⁶ For postdischarge IV antibiotic therapy, one potential explanation for the observed deimplementation is the greater awareness of harm, with up to 20% of cases with IV antibiotics requiring PICC removal.⁷ There is also a readily available alternative (oral antibiotics) with a favorable cost and effectiveness profile.

National condition guidelines advocating early transition to oral antibiotic therapy began to appear before and during the observed inflection point of 2009 and 2010. The 2002 British Thoracic Society guidelines for community-acquired pneumonia suggested considering oral agents after clear evidence of improvement,⁸ and the 2010 Infectious Diseases Society of America guidelines recommended oral antibiotic options for children discharged home with intra-abdominal infections.⁹ A systematic review published in 2002 also questioned the need for prolonged IV antibiotic therapy compared with early transition to oral agents in osteomyelitis.¹⁰ While no targeted national interventions to drive practice change existed, widespread national efforts at reducing overuse (eg, Choosing Wisely^®) and improving safety (eg, reducing central line complications) have increased in the past decade.¹¹

An important agent of change that Fenster and colleagues were not able to tease out was the impact of local hospital level efforts. In parallel to national efforts, there has likely been targeted hospital-level interventions that are disease specific (eg, order sets, pathways/guidelines, shared–decision-making tools) or focused on reducing adverse events (eg, reducing inappropriate PICC use). For example, between 2010 and 2012, one US children’s hospital increased the number of children with osteomyelitis discharged on oral antibiotics from a median of 0% to 100% with a bundle of quality improvement interventions, including standardized treatment protocols and shared decision-making.¹²

Despite the encouraging results, up to 22% of children were discharged from hospitals with postdischarge IV antibiotic therapy, and significant variation persists in 2018. Evidence of harm or even strong recommendations to change practice are themselves inadequate for behavior change.¹³ While it is clear that some element of deimplementation may have occurred organically over the past two decades, it is time for concerted deimplementation strategies that focus on practitioners or hospitals with “entrenched practices.”⁶

Dr Gill has received grant funding from the Canadian Paediatric Society, the Hospital for Sick Children, and the Canadian Institutes of Health Research (CIHR) in the past 5 years. He is on editorial board of BMJ Evidence-Based Medicine (EBM) and on the Institute Advisory Board for the CIHR Institute of Human Development and Child and Youth Health (IHDCYH), for which he has expenses reimbursed to attend meetings. He is a member of the EBMLive steering committee, and he has expenses reimbursed to attend the conference. Dr Mahant has received grant funding from CIHR in the past 5 years and is a Senior Deputy Editor of Journal of Hospital Medicine. The authors reported no conflicts of interest or financial relationships relevant to this manuscript.

“Time is the coin of your life. It is the only coin you have, and only you can determine how it will be spent. Be careful lest you let other people spend it for you.”

—Carl Sandburg

No one went into the practice of medicine to check off boxes. Clinicians find joy and purpose by connecting with patients and interacting with colleagues. Unfortunately, our goal of practicing in an environment that allows these experiences is threatened by extreme levels of regulatory and administrative oversight.^1,2 Decreased enjoyment and meaning in work may follow and lead to burnout, poor performance, and for some, premature departure from medicine.³ The negative effects on individuals can erode the morale and productivity of the group.

Many administrative requirements add value to clinical care. For example, interdisciplinary rounds may include a mandatory review of urinary catheters that reduces catheter-associated infections. The usefulness of some requirements, however, may promote implementation of other requirements of lesser value that interfere with the positive impact of meaningful interventions. Best Practice Alerts (BPAs) that are “clicked through” sap enthusiasm. Perfunctory monthly meetings that are informational rather than productive and exhaustive e-learning modules on institutional requirements such as “Corporate Compliance” take time away from patient care. Despite being a prominent driver of burnout, the most common approach to nuisances is nihilism. It is unrealistic for anyone with a full clinical slate to tackle pervasive irritations. Similarly, leaders may not see decreasing administrative burdens as a priority; the excitement for decreasing hassles pales relative to the excitement for developing a new vision or strategic plan.

Rather than acceptance, leaders should take proactive steps to decrease wasteful tasks. Begin by explicitly assessing the burden of tasks through dialogue with administrators, such as the chief medical officer. Administrators may not realize the impact of seemingly small requests on hospitalist workflow. For example, even adding one required question for every patient at interdisciplinary rounds can meaningfully affect the flow of rounds. Hospitalist leaders are well situated to assess the yield to burden ratio (Y/B) of any requirement. High burden tasks should be justified by substantial benefit, and tasks in which the Y/B is uncertain should be limited in scope until the value proposition is established.

The electronic medical record (EMR) deserves specific attention because it is an established source of annoyance and burnout.³ Leaders should proactively collaborate with administrators to remove EMR requirements with low Y/B. The “Get Rid of Stupid Stuff” (GROSS) program demonstrated the benefits of an innovative approach to eliminating wasteful EMR tasks.⁴ Our own institution discontinued the BPA asking clinicians to add “Chronic Kidney Disease, Stage III” to the Problem List when an assessment revealed that the Problem List was rarely updated and this BPA was frequently presented; the BPA was low yield, high burden.

Lastly, leaders should not become part of the problem. For example, a hospitalist-led quality improvement project may require documentation that a primary care physician has been contacted for each newly admitted patient. Assuming four patients and 5 minutes per call, this ask requires 20 minutes; the burden has been estimated but the yield is unknown, producing an unclear Y/B. Therefore, items generated within the group need to be vetted with the same scrutiny as external tasks.

Explicitly addressing wasteful burdens provides leaders with the opportunity to shift the emphasis from processes that distract from to initiatives that enhance patient care. Promoting a sense of meaning and purpose is an essential component of group success.⁵ Outstanding performance, productivity, and retention can only be realized through a work environment that prioritizes patients and minimizes tasks not aligned with this mission.

The authors have nothing to disclose.

“Time is the coin of your life. It is the only coin you have, and only you can determine how it will be spent. Be careful lest you let other people spend it for you.”

—Carl Sandburg

No one went into the practice of medicine to check off boxes. Clinicians find joy and purpose by connecting with patients and interacting with colleagues. Unfortunately, our goal of practicing in an environment that allows these experiences is threatened by extreme levels of regulatory and administrative oversight.^1,2 Decreased enjoyment and meaning in work may follow and lead to burnout, poor performance, and for some, premature departure from medicine.³ The negative effects on individuals can erode the morale and productivity of the group.

Many administrative requirements add value to clinical care. For example, interdisciplinary rounds may include a mandatory review of urinary catheters that reduces catheter-associated infections. The usefulness of some requirements, however, may promote implementation of other requirements of lesser value that interfere with the positive impact of meaningful interventions. Best Practice Alerts (BPAs) that are “clicked through” sap enthusiasm. Perfunctory monthly meetings that are informational rather than productive and exhaustive e-learning modules on institutional requirements such as “Corporate Compliance” take time away from patient care. Despite being a prominent driver of burnout, the most common approach to nuisances is nihilism. It is unrealistic for anyone with a full clinical slate to tackle pervasive irritations. Similarly, leaders may not see decreasing administrative burdens as a priority; the excitement for decreasing hassles pales relative to the excitement for developing a new vision or strategic plan.

Rather than acceptance, leaders should take proactive steps to decrease wasteful tasks. Begin by explicitly assessing the burden of tasks through dialogue with administrators, such as the chief medical officer. Administrators may not realize the impact of seemingly small requests on hospitalist workflow. For example, even adding one required question for every patient at interdisciplinary rounds can meaningfully affect the flow of rounds. Hospitalist leaders are well situated to assess the yield to burden ratio (Y/B) of any requirement. High burden tasks should be justified by substantial benefit, and tasks in which the Y/B is uncertain should be limited in scope until the value proposition is established.

The electronic medical record (EMR) deserves specific attention because it is an established source of annoyance and burnout.³ Leaders should proactively collaborate with administrators to remove EMR requirements with low Y/B. The “Get Rid of Stupid Stuff” (GROSS) program demonstrated the benefits of an innovative approach to eliminating wasteful EMR tasks.⁴ Our own institution discontinued the BPA asking clinicians to add “Chronic Kidney Disease, Stage III” to the Problem List when an assessment revealed that the Problem List was rarely updated and this BPA was frequently presented; the BPA was low yield, high burden.

Lastly, leaders should not become part of the problem. For example, a hospitalist-led quality improvement project may require documentation that a primary care physician has been contacted for each newly admitted patient. Assuming four patients and 5 minutes per call, this ask requires 20 minutes; the burden has been estimated but the yield is unknown, producing an unclear Y/B. Therefore, items generated within the group need to be vetted with the same scrutiny as external tasks.

Explicitly addressing wasteful burdens provides leaders with the opportunity to shift the emphasis from processes that distract from to initiatives that enhance patient care. Promoting a sense of meaning and purpose is an essential component of group success.⁵ Outstanding performance, productivity, and retention can only be realized through a work environment that prioritizes patients and minimizes tasks not aligned with this mission.

The authors have nothing to disclose.

“Time is the coin of your life. It is the only coin you have, and only you can determine how it will be spent. Be careful lest you let other people spend it for you.”

—Carl Sandburg

No one went into the practice of medicine to check off boxes. Clinicians find joy and purpose by connecting with patients and interacting with colleagues. Unfortunately, our goal of practicing in an environment that allows these experiences is threatened by extreme levels of regulatory and administrative oversight.^1,2 Decreased enjoyment and meaning in work may follow and lead to burnout, poor performance, and for some, premature departure from medicine.³ The negative effects on individuals can erode the morale and productivity of the group.

Many administrative requirements add value to clinical care. For example, interdisciplinary rounds may include a mandatory review of urinary catheters that reduces catheter-associated infections. The usefulness of some requirements, however, may promote implementation of other requirements of lesser value that interfere with the positive impact of meaningful interventions. Best Practice Alerts (BPAs) that are “clicked through” sap enthusiasm. Perfunctory monthly meetings that are informational rather than productive and exhaustive e-learning modules on institutional requirements such as “Corporate Compliance” take time away from patient care. Despite being a prominent driver of burnout, the most common approach to nuisances is nihilism. It is unrealistic for anyone with a full clinical slate to tackle pervasive irritations. Similarly, leaders may not see decreasing administrative burdens as a priority; the excitement for decreasing hassles pales relative to the excitement for developing a new vision or strategic plan.

Rather than acceptance, leaders should take proactive steps to decrease wasteful tasks. Begin by explicitly assessing the burden of tasks through dialogue with administrators, such as the chief medical officer. Administrators may not realize the impact of seemingly small requests on hospitalist workflow. For example, even adding one required question for every patient at interdisciplinary rounds can meaningfully affect the flow of rounds. Hospitalist leaders are well situated to assess the yield to burden ratio (Y/B) of any requirement. High burden tasks should be justified by substantial benefit, and tasks in which the Y/B is uncertain should be limited in scope until the value proposition is established.

The electronic medical record (EMR) deserves specific attention because it is an established source of annoyance and burnout.³ Leaders should proactively collaborate with administrators to remove EMR requirements with low Y/B. The “Get Rid of Stupid Stuff” (GROSS) program demonstrated the benefits of an innovative approach to eliminating wasteful EMR tasks.⁴ Our own institution discontinued the BPA asking clinicians to add “Chronic Kidney Disease, Stage III” to the Problem List when an assessment revealed that the Problem List was rarely updated and this BPA was frequently presented; the BPA was low yield, high burden.

Lastly, leaders should not become part of the problem. For example, a hospitalist-led quality improvement project may require documentation that a primary care physician has been contacted for each newly admitted patient. Assuming four patients and 5 minutes per call, this ask requires 20 minutes; the burden has been estimated but the yield is unknown, producing an unclear Y/B. Therefore, items generated within the group need to be vetted with the same scrutiny as external tasks.

Explicitly addressing wasteful burdens provides leaders with the opportunity to shift the emphasis from processes that distract from to initiatives that enhance patient care. Promoting a sense of meaning and purpose is an essential component of group success.⁵ Outstanding performance, productivity, and retention can only be realized through a work environment that prioritizes patients and minimizes tasks not aligned with this mission.

The authors have nothing to disclose.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

All authors declare they have no conflicts of interest.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

All authors declare they have no conflicts of interest.

The Accreditation Council for Graduate Medical Education (ACGME) first mandated residency work hour restrictions in 2003.¹ In 2011, revised work hour requirements were issued, further limiting the maximum duration of a shift and extending the duration of time off between scheduled shifts.² Academic medical centers have been forced to adapt to work hour restrictions, and cuts in funding to research and educational missions have pressured institutions to restructure with a greater focus on high-quality, lower-cost care.^3,4 In response, many academic hospitals have added hospitalist teams, or incorporated advanced practice clinicians (APCs) (nurse practitioners [NPs] and physician assistants [PAs]) to accommodate resident physician duty hour restrictions on their inpatient general medicine services.^5,6 More recently, the COVID-19 pandemic has created unanticipated physician shortages forcing medical centers to rapidly expand and broaden the scope of their existing APC workforce.⁷

Several comparisons of clinical outcomes, cost, and patient satisfaction between different combinations of hospitalist-based, resident-based, or APC-based inpatient teams have been reported with conflicting observations.^6,8-14 Roy et al reported no significant differences in mortality, length of stay (LOS), or readmissions between PA and resident teams.⁶ Timmermans et al reported similar cost-effectiveness, LOS, and quality of care between PA and physician teams that included a hybrid of attending only and resident teams.^13,14 Alternatively, Singh et al and Iannuzzi et al reported increased LOS among PA teams,^10,12 whereas Chin et al observed an increased LOS and reduced 30-day readmissions among hospitalist teams.⁸ While these observed differences may be attributable to heterogeneous patient populations or institution-specific team structure, the exact reasons remain unknown. Furthermore, understanding the value of alternate staffing models is essential for medical centers to prepare for potential COVID-19 related physician shortages. To our knowledge, no study to date has directly compared outcomes between resident, APC, and hospitalist team structures within an academic medical center.

We believe our institution provides a unique environment to study the differences in inpatient general medicine team structure with respect to quality and efficiency of care delivery. The objective of our study is to directly compare clinical outcomes and resource utilization among three distinct team structures: APC, resident, and solo hospitalist. We hypothesize that clinical outcomes, cost, and utilization of consult services will be similar across all team structures and hospitalist teams will discharge patients earlier than resident and APC teams.

Study Design and Setting

We conducted a retrospective observational cohort study at the University of Utah Medical Center, a 548-bed academic medical center in Salt Lake City. An electronic database query was used to identify all patients discharged from the inpatient general internal medicine service between July 1, 2015, and July 1, 2018. Baseline patient characteristics were collected including age, gender, and Charlson comorbidity index (CCI).¹⁵ Case-mix index was determined for admissions where a Medicare Severity Diagnosis Related Group (MS-DRG) and corresponding weight was assigned.^16,17 Source of admission was collected to identify patients transferred from an outside hospital, typically due to increased medical complexity or need for specialty care not available at the referring center. Time of admission was collected to classify whether a patient was admitted during the day or at night. Length of stay was calculated as the difference between discharge date/time and admission date/time. Discharge order time was collected as a measure of clinician efficiency. The number of consults per admission was determined by the number of different medical or surgical subspecialty services that wrote at least one consultation or progress note after the time of admission and were not the primary service at the time the note was written. The project was reviewed and deemed exempt by the University of Utah Institutional Review Board (IRB 00104884).

Inpatient Care Team Structure

Patients were assigned to one of three cohorts dependent on the assigned treatment team at the time of discharge. The three inpatient team structures were as follows: (1) a “resident team” composed of a senior resident (postgraduate year [PGY] 2 or PGY3) and one to two medical students or one senior resident, two interns (PGY1), and one to two medical students supervised by a hospitalist physician; (2) an “APC team” composed of one to two APCs supervised by a hospitalist physician; and (3) a “hospitalist team” composed of one attending hospitalist independently managing all patients.

Advanced Practice Clinicians

The APC service included 10 APCs (8 PAs and 2 NPs), with a combined workforce of nine APC full-time equivalents during the study period. Their experience ranged from new graduate to 11 years of clinical experience, with an average of 4.2 years. Among the 6 APCs with prior clinical experience, the majority (86%) of their years of clinical experience were within inpatient medicine, oncology, or cardiology. Recognizing the variability in clinical experience, we employed a rigorous onboarding program that entailed an average of 80 hours of didactic sessions including 1:1 teaching of the inpatient Society of Hospital Medicine core lecture series combined with initial intense clinical oversight.¹⁸ This program ranged from 2 weeks to 6 weeks depending on the individual APC’s clinical experience, progress, and comfort working independently. This onboarding program has subsequently been formalized into a 1-year APC fellowship that began after the study period concluded.

The degree of autonomy for each APC was individualized based on their clinical experience and ability to recognize limitations such as medical decision-making, clinical knowledge, and effective use of interprofessional team members (eg, peers, nursing, ancillary staff, consultants, and support personnel). Those APCs who demonstrated a sufficient level of clinical competence functioned with a high level of autonomy. During the day, APCs were expected to be the first point of contact for interprofessional team members, to respond to acute clinical changes in a patient’s condition, and to discuss active issues with the supervising attending, all with the majority of medical decision-making, direct patient communication, documentation, and care coordination performed by the APC. An experienced subset of the APC service was responsible for overnight coverage. Nocturnist APCs independently managed all cross-cover issues on patients assigned to APC and hospitalist teams and performed admissions with very little to no direct supervision of the overnight attending physician.

Patient Admission and Redistribution Process

During the study period, resident teams performed all daytime admissions (6 am to 6 pm) on a rotating basis. On any given day, three of four resident teams performed daytime admissions with the fourth team designated as “golden” and free from admitting duties. Patients admitted during the day remained assigned to the resident team for continuity. The APC and hospitalist teams did not accept new admissions during the day. Nighttime admissions (6 pm to 6 am) were performed by a separate team composed of two senior residents, two interns, one APC, occasional APC and medical students, and one supervising attending hospitalist. This team functioned as a single unit. Nighttime admissions were performed in a sequential and rotating fashion (eg, Intern A > Intern B > Resident A > Resident B > APC > student(s) > Intern A > Intern B, etc). Patients admitted overnight were randomly redistributed the following morning, with the majority reassigned to an APC team or hospitalist team in order to offset the workload of the resident teams performing daytime admissions. Following redistribution, a patient would remain assigned to the daytime APC or hospitalist team for the duration of their hospitalization. The redistribution decisions were based on individual team census, without systematic consideration of an individual patient’s diagnosis, medical complexity, socioeconomic status, or perceived quality of learning potential (eg, good teaching case).

Study Outcomes

We divided study outcomes into two categories, clinical outcomes and resource utilization. Clinical outcomes included LOS, unplanned readmission within 30-days, and inpatient mortality and were designed to measure patient-related outcomes as a reflection of the quality of care delivered by different team structures. Resource utilization included discharge order time, discharge time, consults per admission, and total direct cost, which were designed to measure provider-related differences in efficiency and cost of care.

Statistical analysis

Baseline characteristics and unadjusted outcomes are reported as frequency and percent, normally distributed variables as mean with SD, and nonnormally distributed variables as median with interquartile range (IQR). Baseline characteristics and unadjusted outcomes were compared using the chi-square test or the t test, where appropriate. Multivariable regression analysis using generalized linear models with a log link function and gamma distribution was used for continuous outcomes. Multivariable logistic regression was used for binary outcomes.¹⁰ Covariates included in regression models were age, gender, CCI, transfer from an outside hospital, and nighttime admission. In a sensitivity analysis, we included MS-DRG weight as a covariate for 85% of hospitalizations in our cohort exclusive of observation stays, and our findings were qualitatively similar (data not reported but available on request). Adjusted continuous outcomes were estimated using marginal effects at the means.¹⁹ Due to the sensitivity of cost data and an institutional policy against disclosing cost figures, total direct costs were normalized using the unadjusted median and adjusted mean total direct cost of an admission to an APC team as the normalizing value. A P value cutoff of .05 was used to determine statistical significance. Stata/IC version 16.1 (StataCorp) was used for all analyses.

Study Population

A total of 12,716 hospital admissions were identified during the study period. Of these, 7,943 (62.5%) admissions were assigned to a resident team, 3,519 (27.7%) admissions were assigned to an APC team, and the remaining 1,254 (9.9%) were assigned to a hospitalist team. Baseline patient characteristics are reported in Table 1. Patients admitted to resident teams (mean age [SD], 56.9 [19.1] years) were younger than those admitted to an APC team (58.0 [19.3] years; P = .004) or a hospitalist team (58.2 [19.3] years; P = .026). The case-mix index (mean MS-DRG weight [SD], 1.44 [0.87]) was slightly lower for resident teams than that for APC teams (1.49 [0.90]; P = .025).Resident teams had a significantly lower proportion of night admissions than did APC teams (32.0% vs 49.5%; P < .001) and hospitalist teams (48.6%; P < .001). APC teams were assigned more patients transferred from an outside hospital (19.1%), compared with resident teams (15.0%; P < .001) and hospitalist teams (16.0%; P = .015). No other significant differences were observed in baseline characteristics between cohorts.

Clinical Outcomes

Unadjusted analysis demonstrated the LOS was similar among resident, APC, and hospitalist teams with a median (IQR) LOS of 2.90 (1.86, 4.26) days, 2.93 (1.89, 4.66) days, and 2.86 (1.84, 4.67) days, respectively. No significant differences were observed in unadjusted 30-day readmissions or inpatient mortality among the team structures (Table 2). Following multivariable adjustment for differences in baseline characteristics, no significant differences were observed in LOS, 30-day readmission, or inpatient mortality among teams (Table 3).

Resource Utilization

In unadjusted comparisons, hospitalist teams were observed to place discharge orders more than 30 minutes earlier than APC teams (median hours after midnight [IQR], 11.20 [9.63, 13.60] vs 11.73 [10.00, 13.87]; P < .001) and 54 minutes earlier than resident teams (12.10 [10.38, 13.90]; P < .001) (Table 2). Consistent with the earlier placement of discharge orders, hospitalist patients were also discharged from the hospital 26 and 32 minutes earlier than APC and resident patients, respectively. APC teams also discharged patients slightly earlier (6 minutes) than resident teams (median hours after midnight [IQR], 14.97 [13.23, 16.72] vs 15.07 [13.42, 16.73]; P = .045). Median consultation use among teams was similar, although statistically significant differences were present. Normalized total direct cost was 8% higher (P < .001) for admissions to APC teams than that for resident teams and 7% higher (P = .008) than that for hospitalist teams in unadjusted analysis (Table 2).

Following multivariable adjustment, the mean differences in discharge order time and discharge time remained significant with hospitalist teams discharging patients an average of 20 to 30 minutes earlier than APC and resident teams (Table 3). Consultant utilization remained significantly different between teams, with APC teams utilizing consultants on average 15% more than hospitalist teams (P < .001) and 7% more than resident teams (P = .001). The differences in total direct costs were not significant after adjusted analysis.

Many academic medical centers have expanded their workforce with APC or nonteaching hospitalist teams to accommodate the increasing volume of hospital admissions, resident work hour restrictions,^1,2 and medical complexity of an aging population. Several hospitals have reported comparative outcomes between different care delivery models, with conflicting results.^6,8,10-12 In our study, we directly evaluated three inpatient care delivery models and found that hospitalist teams discharged patients more efficiently and utilized fewer consultants, compared with APC and resident teams. In spite of this improved efficiency, no significant differences were observed in cost or other clinical outcomes.

Our findings are important and further strengthen the evidence supporting the use of APCs on inpatient general medicine services and are of particular interest to academic centers struggling to expand staffing in order to offset the growth in patient volume and reduction in resident workforce. We believe several findings from our study warrant further discussion.

First, although hospitalist teams were able to discharge patients more efficiently, this observation may be influenced by factors of workflow rather than caused by significant disparities in efficiency between provider types (ie, APC vs hospitalist vs resident physician). As with most academic centers, patients assigned to resident teams are presented by house staff to an attending physician who is ultimately responsible for patient care decisions. Therefore, it is conceivable that delays in the discharge process are in part related to the convention of bedside rounding and discussing the care plan prior to discharge.²⁰ In fact, we recognized this as a bottleneck and changed our discharge process for resident teams in June 2017, with a measurable improvement in discharge times. In the absence of this intervention, our observed differences in discharge times among teams may have been even greater.

Second, no significant differences in clinical outcomes were observed in our adjusted analyses, which suggests that a similar quality of care is delivered to patients regardless of team structure, an important observation when considering different staffing models.

Third, we observed a significant increase in consultation use among resident and APC teams, compared with hospitalists. While we are not able to precisely identify the basis for this variation, we believe it could reflect differences in clinical experience, comfort with diagnostic uncertainty, or the unequal distribution of patients transferred from outside hospitals for tertiary care. Interestingly, the greater consultation use did not correlate with higher healthcare costs, a finding recently reported by Stevens et al.²¹

Fourth, we believe the lack of differences in cost and clinical outcomes among team structures may be of particular interest to academic centers when considering physician burnout, salaries, and clinical education. The relationship between clerical burden, such as completing clinical documentation and computerized physician order entry, has been implicated as a risk factor for physician burnout.²² Incorporating APCs into roles similar to those performed by resident physicians may reduce the clerical burden on hospitalists, thereby reducing the risk of physician burnout. The addition of APCs may also represent opportunities for cost savings for healthcare centers when comparing the median salary of an APC to that of an internal medicine hospitalist.^23,24 Moreover, academic hospitalists have been shown to be excellent medical educators and report increased job satisfaction with a variety of duties beyond direct patient care.^24,25 Unforeseen benefits of adding APC teams within our institution has been the added teaching opportunities for APCs and APC students, increased collegiality with the APCs, and the creation of an APC fellowship program with a focus on inpatient medicine. Similar postgraduate training programs have been reported and serve as effective models to train APCs for hospital-based practice.²⁶

Lastly, although this project was conceived and completed prior to the COVID-19 pandemic, our observations may be informative for medical centers experiencing a workforce shortage caused by a surge of COVID-19 patients. During a physician shortage we believe our APC team model could be rapidly expanded to accommodate a large influx of patients. This expansion could be accomplished through a single attending physician overseeing multiple APC teams. In this model, the supervising physician would only evaluate the most complex patients with most patients being managed solely by an APC from admission to discharge. Such changes may require temporary suspension of state laws restricting APC independent practice.^27,28

Our findings contrast those of previous reports in that we did not observe significant differences in clinical outcomes (ie, LOS, inpatient mortality, and 30-day readmissions) or total direct cost.^8,10,21 Other institutions have noted an increased LOS among APC teams and hospitalist teams, compared with resident teams.^8,10 Furthermore, Chin et al and Iannuzzi et al reported reductions in healthcare cost for resident teams, whereas our study did not identify significant cost differences among team structures. Although we cannot pinpoint the exact reason(s) for these dissimilarities, it is plausible that unmeasured factors such as institutional differences in APC training, direct physician supervision, admission processes, or inpatient team census may play a role.

Several study limitations should be recognized. First, the retrospective, nonrandomized design is one of the largest limitations of our study. Administrative data was obtained via an electronic query of our data warehouse, and although we aimed to identify as many patient characteristics as possible to adjust for cofounding effects, undetected differences among cohorts may exist. Second, our inpatient admission process may have placed undue burden on resident teams to perform all daytime admissions, inadvertently affecting study outcomes. It is possible the observed benefits of a solo hospitalist team are attributable to the lack of admitting duties rather than inherent advantages of the team structure. If this were the case, we would expect similar benefits among APC teams, which we did not note. Third, the study was performed at a single academic center, which may limit the generalizability of our results. Fourth, it is possible the outcomes are similar among teams because our hospitalist faculty rotate proportionately between the different teams. Lastly, the study was underpowered to detect a significant difference in mortality between hospitalist and APC teams. A post hoc power calculation based on our observed sample and effect sizes estimated 75% power to detect a mortality difference between hospitalists and APCs; other mortality comparisons were adequately powered.

We observed similar total direct costs, LOS, 30-day readmission, and inpatient mortality between hospitalist, APC, and resident teams. APC and resident teams utilized more consultants and discharged patient later than hospitalists. Our analysis suggests clinical outcomes are not significantly affected by inpatient team structure, and the addition of general medicine inpatient APC or hospitalist teams represent safe and efficient alternatives to traditional resident teams within an academic medical center.

All authors declare they have no conflicts of interest.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Readmission rates are frequently used as a hospital quality metric, with use including payment incentive at the hospital level,¹ specific condition quality measurement,² balancing measures for quality improvement projects,^3-5 transition success,^6,7 and use in public hospital rankings.⁸ Currently, four methods are commonly used to evaluate pediatric readmissions, each with strengths and limitations, including the following (Appendix Table 1):

1. All-cause readmissions: A measure of any readmission within a given time period regardless of the reason for readmission.⁹

2. Unplanned readmission/time flag: A measure intended to identify unplanned readmissions. This measure relies on time designations within the electronic health record. The time between hospital registration and admission is calculated, and if the readmission is registered more than 24 hours prior to admission, the readmission is considered planned.¹⁰ Hereafter, this measure will be referred to as the time flag measure.

3. Pediatric all-condition readmission (PACR): A measure intended to identify unplanned readmission through the exclusion of certain procedures and diagnoses.¹¹

4. Potentially preventable readmission (PPR): A method to identify preventable readmissions based on a proprietary algorithm developed by 3M Health Information Systems.^12,13

While all four of these measures are used to assess quality, there is little known about these measures’ ability to exclude planned readmissions and identify only preventable pediatric readmission, which conceptually is most relevant to the quality of care. However, many of these measures were not intended to capture preventability, but instead capture the related issue of whether the readmission was planned. Therefore, we sought to evaluate the four readmission measures as they relate to both preventability and unplanned status as determined through medical record review with multidisciplinary care provider input.

As part of a hospital-wide readmission reduction quality improvement collaborative at a free-standing tertiary care children’s hospital, clinicians from hospital medicine, cardiology, neonatology, and neurology teams reviewed 30-day readmissions using a standardized abstraction tool. All readmission events (observation or inpatient encounter) after any discharge (observation or inpatient encounter) from eligible units were reviewed; therefore, each hospitalization was a potential index hospitalization. We classified the preventability of each readmission with use of a previously described Likert scale with high interrater reliability.¹⁴ For these analyses, readmissions were considered preventable if the reviewing team rated them as either “more likely preventable” or “preventable in most circumstances.” Each readmission was also evaluated as planned or unplanned. Methods for readmission review and classification are in the Appendix.

We included all readmissions between July 2014 and June 2016. We compared the medical record review classifications with the assessments from each of the four measures of pediatric readmission. We calculated sensitivity and specificity for both outcomes (planned/unplanned and preventable/not preventable) for all four measures. For standardization of discussion, we categorized description of measure performance as “very poor” as less than 50%, “poor” between 50%-75%, “fair” as 75%-85%, “good” as 85%-90%, “very good” as 90%-95% and excellent as greater than 95%. We also calculated positive and negative predictive value (PPV and NPV) over plausible ranges of prevalence using the sensitivity and specificity of each comparison (Appendix).

Of note, certain exclusions are outlined by the PACR and PPR algorithms. The PACR evaluates only readmission events that occur in children younger than 18 years. The PPR algorithm does not assign preventability if either the index or readmission event is classified as an observation stay or if it is part of a larger chain of readmissions.

Among 30-day readmissions considered, 1,643 were eligible for medical record review; 1,125 reviews were completed by the clinical teams (68.5%). The median time to readmission was 7 days (interquartile range [IQR], 4-18). Most children were non-Hispanic White (71%) or Black (20%). The median age at hospitalization was 2.3 years (IQR 0.4-12.1). Most children had Medicaid (56%) or private (41%) insurance. Most of the reviews were performed in cardiology (43%) and hospital medicine (37%) with patients in neurology (13%) and neonatology (7%) constituting the remaining reviews. Uncontrolled advancement of chronic disease was the most common readmission category on medical record review (25.1%), followed by unrelated readmission (20.7%), scheduled readmission (20.4%), and progression of acute disease (16.6%) (Appendix Table 2).

Assessment of Preventable and Unplanned Readmissions

On multidisciplinary medical record review, most readmissions were classified as not preventable (84.5%). Specifically, 64% were not preventable and unplanned; 20% were deemed not preventable and planned. Only 15% were classified as unplanned and preventable and 1% as planned and preventable (Appendix Figure: Population A/B).

Matching Chart Review to the Four Algorithms

All 1,125 readmissions were assessed by the all-cause and time flag readmission measures (Appendix Figure: Population A/B). After applying algorithm exclusions (details in Appendix), only 804 of the 1,125 (71.5%) reviewed readmissions matched for PACR readmission comparison (Appendix Figure: Population C); 487 of the 1,125 (43.3%) of the reviewed readmissions matched for PPR comparison (Appendix Figure: Population D).

All-Cause

Because all-cause determines only if a readmission occurs, the measure is by definition 100% sensitive and 0% specific in both assessment of preventability and unplanned readmission (Table: Section A).

Time Flag

The time flag measure identified 80% (866/1,112) of the readmissions as unplanned. This measure had very good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and good to excellent NPV. In terms of identifying unplanned readmissions, the time flag measure had excellent sensitivity and very good specificity, which corresponded to very good to excellent PPV and good to very good NPV (Table: Section B).

PACR

The PACR algorithm identified 75% (599/796) of readmissions as unplanned. The PACR has good sensitivity but very poor specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PACR had fair sensitivity but poor specificity, which corresponded to fair PPV and poor NPV (Table: Section C).

PPR

The PPR algorithm identified 53% (257/487) of admissions as potentially preventable. The PPR algorithm had poor sensitivity and specificity in identifying preventable readmissions, which corresponded to very poor PPV and fair to very good NPV. In terms of identifying unplanned readmissions, the PPR algorithm had poor sensitivity and fair specificity in identifying unplanned readmissions, which corresponded to fair to good PPV and very poor to poor NPV (Table: Section D).

Evaluation of Excluded Readmission Events

Because both the PACR and PPR had large numbers of algorithm exclusions, we describe the preventability and unplanned assessment of the excluded readmission events. Both algorithms excluded preventable events. Of the 321 readmissions excluded by the PACR algorithm, 13.4% were classified as preventable by chart review. Likewise, 14.9% of 638 readmissions excluded by PPR were classified as preventable by chart review.

The ability to accurately capture preventable pediatric readmission is a goal for hospital quality experts and health policymakers alike. Of the four commonly used readmission measures to assess readmission, only PPR is designed to focus on preventability. Unfortunately, none of these four measures is adequately sensitive or specific to identify preventable readmissions; all measures had very poor PPV for preventability. Of the four measures, the time flag measure had the best sensitivity, specificity, PPV, and NPV for identifying unplanned readmissions.

The overall percentage of unplanned readmissions identified by both the time flag and by PACR measures match the overall percentage of unplanned readmissions identified in chart review: The time flag measure identified 80% of admissions as unplanned versus 79% identified by chart review (Appendix Figure: Population A/B); PACR classified 75% as unplanned versus 81% identified by chart review for PACR-eligible readmissions (Appendix Figure: Population C). In contrast, the PPR algorithm classified many more readmissions as potentially preventable (53%) than were identified by chart review at only 16% (Appendix Figure: Population D). The PACR and PPR algorithms also exclude a significant number of readmissions that are unplanned and a smaller, but not trivial, number of readmissions that are preventable; these exclusions limit their accuracy.

The ability to apply these four measures in real time during a hospitalization varies by metric. Two of the measures, the all-cause and time flag, can be applied during a readmission event, which is appealing for quality improvement initiatives. These measures allow for notification of providers that a current hospitalization is a readmission event, which allows providers the opportunity to learn from these events as they occur (Appendix Table 1). While “unplanned” is not the same as “potentially preventable,” almost all potentially preventable readmissions are unplanned; therefore, accurately identifying unplanned readmissions is more beneficial than all-cause. Additionally, a low all-cause readmission rate can be indicative of poor access to scheduled procedures. Nevertheless, all-cause readmission is sometimes used to measure quality.^1,8 While the time flag measure may be more useful for quality improvement initiatives and hospital providers, it relies on hospital registration time, which is not widely available in administrative data sources and, therefore, has limited usefulness to policymakers.

Both PACR and PPR require administrative claims analysis, which is appealing from a policy standpoint. However, the reliance on claims data means the inclusion/exclusion of events can occur only retrospectively, which limits the usefulness of these measures in learning and intervening in real time. When the two measures are compared, PACR offers better sensitivity and PPR offers better specificity with regard to identifying unplanned readmission. The PPR software overcalls preventable readmissions, identifying more readmissions as preventable than there actually are. Nevertheless, Medicaid in several states uses PPR for payment incentive.^1,15-17 Given the poor performance of PPR in assessing both preventable and unplanned pediatric readmission, the use of this measure as a quality metric should be limited.

This study should be considered in the context of several limitations. Because the assessment of preventability was determined as part of a learning quality improvement collaborative and not as a planned research endeavor, not all readmission reviews were completed nor were other existent tools¹⁸ that allow for preventability assessment via more structured medical record review used. Second, we reviewed cases only from certain clinical services, which would limit generalizability of these findings to all pediatric admissions. However, given the low sensitivity and specificity of some of the metrics, we would not anticipate that the addition of other types of admissions would improve the sensitivity and specificity enough to ensure reliability. Third, while we relied on an established method to determine preventability, prior work has demonstrated that additional information gathered from families may change preventability.¹⁹ Finally, due to the exclusions required by the PPR and PACR algorithms, not all readmission events were reviewed. However, these exclusions reflect the actual specifications of use for both measures.

The PPR software has poor fidelity in identifying preventable and unplanned pediatric readmission; this finding has broad policy implications given how widely it is used by state Medicaid offices to assess financial penalties. Among the four pediatric readmission measures used, the time flag metric best identifies unplanned readmissions.

The authors have no conflicts of interest or financial relationships relevant to this article to disclose.

Dr Auger’s research is supported by a grant from the Agency for Healthcare Research and Quality (1K08HS204735-01A1). The project described was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health, under Award Number 5UL1TR001425-04. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Discharges against medical advice (DAMAs), in which a patient leaves the hospital prior to a physician-recommended endpoint, represent approximately 1% to 2% of inpatient discharges in the United States.¹ When compared with routine discharges, a DAMA is associated with adverse clinical consequences, including an increased risk of all-cause mortality.^2,3 Additionally, due to incomplete care, a DAMA may result in increased healthcare resource utilization (HcRU), including the use of inpatient, emergency department (ED), and outpatient services in the postdischarge period. Quantifying these relationships can provide important information regarding an individual’s healthcare-seeking behavior following a DAMA.

Prior literature has focused on the association between a DAMA and the risk of inpatient readmission. Relative to routine discharges, a DAMA is associated with a 1.5 to 2 times increased risk of a 30-day readmission.^3-9 However, these estimates are based on mixed-payer populations primarily composed (65%-80%) of individuals with public (Medicaid, Medicare) or no insurance. Further, they do not differentiate this association by payer type. It is unclear if prior results apply to commercially insured adults. These individuals represent a small but nonnegligible proportion (19%) of all DAMAs in the United States.¹⁰ Quantifying relationships among commercially insured adults can help advance our understanding of readmission patterns in the DAMA population.

There is limited evidence regarding the relationship between a DAMA and outpatient HcRU in the postdischarge period. Use of ED services after a DAMA has been explored only in specific disease populations such as asthma.⁴ Additionally, prior studies have reported a reduced frequency in the receipt of medication prescriptions and outpatient follow-up plans among individuals with a DAMA at the time of discharge.^11,12 Whether these practices translate to altered patterns of postdischarge prescription drug fills or use of outpatient services is not known.

To address these substantive gaps in the literature, the present study evaluates the association between a DAMA and all-cause HcRU in the postdischarge period among commercially insured adults. We examined HcRU across all points of service including inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters, and prescription drug fills. These results can serve as a benchmark for comparison to future studies on DAMAs among publicly insured or uninsured individuals. Furthermore, such knowledge can help providers, payers, and policy planners make evidence-based decisions regarding postdischarge healthcare delivery.

Data Source

This retrospective study used a 10% random sample of enrollees in the IQVIA PharMetrics^® Plus database (purchased by University of Maryland, Baltimore, under license from IQVIA). The database is composed of fully adjudicated claims and enrollment information from over 70 contributing US health plans and self-insured employer groups for over 140 million unique enrollees from 2006 onward. The enrollee population is generally representative of the commercially insured population that is younger than 65 years of age (with a subset of commercial Medicare and Medicaid) with respect to age and gender.

The database allows longitudinal follow-up for individuals using three files: medical claims, pharmacy claims, and insurance eligibility. The average length of enrollment is 39 months. The claims data represent payments to providers for services rendered to individuals covered by health plans. The medical claims file contains information on diagnostic and therapeutic services rendered in the inpatient and outpatient settings. The pharmacy claims file captures data on prescription drugs dispensed in retail and mail-order settings. The eligibility file contains demographic and insurance eligibility information for individuals.

Study Population

We identified all individuals aged 18 to 64 years with an inpatient admission record between January 1, 2007, and December 31, 2015. All individuals with continuous medical and prescription drug coverage from 6 months prior to the hospital admission date (baseline period) through 30 days following the discharge date (follow-up period) were included. Inpatient admissions with a missing discharge disposition or those that resulted in in-hospital death, discharge to a short-term hospital, skilled nursing facility, intermediate care facility, or any other type of facility were not considered for analysis. Only the first eligible inpatient admission was considered for analysis.

Main Predictor Variable

Individuals with a DAMA were analyzed as the case group. A DAMA was identified using the “Patient Status Code” variable, which represents the discharge disposition of each individual. Individuals who were discharged to home/self-care or discharged to a home health organization formed the control group (hereafter referred to as routine discharge).

Demographic, Clinical, and Hospitalization Characteristics

An individual’s age, sex, and region of residence were determined at the date of hospital admission. The Elixhauser algorithm was used to categorize comorbid conditions (as scores of 0, 1-2, ≥3 depending on number of comorbidities) based on International Classification of Diseases, Ninth Revision, Clinical Modification, diagnosis codes during the baseline period.^13,14 The following characteristics of each individual’s eligible inpatient admission were captured: year, timing (weekday or weekend), length of stay (LOS, measured in days), and receipt of a surgical procedure.

Outcomes

All-cause HcRU was identified during the 30-day postdischarge period. Specifically, we identified inpatient readmissions, ED visits, physician office visits, nonphysician outpatient encounters (for example, pathology, radiology, outpatient surgical services), and prescription drug fills. Binary variables (yes or no) were created for inpatient readmissions and ED visits while the remaining HcRU categories (ie, physician office visits, nonphysician outpatient encounters, and prescription drug fills) were analyzed as count variables. In the sensitivity analyses, we provide results for HcRU outcomes among a subgroup of individuals who had at least 90 days of continuous medical and prescription drug benefits following the hospital discharge.

Statistical Analysis

Descriptive Analysis

Measures of interest were reported using summary statistics depending on the nature of the variable. Continuous variables were described using t tests, and categorical variables were described using chi-square tests.

Propensity Score Matching

Cases and controls were matched using a 1:1 greedy matching algorithm based on propensity scores.¹⁵ We developed propensity scores based on confounders that we hypothesized would be associated with a DAMA and postdischarge HcRU. The propensity score model included the following variables: age, sex, region of residence, Elixhauser comorbidity index score, year of admission, timing of admission, LOS, and presence of any surgical procedure during the inpatient admission. The best match between cases and controls was determined based on the absolute difference in their propensity scores, which allowed for a maximal caliper width of 0.2 of the standard deviation of the logit of the propensity score.¹⁶ A standardized difference value of less than 0.1 was used to assess balance in baseline patient and hospital characteristics between cases and controls consistent with prior literature.^17,18 Proportions and balance, as measured by standardized differences between baseline covariates across cases and controls in the matched sample, are displayed in tabular format (Appendix Table 1).

Healthcare Resource Utilization

We estimated the adjusted odds ratio (AOR) using a logistic regression model. The AOR quantified the association between a DAMA and the prevalence of all-cause inpatient readmissions and ED visits during the 30-day postdischarge period. We estimated incident rate ratios (IRR) for count outcomes. Given the large number of individuals with no physician office visits, nonphysician outpatient encounters, or prescription drug fills, we estimated model parameters for IRRs using a finite mixture negative binomial hurdle model.¹⁹ We considered the data to represent a mixture of a constant distribution (which always generates zero counts) and a zero-truncated distribution (which always generates nonzero counts). The finite mixture count models include two outcomes: the mixing probabilities and the count distribution. The mixing probabilities quantify the probability that an observation for the HcRU category will be drawn from either the constant distribution (with mass at zero) or the count distribution. Conditional on having positive values, a zero-truncated generalized linear model (GLM) governs the count variable. Compared with other GLM specifications (eg, Poisson, negative binomial, zero-inflated), the negative binomial hurdle model presented the best-fitting model across several information criteria statistics (Appendix Figures 1-3 and Appendix Tables 2-4).

The GLM results provided IRR for the counts of HcRU. Ratios were interpreted as evidence of increased HcRU (IRR ≥ 1.0) or decreased HcRU (IRR < 1.0) among individuals with a DAMA compared with those discharged routinely. For all HcRU analyses, we reported results for the matched sample. All analyses were conducted using SAS version 9.4 (SAS Institute), and statistical significance was determined at α= .05. The study received the University of Maryland, Baltimore, Institutional Review Board approval (HP-00081497).

The unmatched sample included 457,530 individuals, of whom 0.5% had a DAMA. A consort diagram illustrating cohort inclusion and exclusion criteria is presented in Appendix Figure 4. Demographic, clinical, and inpatient admission characteristics of the unmatched sample and for subgroups defined by discharge status are displayed in Table 1. In the unmatched sample, the median age at admission was higher for individuals with a DAMA than it was for those discharged routinely (43 vs 42 years, respectively), and the proportion of males was higher among those with a DAMA (58.4% vs 33.1%). There were statistically significant differences based on the geographic region of residence and the comorbidity burden across both groups. The median LOS was shorter (1 day vs 2 days), the proportion of weekend admissions was higher (22.2% vs 16.3%), and the proportion of inpatient surgical procedures was lower (12.9% vs 59.2%) among those with a DAMA compared with that among those with routine discharges. The propensity score-matched sample included 2,245 cases and 2,245 controls (Appendix Table 1). Standardized differences for all baseline factors were less than 0.1, indicating that cases and controls were matched on the included baseline factors.

Summary Statistics: Proportions and Counts

Across the DAMA and routine discharge groups, the proportion of individuals with a 30-day inpatient readmission was similar (19.5% vs 18.7%; P = .47), whereas the proportion with an ED visit was higher (18.6% vs 9.1%; P < .01). There were no differences in the median number of inpatient readmissions (median, 0) and ED visits (median, 0) across both groups. Individuals with a DAMA and those discharged routinely displayed similar median counts of 30-day physician office (median, 1) and nonphysician outpatient encounters (median, 1) (Table 2). Individuals with a DAMA displayed a lower median number of prescription drug fills (median, 2 vs 3) than that among those with a routine discharge (Table 2).

Main Analysis: Thirty-Day Healthcare Resource Utilization

The associations between a DAMA and 30-day inpatient readmissions and ED visits based on the matched sample are presented in Table 3. Individuals with a DAMA had increased odds for an ED visit (AOR, 2.28; 95% CI, 1.90-2.72) but no significant difference in the odds of a 30-day inpatient readmission (AOR, 1.06; 95% CI, 0.91-1.23) compared with those discharged routinely.

The association between a DAMA and count HcRU outcomes is presented in Table 4. Compared with those discharged routinely, individuals with a DAMA displayed no significant difference in rates for physician office visits (IRR, 1.01; 95% CI, 0.91-1.11), nonphysician outpatient encounters (IRR, 0.89; 95% CI, 0.78-1.00), and prescription drug fills (IRR, 1.03; 95% CI, 0.97-1.09) during the 30-day postdischarge period.

Sensitivity Analysis: Ninety-Day Healthcare Resource Utilization

Relative to those discharged routinely, individuals with a DAMA had statistically significant increased odds of 90-day inpatient readmissions (AOR, 1.18; 95% CI, 1.02-1.36), odds of ED visits (AOR, 2.16; 95% CI, 1.85-2.51), and rates of prescription drug fills (IRR, 1.32; 95% CI, 1.29-1.35). No statistically significant differences were observed in the rates of physician office visits and nonphysician outpatient encounters across both groups.

In this commercially insured sample of working age individuals, we identified an association between a DAMA and the likelihood and intensity of postdischarge HcRU. The direction of the association varied across categories of HcRU and the duration of follow-up. A DAMA was associated with increased odds of 30-day ED visits but not 30-day readmissions compared with routine discharges. No significant differences were observed in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups. To our knowledge, this is the first study on DAMAs that examines postdischarge HcRU outside the inpatient setting.

The 0.5% prevalence of DAMAs in our study was lower than the approximate 1% to 2% value that is typically reported in the literature. Prior studies have typically reported results based on mixed-payer populations.^3-10 These mixed-payer populations include publicly insured (Medicare or Medicaid) or uninsured stays, which account for a disproportionate share of all DAMAs. In contrast, commercially insured stays account for the lowest proportion of all DAMAs.¹⁰ Similar to prior literature,⁵ the DAMA group in our study was younger, had a higher proportion of males, had a higher comorbidity burden, and had a shorter LOS than the routinely discharged group.

We observed a greater likelihood of ED utilization after a DAMA. Similar findings have been reported, which may indicate that patients with a DAMA receive inadequate treatment at the time of discharge and may require further acute treatment. For example, a prior study reported that, after a DAMA, individuals with asthma were four times more likely to have an ED visit within 14 days compared with those discharged routinely.⁴

Contrary to prior findings,^3-9 we found no significant difference in the odds of a 30-day inpatient readmission across the DAMA and routine discharge groups, which may be attributable to differences in the populations studied. Those previous studies used mixed payer populations and did not differentiate results by payer type. The mixed payer populations in these studies were older (mean ages, 55 years and above) and had an increased comorbidity burden compared with our commercially insured population. Furthermore, some of these studies were either limited to single sites,⁸ single state hospital systems,^3,4,9 or focused on specific medical populations.^3,4,6-9 Our national sample of commercially insured adults is considerably younger, with a mean age of 43 years. Thirty days may be too brief to observe enough inpatient readmissions for the purpose of comparative analyses. This is suggested by our results, which indicated that there is an association between DAMA and 90-day inpatient readmission. Additionally, nonsignificant findings for 30-day inpatient readmissions may also be due to the small sample size of the DAMA group in our study, which may have limited robust statistical inference. Future studies in a larger population of commercially insured individuals with a DAMA are required to confirm these findings.

Nonsignificant differences in the rates of 30-day physician office visits, nonphysician outpatient encounters, and prescription drug fills across both groups may explain the null association with 30-day inpatient readmissions. Prior literature on specific medical populations or individuals with general hospital admissions report that early outpatient follow-up can help prevent 30-day readmissions.^20-25 In our sample, we observed similar rates of outpatient follow-up across the DAMA and routinely discharged groups. Prior studies based on single hospital sites have reported that, at the time of discharge, a lower proportion of individuals with a DAMA received medication prescriptions and outpatient follow-up plans compared with those discharged routinely.^11,12 In contrast, we evaluated prescription drug fills and outpatient visits during the postdischarge period, which may explain the difference in findings.

The present study has several strengths. To the best of our knowledge, our study represents the first and largest retrospective analysis of DAMAs in a national sample of commercially insured adults. In addition to a large generalizable sample, we examine HcRU after a DAMA across major points of service over a longitudinal postdischarge period. Our results provide a comprehensive understanding of utilization outcomes in this population including those outside the inpatient setting, which has been the focus of prior literature. These findings can help guide the implementation of appropriate patient- and system-level interventions to optimize DAMA prevention and mitigate the associated utilization burden on the healthcare system in the postdischarge period.^26,27

Our findings should be interpreted with certain limitations in mind. First, this study used data based on a commercially insured sample of patients and may not be generalizable to publicly insured or uninsured samples. Second, like prior DAMA studies that used the Nationwide Readmissions Database instead,^5-7 our study was unable to account for individual-level factors such as race, marital status, family social support, income, health literacy, and activation in self-care. Further, given the limitations of our data, we were unable to control for hospital characteristics such as bed size, urban-rural designation, teaching status, and control (eg, private or government ownership). Despite the use of propensity score methods to balance both comparison groups on observable sources of confounding, we cannot rule out the possibility of residual confounding. Lastly, due to a lack of data on postdischarge mortality outcomes, we could not control for competing risk of death in our analysis. However, in a population with an average age of 43 years, we did not expect high or differential 30- or 90-day postdischarge mortality rates across both groups.

Our findings suggest several important directions for future research. First, it will be useful to examine these associations among publicly insured and uninsured samples in which a DAMA is more prevalent and in which the associations with HcRU may be more pronounced than they are in the commercially insured population. Secondly, future research should identify subgroups of DAMA patients with an increased propensity for postdischarge HcRU. This can help in the design of individualized outpatient follow-up plans that address patient-specific medical and social needs. Finally, our findings highlight the need for education, practice guidelines, and suitable interventions to help providers in the prevention and management of a DAMA.

Using data from a commercially insured population, we identified associations between a DAMA and postdischarge HcRU. The associations differed by category of HcRU. We identified a positive association with the likelihood of ED utilization but no association with the likelihood of 30-day inpatient readmission or general outpatient utilization. Our results indicate that the examination of inpatient readmissions after a DAMA should not be considered in isolation. The identification of the full range of outpatient and inpatient HcRU after a DAMA in a broad population of patients can improve our understanding of outcomes following a DAMA and support appropriate system-level interventions designed to reduce their prevalence.

The statements, findings, conclusions, views, and opinions contained and expressed in this manuscript are based in part on data obtained under license from IQVIA. Source: IQVIA PharMetrics^® Plus January 2006 – December 2015, IQVIA. All Rights Reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IQVIA or any of its affiliated or subsidiary entities.

Dr Onukwugha reports grants from Bayer Healthcare Pharmaceuticals, grants from Pfizer, Inc, and personal fees from Novo Nordisk outside the submitted work. The other authors have nothing to disclose. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US Department of Veterans Affairs, the U.S. Government, or the VA National Center for Ethics in Health Care.

The authors acknowledge the support of the University of Maryland, Baltimore Institute for Clinical & Translational Research (ICTR) through the ICTR Voucher Program.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.

Hypoxemic respiratory failure is a hallmark of severe coronavirus disease 2019 (COVID-19). Initial guidelines favored early mechanical ventilation (MV) over traditional noninvasive strategies, such as high-flow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NIV), based on perceived ineffectiveness and dangers extrapolated from severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle East respiratory syndrome coronavirus (MERS-CoV) patients.^1,2 As COVID-19 progressed, early MV became associated with prolonged ventilator courses and high mortality.^3-6 Simultaneously, data emerged that HFNC/NIV and self-proning, could successfully stabilize some COVID-19 patients.^7-10 Based on evolving evidence, we implemented a noninvasive COVID-19 respiratory protocol (NCRP) that promoted the early use of HFNC, NIV, and self-proning for hypoxemia in patients with COVID-19, with the intention of avoiding MV in some patients. The protocol was implemented throughout our hospital system, from the Emergency Departments (EDs) to the medical floors and critical care units.

Although preliminary evidence supported the use of HFNC, NIV, and self-proning, the impact of a system-wide noninvasive COVID-19 respiratory protocol on safety has not been well described. The objective of this study was to evaluate patient safety outcomes after implementation of the NCRP, including intubation rate and mortality.

Study Design and Setting

We performed a retrospective chart review, adhering to SQUIRE (Standards for Quality Improvement Reporting Excellence) Guidelines, to assess safety outcomes after implementation of the NCRP.¹¹ Baystate Health is a not-for-profit, integrated healthcare system in western Massachusetts composed of four hospitals and one free-standing ED with 980 beds serving over 800,000 people. The Baystate Health IRB determined that this project did not meet criteria for Human Subjects Research.

Selection of Participants

A consecutive sample of adults (≥18 years old) admitted to the hospital with a positive nucleic acid test for SARS-CoV-2 (reverse transcriptase–polymerase chain reaction [RT-PCR]) test via nasopharyngeal swab (Cepheid or Roche Cobas 6800) between March 15, 2020, and April 15, 2020, were included. Participants were identified by either an order for the COVID-19 test with a positive result or a discharge diagnosis of COVID-19. Daily rapid response team (RRT), intensive care unit (ICU), and COVID-19 unit logs were reviewed to ensure all COVID-19 patients were included. Patients with positive tests admitted for reasons unrelated to COVID-19 infections, such as patients in labor, were excluded.

Interventions

At the start of the COVID-19 pandemic, the Baystate Health system adopted a conservative approach to the respiratory management of patients with COVID-19. This approach started with nasal cannula up to 6 L/min or nonrebreather up to 15 L/min. If the patient remained in respiratory distress, intubation was recommended.

Based on emerging evidence, the NCRP was created. The details of the NCRP implementation have been previously described.¹² Briefly, over a 4-day period (April 3, 2020, to April 7, 2020), a multidisciplinary team developed, refined, and rapidly implemented a COVID-19 respiratory protocol that encouraged the early use of HFNC, NIV, and self-proning in clinically appropriate patients with hypoxemia and respiratory distress due to COVID-19 prior to intubation across all departments of the Baystate Health system (Appendix 1).

Measurements

A chart review was performed using a structured data collection form (Appendix 2). The data collection form was piloted by three physician-researchers. Data abstraction was performed by 16 clinicians. Abstractors were practicing emergency providers and hospitalists and were blinded to the study outcomes. Abstractors received a 1-hour training and abstracted data from at least five charts in parallel with investigators. An additional 10% of charts were double abstracted to calculate interrater reliability for five variables determined a priori.

To validate the capture of outcomes of interest, we triangulated data sources by cross-referencing the monthly RRT log, the ICU list, all orders for HFNC, and RRT activations. Data abstraction occurred from April 21, 2020, to April 30, 2020. Patients who were still hospitalized after April 30,2020, were followed until hospital discharge, ending July 1, 2020.

Outcomes and Analysis

The primary outcome was mortality, defined as the proportion of deaths by admissions during the post–NCRP implementation period (April 3, 2020, to April 15, 2020), compared with the preimplementation period (March 15, 2020, to April 2, 2020). Deaths were stratified by patient code status (do not resuscitate/do not intubate [DNR/DNI] established prior to admission vs Full Code or presumed Full Code). Mortality outcomes were evaluated using one-sided Fisher exact tests.

To assess whether the protocol led to an increase in the use of the interventions and a decrease in intubations, we compared the use of proning, HFNC, NIV, and intubation before the protocol was implemented and with use after. Intubation rates were analyzed using interrupted time series (piecemeal regression), without adjustments, using a cut point of April 2, 2020.

Secondary outcomes included unexpected cardiac arrests, ICU transfers and consultations, and RRT activations during the postimplementation period, compared with the preimplementation period. Secondary outcomes were evaluated using standard chi-square tests (χ²). Additional descriptive outcomes included use of the NCRP, overall and by components, and in-hospital rates of MV.

From March 15, 2020, through April 15, 2020, there were 469 patients with COVID-19 admitted to the four hospitals of the Baystate Health system. Patients had an average age of 70 years (SD, 16.4), 241 (52%) were female, and 336 (72%) spoke English as their primary language. Most patients, 405 (86.4%), required supplemental oxygen upon being admitted to the hospital (Table 1).

Postimplementation Mortality

Overall, 123 (26.2%) patients died during the study period. In the preimplementation cohort, 24% (61 of 254) of patients died, compared with 28.8% (62 of 215) in the postimplementation cohort (one-sided Fisher exact, P = .14). Excluding patients with an established DNR/DNI prior to admission, 21.8% (48 of 220) patients died in the preimplementation period vs 21.9% (35 of 160) patients after implementation of the NCRP (Table 2).

Secondary Safety Outcomes

There was no increase in RRT activations (preimplementation, 16.5% [42 of 254], vs postimplementation, 11.6% [25 of 215]; χ²P = 0.17) or ICU consultations (preimplementation, 18.1% [47 of 254], vs postimplementation, 16.3% [35 of 215]; χ²P = 0.52). ICU transfers decreased in the postimplementation period (preimplementation, 26.8% [68 of 254], vs postimplementation, 13.5% [29 of 215], χ²P < .001). There was one unexpected cardiac arrest documented in the postimplementation period, compared to none before implementation.

NCRP Protocol Implementation

After implementation, the proportion of patients using HFNC increased from 5.5% (14 of 254) to 24.7% (53 of 215), and self-proning increased from 7.5% (19 of 254) to 22.8% (49 of 215). The proportion of patients who were intubated (MV) decreased from 25.2% (64 of 254) to 10.7% (23 of 215) (χ²P < .01). Interrupted time series analysis demonstrated an immediate reduction in the proportion of patients intubated after the intervention (incident rate ratio, 0.44; 95% CI, 0.23-0.83; P = .012) (Figure). The median time from admission to MV was longer in the postimplementation period patients (postimplementation, 1.4 days; interquartile range, 0.21-2.9; vs preimplementation, 0.66 days; IQR 0.23-1.69).

Interrater Reliability

Interrater reliability for variables chosen a priori was k = 1.0 for self-proning, k = 1.0 for intubation, k = 0.95 for discharge disposition, k = 0.94 for nasal cannula, and k = 0.74 for HFNC.

The rapid spread of SARS-CoV-2 led to early recommendations based on minimal data. As evidence emerged, hospitals were forced to adapt to protect patients and medical providers. As a healthcare system, we incorporated emerging evidence to rapidly implement a noninvasive respiratory treatment protocol. Aware of the methodological problems in evaluating the NCRP itself, we integrated best practices of quality improvement to examine multiple patient safety outcomes after NCRP implementation. We found the rate of intubation decreased with no significant increase in mortality, ICU transfers, RRT activations, or unexpected deaths after the implementation of the NCRP.

Although we were unable to measure all confounders and changes that co-occurred during the study period, initial vital signs, age, BMI, past medical history, and use of oxygen were similar between the pre- and postimplementation cohorts. Further, there were many constants worth noting. First, COVID-19 respiratory protocols were highly regulated to ensure patient safety and minimize COVID-19 transmission. Second, there were no new nonrespiratory treatments or medications during the study. Third, although the COVID-19 hospital census rose during the study, it never overwhelmed resources; there was no rationing of clinical care.

The nonsignificant increase in mortality in the postimplementation period was limited to patients with an established DNR/DNI prior to admission. Established DNR/DNI patients were largely from skilled nursing facilities that were disproportionally impacted in the postimplementation period through clustered outbreaks of COVID-19 in our region, which likely contributed to the increased mortality.¹³

Additionally, despite decreased MV rates in the postimplementation period, we did not find a concurrent decrease in mortality. We do not believe this is a failure of noninvasive treatments. Rather, the increased proportion of DNR/DNI patients, combined with increased nursing home outbreaks in the postimplementation period likely influenced mortality. The postimplementation decreases in ICU transfers and RRT activations supports this hypothesis.

Finally, it is worth nothing that, although the goal of decreasing intubations was to improve patient care and decrease mortality, a decrease in intubations alone, without a change in mortality, may be important because mechanical ventilation has been associated with increased morbidity, such as posttraumatic stress disorder.¹⁴

Taken together, the post–NCRP implementation period appears to have been safe for patients, compared to the preimplementation period’s protocol. Future research may help understand the impact of specific noninvasive interventions on COVID-19–related MV and mortality.

Limitations

Given the urgency of COVID-19 treatment, the NCRP was designed as a quality improvement initiative rather than a prospective trial. Issues of selection bias and confounding limit our ability to evaluate the effect of the NCRP itself. Additionally, unmeasured patient and provider factors may have influenced outcomes. For example, increased provider knowledge and experience treating COVID-19 may have improved outcomes over time, and unmeasured patient characteristics may have been different in the pre- and postimplementation groups. Finally, our study was limited to a single healthcare system, which may limit generalizability

That said, the objective of our study was to evaluate patient safety outcomes of the NCRP, an important first step while other hospital systems continue to confront increasing rates of COVID-19 and must decide on appropriate respiratory management. To that end, our enrollment captured 469 COVID-19 admissions across four diverse hospitals without obvious differences in initial measured covariates. Further, the strict protocolization of respiratory treatments, the evaluation of multiple safety outcomes, and the complete patient follow-up all support the conclusion that NCRP in the postimplementation period did not increase adverse patient outcomes. Further studies are needed to determine the efficacy of the NCRP protocol itself.

In our health system, patients with COVID-19 did not experience a significant increase in mortality, RRT activations, or ICU admissions despite decreased rates of MV after implementation of a respiratory protocol that encouraged early noninvasive management of COVID-19 respiratory distress.

The authors would like to acknowledge Elizabeth Coray, Joseph Lahey, Richard Gabor, Cheryl Greenstein, Sarah Badach, Marie Boutin, Adrienne Wurl, Anthony Kitchen, Michelle Holton, Matthew Shapiro, Eleanor Ragone, Nageshwar Jonnalagadda, Ryan Flynn, Raghuveer Rakasi, and Jasmine Paadam.