Hospital Medicine

India is emerging as one of the world’s largest hotspots for SARS-CoV-2 infection (COVID-19)—second only to the United States—with more than 13,000,000 documented infections since the first case was recorded on January 30, 2020.^1,2 Kashmir, a northern territory of India, reported its first case of COVID-19 on March 18, 2020, from the central District Srinagar; this region has accounted for more cases of COVID-19 than any other district throughout the pandemic.³ The large majority of healthcare in District Srinagar is provided by three tertiary care institutions, one district hospital, two subdistrict hospitals, and 70 primary healthcare centers. Potential occupational exposures place healthcare workers (HCWs) at higher risk of acquiring SARS-CoV-2 infection, which in turn may serve as an important source of infection for their families and other community members.^4-6 Given the high frequency and geographic variability of asymptomatic infection, growing evidence suggests this hidden reservoir is a source of infection for the general population.^7,8

Many countries have started testing for antibodies against SARS-CoV-2, both at the population level and in specific groups, such as HCWs. Seroepidemiological studies are crucial to understanding the dynamics of SARS-CoV-2 infection. Many seroepidemiological studies have been conducted among community populations, but there are insufficient data on HCWs. The World Health Organization also encouraged its member states to conduct seroepidemiological studies to attain a better understanding of COVID-19 infection prevalence and distribution.^9-11 Therefore, to quantify the prevalence of SARS-CoV-2 infection among HCWs, we conducted a seroepidemiological study by testing for SARS-CoV-2–specific immunoglobulin (IgG) to gain insight into the extent of infection among specific subgroups of HCWs and to identify risk-factor profiles associated with seropositivity.

Study Design and Settings

We conducted this seroepidemiological study to ascertain the presence of IgG antibodies against SARS-CoV-2 among HCWs in the District Srinagar of Kashmir, India. The 2-week period of data collection began on June 15, 2020. As part of healthcare system pandemic preparedness efforts, India’s Ministry of Health provided specific guidelines for health facilities to manage COVID-19. Hospitals were categorized as dedicated COVID and non-COVID hospitals. Dedicated COVID hospitals provided comprehensive care exclusively to patients with COVID-19 and were equipped with fully functional intensive care units, ventilators, and beds with reliable access to oxygen support.¹² In addition, infection prevention and control strategies to limit the transmission of SARS-CoV-2 infection were implemented according to guidelines specified by India’s National Center for Disease Control.¹³ To strengthen service provision, HCWs from other hospitals, including resident physicians, were relocated to these dedicated COVID hospitals. The additional staff were selected by administrative leadership, without input from HCWs.

Study Population and Data Collection

We approached administrative heads of the hospitals in District Srinagar for permission to conduct our study and to invite their HCWs to participate in the study. As Figure 1 shows, we were denied permission by the administrative heads of two tertiary care hospitals. Finally, with a point person serving as a study liaison at each institution, HCWs from three dedicated COVID and seven non-COVID tertiary care hospitals, two subdistrict hospitals, and six primary healthcare centers across the District Srinagar were invited to participate. The sample primary healthcare centers were each selected randomly, after stratification, from six major regions of the district. All frontline HCWs, including physicians, administrative and laboratory personnel, technicians, field workers involved in surveillance activity, and other supporting staff were eligible for the study.

We collected information on an interview form using Epicollect5, a free data-gathering tool widely used in health research.¹⁴ Physicians specifically trained in the use of Epicollect5 conducted the face-to-face interview on a prespecified day and recorded the collected information through mobile phones. This information included the participants’ role in providing care to patients with COVID-19 and risk factors for SARS-CoV-2 infection (eg, history of travel since January 1, 2020, symptoms of an influenza-like illness [ILI] in the 4 weeks prior to the interview, close contact with a COVID-19 case). We defined close contact as an unmasked exposure within 6 feet of an infected individual for at least 15 minutes, irrespective of location (ie, community or the hospital).

Following the interview, trained phlebotomists collected 3 to 5 mL of venous blood under aseptic conditions. We strictly adhered to standard operating procedures during collection, transportation, and testing of blood samples. Following collection, the blood samples remained undisturbed for at least 30 minutes before centrifugation, which was performed at the collection site (or at the central laboratory for sites lacking the capability). The samples were then transported for further processing and testing through a cold chain supply line, using vaccine carriers with conditioned icepacks. All testing procedures were conducted with strict adherence to the manufacturers’ guidelines.

Laboratory Procedure

In accordance with the manufacturer’s recommendations, we used a chemiluminescent microparticle immunoassay to detect SARS-CoV-2–specific IgG antibodies in serum samples. The assay is an automated two-step immunoassay for the qualitative detection of IgG antibodies against the nucleocapsid of SARS-CoV-2 in human serum and plasma. The sensitivity and specificity of this test are 100% and 99%, respectively. The test result was considered positive for SARS-CoV-2 IgG if the index value was ≥1.4, consistent with guidance provided by the manufacturer.¹⁵

The IgG values were also entered into Epicollect5. Two trained medical interns independently entered the laboratory results in two separate forms. A third medical intern reviewed these forms for discrepancies, in response to which they referenced the source data for adjudication. The information gathered during the interview and the laboratory results were linked with the help of a unique identification number, which was generated at the time of the interview.

Statistical Analysis

We estimated the proportion (and logit-transformed 95% CI) of HCWs with a positive SARS-CoV-2–specific IgG antibody level, the primary outcome of interest. We compared seroprevalence rates by gender, age group, specific occupational group, and type of health facility (dedicated COVID hospital vs non-COVID hospital). Seroprevalence was also estimated separately for HCWs who reported symptoms in the past 4 weeks, had a history of exposure to a known case of COVID-19, or had undergone testing by reverse transcriptase-polymerase chain reaction (RT-PCR). In the case of zero seroprevalences, Jeffreys 95% CIs were reported. We used a chi-square test to report two-sided P values for comparison of seroprevalence between groups. When the expected frequency was <5 in more than 20% of the cells, the exact test was used instead of the chi-square test. We additionally performed multivariable logistic regression analysis to evaluate the independent association between place of work (primary independent variable) and seropositivity (dependent variable). We adjusted for the following observable covariates by including them as categorical variables: age, gender, occupational group, and history of close contact with a patient who was COVID-positive. We performed data analysis using Stata, version 15.1 (StataCorp LP). The Institutional Ethics Committee of Government Medical College, Srinagar, approved the study (Reference No. 1003/ETH/GMC dated 13-05-2020). We obtained written, informed consent from all participants.

Of the 7,346 HCWs we were granted permission to approach, 2,915 (39.7%) agreed to participate in the study. The participation rate was 49% at the dedicated COVID hospitals (57% physicians and 47% nonphysicians) and 39% at the non-COVID hospitals (46% physicians and 36% nonphysicians). We analyzed information gathered from 2,905 HCWs (Epicollect5 interview forms were missing for nine participants, and the laboratory report was missing for one participant).

The mean age of the participants was 38.6 years, and 35.8% of participants identified as female (Table 1). One third (33.7%) of the participants were physicians, nearly half of whom were residents. In our sample, the overall seroprevalence of SARS-CoV-2–specific antibodies was 2.5% (95% CI, 2.0%-3.1%).

Of the 2,905 participating HCWs, 123 (4.2%) reported an ILI (ie, fever and cough) in the 4 weeks preceding the interview, and 339 (11.7%) reported close contact with a person with COVID-19 (Table 2). A total of 760 (26.2%) HCWs had undergone RT-PCR testing, 29 (3.8%) of whom had a positive result. Stratifying by workplace, history of nasopharyngeal RT-PCR positivity was reported by 4 of 77 (5.1%) participants from dedicated COVID hospitals compared to (3.7%) participants from the non-COVID hospital (P = .528).

As Table 2 also demonstrates, we found a significantly higher seropositivity rate among HCWs who had a history of ILI (P < .001), a history of positive RT-PCR (P < .001), history of ever being put under quarantine (P = .009), and a self-reported history of close contact with a person with COVID-19 (P = .014). Healthcare workers who had ever worked at a dedicated COVID hospital had a significantly lower seroprevalence of infection (P = .004).

Among HCWs who reported no ILI symptoms in the 4 weeks prior to the interview but who had positive RT-PCR test, 20.8% were seropositive. Of HCWs who reported both ILI and a positive RT-PCR test result, 60.0% were seropositive. Compared to employment at a non-COVID hospital, HCWs working in dedicated COVID hospitals had a reduced multivariate-adjusted risk of seropositivity (odds ratio, 0.21; 95% CI, 0.06-0.66).

We aimed to estimate the seroprevalence of SARS-CoV-2 infection in HCWs in different hospital settings in the District Srinagar of Kashmir, India. In general, seroprevalence was low (2.5%), with little difference across gender or occupational group.

Seroprevalence studies of HCWs across divergent workplace environments have revealed estimates ranging from 1% to 10.2%.^16-19 Generally, the seroprevalence rates among HCWs are not significantly different from those of the general population, which reflects how different the dynamics of COVID-19 are compared to other infections in healthcare settings. The low seroprevalence observed in our study coincides with the overall low infection rate in the community population. During the study period, District Srinagar reported a median of 28 new infections daily (interquartile range, 17-46), which is indicative of the early phase of the pandemic in the population at the time of the study.²⁰

Among the HCW occupational groups, ambulance drivers and housekeeping staff had the highest seroprevalence rates, followed by nurses and physicians. Possible explanations for higher seropositivity in these groups are improper use or inadequate supply of protective gear and lack of training on the use of personal protective equipment (PPE), resulting in increased exposure risk.²¹ Concordance of HCW and community infection rates in specific geographic areas suggests that community exposure may be the dominant source of healthcare exposure and infection. Additionally, careful in-hospital behavior of HCWs in dedicated COVID hospitals may have had a spillover effect on their out-of-hospital behavior, which may partially explain our finding that employment at dedicated COVID hospitals was associated with a markedly lower chance of seropositivity. A study of 6,510 HCWs in Chicago, Illinois, showed high seropositivity rates among support service workers, medical assistants, and nurses, with nurses identified as having a markedly higher adjusted odds of seropositivity relative to administrators. The authors of the study concluded that exposure in the community setting plays a crucial role in transmission among HCWs.²² Similarly, higher seroprevalence among housekeeping, nonadministrative staff, and other support service staff has been reported elsewhere.²³ Certain underlying factors related to socioeconomic status and lifestyle may also contribute to higher seroprevalence in some occupational groups.²⁴ Nonadherence to masking, social distancing, and proper hand hygiene outside the hospital setting could result in community-acquired infection.

Interestingly, participants who were working in a dedicated COVID hospital or who had ever worked at one had a seroprevalence of 0.6%, much lower than the 2.8% observed among other participants. This difference remained statistically significant after controlling for age, sex, place of work, and occupational group. As these facilities were dedicated to the management and care of patients with COVID-19, the hospital staff strictly adhered to safety precautions, with particular vigilance during patient contact. These hospitals also strictly adhered to infection prevention and control practices based on the latest guidelines released by India’s Ministry of Health and Family Welfare.¹³

A commitment was made to provide adequate PPE to the dedicated COVID hospitals and staff, commensurate with expected infected patient volumes and associated exposure risks. Healthcare workers were specifically trained on proper donning and doffing of PPE, self-health monitoring, and protocols for reporting symptoms and PPE breaches during patient encounters. Healthcare workers were regularly tested for COVID-19 using nasopharyngeal RT-PCR. Of critical importance, these hospitals implemented a buddy system wherein a team of two or more staff members was responsible for ensuring each other’s safety, proper PPE use, conformance to other protective measures, and reporting breaches of PPE compliance.²⁵ Universal masking was mandatory for all hospital staff and patients at the COVID-focused facilities, with the additional use of N-95 masks, gloves, and face shields during times of patient contact. Administrative measures, including visitor restrictions and environmental sanitation, were rigorously enforced. Also, being a potentially high-risk area for transmission of infection, these facilities implemented staff-rationing to reduce the duration of exposure to the healthcare staff. Third, the HCWs of COVID-dedicated hospitals were provided with separate living accommodations during the period in which they were employed at a dedicated COVID hospital.

In contrast, in non-COVID hospitals, with the exception of HCWs, patients and the hospital visitors were not subject to a masking policy. Moreover, an adequate and timely supply of PPE was not prioritized at the non-COVID facilities due to resource constraints. Further, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. Though routine infection prevention and control activities were performed at non-COVID hospitals, we did not assess adherence to infection prevention and control guidelines in the two different categories of hospitals. Our results are also supported by evidence from studies conducted in different hospital settings, the findings of which reiterate the importance of fundamental principles of prevention (eg, proper masking, hand hygiene, and distancing) and are of particular importance in resource-limited settings.^17,26,27 The only published study quantifying seroprevalence among HCWs in India was performed in a single hospital setting with separate COVID and non-COVID units. The authors of that study reported a higher seroprevalence among HCWs in the COVID unit. However, this difference seems to be confounded by other factors as revealed by the multivariable analysis result.²³

We found a two-fold higher seroprevalence (4.4%) in HCWs who reported close contact with a patient with COVID-19. Respiratory infections pose a greater health risk to HCWs in an occupational setting. Substantial evidence has emerged demonstrating that the respiratory system is the dominant route of SARS-CoV-2 transmission, with proximity and ventilation as key predictive factors.²⁸ Globally, among thousands of HCWs infected with SARS-CoV-2, one of the leading risk factors identified was close contact with a patient with COVID-19; other identified risk factors were lack of PPE, poor infection prevention and control practices, work overload, and a preexisting health condition.²⁹

The seroprevalence estimate among participants who reported an ILI in the 4 weeks preceding the interview was only 12.2%, suggesting an alternative etiology of these symptoms. Among those who reported a previously positive RT-PCR for SARS-CoV-2, only 27.6% showed the presence of SARS-CoV-2–specific IgG antibodies. The inability to mount an antibody-mediated immune response or early conversion to seronegative status during the convalescence phase has been suggested as an explanation for such discordant findings.³⁰ On the contrary, seropositivity among participants who reported having a negative RT-PCR test was 1.9%. There are few plausible explanations for such observations. First, several studies have reported false-negative result rates from RT-PCR testing ranging from 2% to 29%.^31-33 Second, the sensitivity of the SARS-CoV-2 assay is influenced by the timing of the test after the onset of symptoms or RT-PCR positivity. The sensitivity of the assay we used varies from 53.1% at day 7 to 100% at day 17 postinfection.³⁴ Variable viral load and differences in duration of viral shedding are other possible reasons for false-negative RT-PCR results.^35,36

In our study, seroconversion among asymptomatic HCWs who were RT-PCR-positive was 20.8%. Among HCWs who reported an ILI and were RT-PCR-positive, seropositivity was 60%. In one study, 40% of asymptomatic and 13% of symptomatic patients who tested positive for COVID-19 became seronegative after initial seropositivity—that is, 8 weeks after hospital discharge.³⁷

Serological testing offers insight into both the exposure history and residual COVID-19 susceptibility of HCWs. However, current immunological knowledge does not allow us to conclude that seropositivity conveys high-level immunity against reinfection. As the epidemic evolves, HCWs will continue to be exposed to COVID-19 in the community and the workplace. Serial cross-sectional serosurveys can help monitor the progression of the pandemic within the healthcare setting and guide hospital authorities in resource allocation.

Strengths and Limitations

We used the Abbott Architect SARS-CoV-2 IgG assay, which has exhibited a high level of consistency and performance characteristics when tested in different patient populations. The participation rate was acceptable compared to similar studies, and we included all the major hospitals in the District Srinagar. The findings from our study can therefore be considered representative of the HCWs in the district.

The study results should be interpreted in the context of the following limitations. First, information on risk factors for seropositivity were based on participant report. Also, we did not collect information on the timing of symptoms or the date on which a participant became RT-PCR-positive. Second, information regarding place of exposure (ie, community or hospital setting) was not recorded, limiting conclusions regarding the effect of workplace exposures. Third, given the voluntary nature of participation in the study, there is a possibility of selection bias that may have limited the generalizability of our findings. For example, some HCWs with a recent exposure to COVID-19 or those who were symptomatic at the time of the study might not have participated based on the absence of an individual benefit from IgG testing in the early phase of infection. Conversely, some HCWs who had symptoms in the distant past might have been more likely to have participated in the study. However, we believe that selection bias does not vitiate the validity of the associations based on the plausible assumption that infection risk should be similar between respondents and nonrespondents due to comparable work environments. Finally, with a cross-sectional study design, we cannot ascertain the reconversion from an initial positive-IgG to negative-IgG status, which warrants a cohort study.

We conclude that the seroprevalence of SARS-CoV-2 infection was low among HCWs of District Srinagar at the time of the study. Healthcare workers in a dedicated COVID hospital or HCWs who had ever worked in such a facility had lower seroprevalence, suggesting both adherence to and effectiveness of standard protective measures during contact with patients who had COVID-19. Nonetheless, the careful in-hospital behavior of the HCWs at the COVID hospitals may have had a spillover effect on their out-of-hospital behaviors, which lead to community-acquired infection. On the contrary, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. We believe that our findings highlight the value of implementing infection prevention and control measures in the hospital setting. Moreover, training and retraining of sanitation and other housekeeping staff on standard hygienic practices and appropriate use of the protective gear may further help reduce their rates of exposure.

The authors thank Principal and Dean of the Government Medical College, Srinagar, Professor Samia Rashid, and District Commissioner, Srinagar, Shahid Iqbal Chowdhary for their support. We also acknowledge the support rendered by the Directorate of Health Services, Kashmir; Chief Medical Officer Srinagar; Block Medical Officers; and Zonal Medical Officers of District Srinagar, Kashmir, and extend our appreciation to the medical interns for their efforts in data collection, and to laboratory in-charge Gulzar Ahmad Wani, PhD scholar, Biochemistry, and his staff, who were involved in this study. Finally, we thank the study participants for their understanding of the importance of this study and for their time and participation.

Data shall be made available on request through the corresponding author.

India is emerging as one of the world’s largest hotspots for SARS-CoV-2 infection (COVID-19)—second only to the United States—with more than 13,000,000 documented infections since the first case was recorded on January 30, 2020.^1,2 Kashmir, a northern territory of India, reported its first case of COVID-19 on March 18, 2020, from the central District Srinagar; this region has accounted for more cases of COVID-19 than any other district throughout the pandemic.³ The large majority of healthcare in District Srinagar is provided by three tertiary care institutions, one district hospital, two subdistrict hospitals, and 70 primary healthcare centers. Potential occupational exposures place healthcare workers (HCWs) at higher risk of acquiring SARS-CoV-2 infection, which in turn may serve as an important source of infection for their families and other community members.^4-6 Given the high frequency and geographic variability of asymptomatic infection, growing evidence suggests this hidden reservoir is a source of infection for the general population.^7,8

Many countries have started testing for antibodies against SARS-CoV-2, both at the population level and in specific groups, such as HCWs. Seroepidemiological studies are crucial to understanding the dynamics of SARS-CoV-2 infection. Many seroepidemiological studies have been conducted among community populations, but there are insufficient data on HCWs. The World Health Organization also encouraged its member states to conduct seroepidemiological studies to attain a better understanding of COVID-19 infection prevalence and distribution.^9-11 Therefore, to quantify the prevalence of SARS-CoV-2 infection among HCWs, we conducted a seroepidemiological study by testing for SARS-CoV-2–specific immunoglobulin (IgG) to gain insight into the extent of infection among specific subgroups of HCWs and to identify risk-factor profiles associated with seropositivity.

Study Design and Settings

We conducted this seroepidemiological study to ascertain the presence of IgG antibodies against SARS-CoV-2 among HCWs in the District Srinagar of Kashmir, India. The 2-week period of data collection began on June 15, 2020. As part of healthcare system pandemic preparedness efforts, India’s Ministry of Health provided specific guidelines for health facilities to manage COVID-19. Hospitals were categorized as dedicated COVID and non-COVID hospitals. Dedicated COVID hospitals provided comprehensive care exclusively to patients with COVID-19 and were equipped with fully functional intensive care units, ventilators, and beds with reliable access to oxygen support.¹² In addition, infection prevention and control strategies to limit the transmission of SARS-CoV-2 infection were implemented according to guidelines specified by India’s National Center for Disease Control.¹³ To strengthen service provision, HCWs from other hospitals, including resident physicians, were relocated to these dedicated COVID hospitals. The additional staff were selected by administrative leadership, without input from HCWs.

Study Population and Data Collection

We approached administrative heads of the hospitals in District Srinagar for permission to conduct our study and to invite their HCWs to participate in the study. As Figure 1 shows, we were denied permission by the administrative heads of two tertiary care hospitals. Finally, with a point person serving as a study liaison at each institution, HCWs from three dedicated COVID and seven non-COVID tertiary care hospitals, two subdistrict hospitals, and six primary healthcare centers across the District Srinagar were invited to participate. The sample primary healthcare centers were each selected randomly, after stratification, from six major regions of the district. All frontline HCWs, including physicians, administrative and laboratory personnel, technicians, field workers involved in surveillance activity, and other supporting staff were eligible for the study.

We collected information on an interview form using Epicollect5, a free data-gathering tool widely used in health research.¹⁴ Physicians specifically trained in the use of Epicollect5 conducted the face-to-face interview on a prespecified day and recorded the collected information through mobile phones. This information included the participants’ role in providing care to patients with COVID-19 and risk factors for SARS-CoV-2 infection (eg, history of travel since January 1, 2020, symptoms of an influenza-like illness [ILI] in the 4 weeks prior to the interview, close contact with a COVID-19 case). We defined close contact as an unmasked exposure within 6 feet of an infected individual for at least 15 minutes, irrespective of location (ie, community or the hospital).

Following the interview, trained phlebotomists collected 3 to 5 mL of venous blood under aseptic conditions. We strictly adhered to standard operating procedures during collection, transportation, and testing of blood samples. Following collection, the blood samples remained undisturbed for at least 30 minutes before centrifugation, which was performed at the collection site (or at the central laboratory for sites lacking the capability). The samples were then transported for further processing and testing through a cold chain supply line, using vaccine carriers with conditioned icepacks. All testing procedures were conducted with strict adherence to the manufacturers’ guidelines.

Laboratory Procedure

In accordance with the manufacturer’s recommendations, we used a chemiluminescent microparticle immunoassay to detect SARS-CoV-2–specific IgG antibodies in serum samples. The assay is an automated two-step immunoassay for the qualitative detection of IgG antibodies against the nucleocapsid of SARS-CoV-2 in human serum and plasma. The sensitivity and specificity of this test are 100% and 99%, respectively. The test result was considered positive for SARS-CoV-2 IgG if the index value was ≥1.4, consistent with guidance provided by the manufacturer.¹⁵

The IgG values were also entered into Epicollect5. Two trained medical interns independently entered the laboratory results in two separate forms. A third medical intern reviewed these forms for discrepancies, in response to which they referenced the source data for adjudication. The information gathered during the interview and the laboratory results were linked with the help of a unique identification number, which was generated at the time of the interview.

Statistical Analysis

We estimated the proportion (and logit-transformed 95% CI) of HCWs with a positive SARS-CoV-2–specific IgG antibody level, the primary outcome of interest. We compared seroprevalence rates by gender, age group, specific occupational group, and type of health facility (dedicated COVID hospital vs non-COVID hospital). Seroprevalence was also estimated separately for HCWs who reported symptoms in the past 4 weeks, had a history of exposure to a known case of COVID-19, or had undergone testing by reverse transcriptase-polymerase chain reaction (RT-PCR). In the case of zero seroprevalences, Jeffreys 95% CIs were reported. We used a chi-square test to report two-sided P values for comparison of seroprevalence between groups. When the expected frequency was <5 in more than 20% of the cells, the exact test was used instead of the chi-square test. We additionally performed multivariable logistic regression analysis to evaluate the independent association between place of work (primary independent variable) and seropositivity (dependent variable). We adjusted for the following observable covariates by including them as categorical variables: age, gender, occupational group, and history of close contact with a patient who was COVID-positive. We performed data analysis using Stata, version 15.1 (StataCorp LP). The Institutional Ethics Committee of Government Medical College, Srinagar, approved the study (Reference No. 1003/ETH/GMC dated 13-05-2020). We obtained written, informed consent from all participants.

Of the 7,346 HCWs we were granted permission to approach, 2,915 (39.7%) agreed to participate in the study. The participation rate was 49% at the dedicated COVID hospitals (57% physicians and 47% nonphysicians) and 39% at the non-COVID hospitals (46% physicians and 36% nonphysicians). We analyzed information gathered from 2,905 HCWs (Epicollect5 interview forms were missing for nine participants, and the laboratory report was missing for one participant).

The mean age of the participants was 38.6 years, and 35.8% of participants identified as female (Table 1). One third (33.7%) of the participants were physicians, nearly half of whom were residents. In our sample, the overall seroprevalence of SARS-CoV-2–specific antibodies was 2.5% (95% CI, 2.0%-3.1%).

Of the 2,905 participating HCWs, 123 (4.2%) reported an ILI (ie, fever and cough) in the 4 weeks preceding the interview, and 339 (11.7%) reported close contact with a person with COVID-19 (Table 2). A total of 760 (26.2%) HCWs had undergone RT-PCR testing, 29 (3.8%) of whom had a positive result. Stratifying by workplace, history of nasopharyngeal RT-PCR positivity was reported by 4 of 77 (5.1%) participants from dedicated COVID hospitals compared to (3.7%) participants from the non-COVID hospital (P = .528).

As Table 2 also demonstrates, we found a significantly higher seropositivity rate among HCWs who had a history of ILI (P < .001), a history of positive RT-PCR (P < .001), history of ever being put under quarantine (P = .009), and a self-reported history of close contact with a person with COVID-19 (P = .014). Healthcare workers who had ever worked at a dedicated COVID hospital had a significantly lower seroprevalence of infection (P = .004).

Among HCWs who reported no ILI symptoms in the 4 weeks prior to the interview but who had positive RT-PCR test, 20.8% were seropositive. Of HCWs who reported both ILI and a positive RT-PCR test result, 60.0% were seropositive. Compared to employment at a non-COVID hospital, HCWs working in dedicated COVID hospitals had a reduced multivariate-adjusted risk of seropositivity (odds ratio, 0.21; 95% CI, 0.06-0.66).

We aimed to estimate the seroprevalence of SARS-CoV-2 infection in HCWs in different hospital settings in the District Srinagar of Kashmir, India. In general, seroprevalence was low (2.5%), with little difference across gender or occupational group.

Seroprevalence studies of HCWs across divergent workplace environments have revealed estimates ranging from 1% to 10.2%.^16-19 Generally, the seroprevalence rates among HCWs are not significantly different from those of the general population, which reflects how different the dynamics of COVID-19 are compared to other infections in healthcare settings. The low seroprevalence observed in our study coincides with the overall low infection rate in the community population. During the study period, District Srinagar reported a median of 28 new infections daily (interquartile range, 17-46), which is indicative of the early phase of the pandemic in the population at the time of the study.²⁰

Among the HCW occupational groups, ambulance drivers and housekeeping staff had the highest seroprevalence rates, followed by nurses and physicians. Possible explanations for higher seropositivity in these groups are improper use or inadequate supply of protective gear and lack of training on the use of personal protective equipment (PPE), resulting in increased exposure risk.²¹ Concordance of HCW and community infection rates in specific geographic areas suggests that community exposure may be the dominant source of healthcare exposure and infection. Additionally, careful in-hospital behavior of HCWs in dedicated COVID hospitals may have had a spillover effect on their out-of-hospital behavior, which may partially explain our finding that employment at dedicated COVID hospitals was associated with a markedly lower chance of seropositivity. A study of 6,510 HCWs in Chicago, Illinois, showed high seropositivity rates among support service workers, medical assistants, and nurses, with nurses identified as having a markedly higher adjusted odds of seropositivity relative to administrators. The authors of the study concluded that exposure in the community setting plays a crucial role in transmission among HCWs.²² Similarly, higher seroprevalence among housekeeping, nonadministrative staff, and other support service staff has been reported elsewhere.²³ Certain underlying factors related to socioeconomic status and lifestyle may also contribute to higher seroprevalence in some occupational groups.²⁴ Nonadherence to masking, social distancing, and proper hand hygiene outside the hospital setting could result in community-acquired infection.

Interestingly, participants who were working in a dedicated COVID hospital or who had ever worked at one had a seroprevalence of 0.6%, much lower than the 2.8% observed among other participants. This difference remained statistically significant after controlling for age, sex, place of work, and occupational group. As these facilities were dedicated to the management and care of patients with COVID-19, the hospital staff strictly adhered to safety precautions, with particular vigilance during patient contact. These hospitals also strictly adhered to infection prevention and control practices based on the latest guidelines released by India’s Ministry of Health and Family Welfare.¹³

A commitment was made to provide adequate PPE to the dedicated COVID hospitals and staff, commensurate with expected infected patient volumes and associated exposure risks. Healthcare workers were specifically trained on proper donning and doffing of PPE, self-health monitoring, and protocols for reporting symptoms and PPE breaches during patient encounters. Healthcare workers were regularly tested for COVID-19 using nasopharyngeal RT-PCR. Of critical importance, these hospitals implemented a buddy system wherein a team of two or more staff members was responsible for ensuring each other’s safety, proper PPE use, conformance to other protective measures, and reporting breaches of PPE compliance.²⁵ Universal masking was mandatory for all hospital staff and patients at the COVID-focused facilities, with the additional use of N-95 masks, gloves, and face shields during times of patient contact. Administrative measures, including visitor restrictions and environmental sanitation, were rigorously enforced. Also, being a potentially high-risk area for transmission of infection, these facilities implemented staff-rationing to reduce the duration of exposure to the healthcare staff. Third, the HCWs of COVID-dedicated hospitals were provided with separate living accommodations during the period in which they were employed at a dedicated COVID hospital.

In contrast, in non-COVID hospitals, with the exception of HCWs, patients and the hospital visitors were not subject to a masking policy. Moreover, an adequate and timely supply of PPE was not prioritized at the non-COVID facilities due to resource constraints. Further, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. Though routine infection prevention and control activities were performed at non-COVID hospitals, we did not assess adherence to infection prevention and control guidelines in the two different categories of hospitals. Our results are also supported by evidence from studies conducted in different hospital settings, the findings of which reiterate the importance of fundamental principles of prevention (eg, proper masking, hand hygiene, and distancing) and are of particular importance in resource-limited settings.^17,26,27 The only published study quantifying seroprevalence among HCWs in India was performed in a single hospital setting with separate COVID and non-COVID units. The authors of that study reported a higher seroprevalence among HCWs in the COVID unit. However, this difference seems to be confounded by other factors as revealed by the multivariable analysis result.²³

We found a two-fold higher seroprevalence (4.4%) in HCWs who reported close contact with a patient with COVID-19. Respiratory infections pose a greater health risk to HCWs in an occupational setting. Substantial evidence has emerged demonstrating that the respiratory system is the dominant route of SARS-CoV-2 transmission, with proximity and ventilation as key predictive factors.²⁸ Globally, among thousands of HCWs infected with SARS-CoV-2, one of the leading risk factors identified was close contact with a patient with COVID-19; other identified risk factors were lack of PPE, poor infection prevention and control practices, work overload, and a preexisting health condition.²⁹

The seroprevalence estimate among participants who reported an ILI in the 4 weeks preceding the interview was only 12.2%, suggesting an alternative etiology of these symptoms. Among those who reported a previously positive RT-PCR for SARS-CoV-2, only 27.6% showed the presence of SARS-CoV-2–specific IgG antibodies. The inability to mount an antibody-mediated immune response or early conversion to seronegative status during the convalescence phase has been suggested as an explanation for such discordant findings.³⁰ On the contrary, seropositivity among participants who reported having a negative RT-PCR test was 1.9%. There are few plausible explanations for such observations. First, several studies have reported false-negative result rates from RT-PCR testing ranging from 2% to 29%.^31-33 Second, the sensitivity of the SARS-CoV-2 assay is influenced by the timing of the test after the onset of symptoms or RT-PCR positivity. The sensitivity of the assay we used varies from 53.1% at day 7 to 100% at day 17 postinfection.³⁴ Variable viral load and differences in duration of viral shedding are other possible reasons for false-negative RT-PCR results.^35,36

In our study, seroconversion among asymptomatic HCWs who were RT-PCR-positive was 20.8%. Among HCWs who reported an ILI and were RT-PCR-positive, seropositivity was 60%. In one study, 40% of asymptomatic and 13% of symptomatic patients who tested positive for COVID-19 became seronegative after initial seropositivity—that is, 8 weeks after hospital discharge.³⁷

Serological testing offers insight into both the exposure history and residual COVID-19 susceptibility of HCWs. However, current immunological knowledge does not allow us to conclude that seropositivity conveys high-level immunity against reinfection. As the epidemic evolves, HCWs will continue to be exposed to COVID-19 in the community and the workplace. Serial cross-sectional serosurveys can help monitor the progression of the pandemic within the healthcare setting and guide hospital authorities in resource allocation.

Strengths and Limitations

We used the Abbott Architect SARS-CoV-2 IgG assay, which has exhibited a high level of consistency and performance characteristics when tested in different patient populations. The participation rate was acceptable compared to similar studies, and we included all the major hospitals in the District Srinagar. The findings from our study can therefore be considered representative of the HCWs in the district.

The study results should be interpreted in the context of the following limitations. First, information on risk factors for seropositivity were based on participant report. Also, we did not collect information on the timing of symptoms or the date on which a participant became RT-PCR-positive. Second, information regarding place of exposure (ie, community or hospital setting) was not recorded, limiting conclusions regarding the effect of workplace exposures. Third, given the voluntary nature of participation in the study, there is a possibility of selection bias that may have limited the generalizability of our findings. For example, some HCWs with a recent exposure to COVID-19 or those who were symptomatic at the time of the study might not have participated based on the absence of an individual benefit from IgG testing in the early phase of infection. Conversely, some HCWs who had symptoms in the distant past might have been more likely to have participated in the study. However, we believe that selection bias does not vitiate the validity of the associations based on the plausible assumption that infection risk should be similar between respondents and nonrespondents due to comparable work environments. Finally, with a cross-sectional study design, we cannot ascertain the reconversion from an initial positive-IgG to negative-IgG status, which warrants a cohort study.

We conclude that the seroprevalence of SARS-CoV-2 infection was low among HCWs of District Srinagar at the time of the study. Healthcare workers in a dedicated COVID hospital or HCWs who had ever worked in such a facility had lower seroprevalence, suggesting both adherence to and effectiveness of standard protective measures during contact with patients who had COVID-19. Nonetheless, the careful in-hospital behavior of the HCWs at the COVID hospitals may have had a spillover effect on their out-of-hospital behaviors, which lead to community-acquired infection. On the contrary, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. We believe that our findings highlight the value of implementing infection prevention and control measures in the hospital setting. Moreover, training and retraining of sanitation and other housekeeping staff on standard hygienic practices and appropriate use of the protective gear may further help reduce their rates of exposure.

The authors thank Principal and Dean of the Government Medical College, Srinagar, Professor Samia Rashid, and District Commissioner, Srinagar, Shahid Iqbal Chowdhary for their support. We also acknowledge the support rendered by the Directorate of Health Services, Kashmir; Chief Medical Officer Srinagar; Block Medical Officers; and Zonal Medical Officers of District Srinagar, Kashmir, and extend our appreciation to the medical interns for their efforts in data collection, and to laboratory in-charge Gulzar Ahmad Wani, PhD scholar, Biochemistry, and his staff, who were involved in this study. Finally, we thank the study participants for their understanding of the importance of this study and for their time and participation.

Data shall be made available on request through the corresponding author.

India is emerging as one of the world’s largest hotspots for SARS-CoV-2 infection (COVID-19)—second only to the United States—with more than 13,000,000 documented infections since the first case was recorded on January 30, 2020.^1,2 Kashmir, a northern territory of India, reported its first case of COVID-19 on March 18, 2020, from the central District Srinagar; this region has accounted for more cases of COVID-19 than any other district throughout the pandemic.³ The large majority of healthcare in District Srinagar is provided by three tertiary care institutions, one district hospital, two subdistrict hospitals, and 70 primary healthcare centers. Potential occupational exposures place healthcare workers (HCWs) at higher risk of acquiring SARS-CoV-2 infection, which in turn may serve as an important source of infection for their families and other community members.^4-6 Given the high frequency and geographic variability of asymptomatic infection, growing evidence suggests this hidden reservoir is a source of infection for the general population.^7,8

Many countries have started testing for antibodies against SARS-CoV-2, both at the population level and in specific groups, such as HCWs. Seroepidemiological studies are crucial to understanding the dynamics of SARS-CoV-2 infection. Many seroepidemiological studies have been conducted among community populations, but there are insufficient data on HCWs. The World Health Organization also encouraged its member states to conduct seroepidemiological studies to attain a better understanding of COVID-19 infection prevalence and distribution.^9-11 Therefore, to quantify the prevalence of SARS-CoV-2 infection among HCWs, we conducted a seroepidemiological study by testing for SARS-CoV-2–specific immunoglobulin (IgG) to gain insight into the extent of infection among specific subgroups of HCWs and to identify risk-factor profiles associated with seropositivity.

Study Design and Settings

We conducted this seroepidemiological study to ascertain the presence of IgG antibodies against SARS-CoV-2 among HCWs in the District Srinagar of Kashmir, India. The 2-week period of data collection began on June 15, 2020. As part of healthcare system pandemic preparedness efforts, India’s Ministry of Health provided specific guidelines for health facilities to manage COVID-19. Hospitals were categorized as dedicated COVID and non-COVID hospitals. Dedicated COVID hospitals provided comprehensive care exclusively to patients with COVID-19 and were equipped with fully functional intensive care units, ventilators, and beds with reliable access to oxygen support.¹² In addition, infection prevention and control strategies to limit the transmission of SARS-CoV-2 infection were implemented according to guidelines specified by India’s National Center for Disease Control.¹³ To strengthen service provision, HCWs from other hospitals, including resident physicians, were relocated to these dedicated COVID hospitals. The additional staff were selected by administrative leadership, without input from HCWs.

Study Population and Data Collection

We approached administrative heads of the hospitals in District Srinagar for permission to conduct our study and to invite their HCWs to participate in the study. As Figure 1 shows, we were denied permission by the administrative heads of two tertiary care hospitals. Finally, with a point person serving as a study liaison at each institution, HCWs from three dedicated COVID and seven non-COVID tertiary care hospitals, two subdistrict hospitals, and six primary healthcare centers across the District Srinagar were invited to participate. The sample primary healthcare centers were each selected randomly, after stratification, from six major regions of the district. All frontline HCWs, including physicians, administrative and laboratory personnel, technicians, field workers involved in surveillance activity, and other supporting staff were eligible for the study.

We collected information on an interview form using Epicollect5, a free data-gathering tool widely used in health research.¹⁴ Physicians specifically trained in the use of Epicollect5 conducted the face-to-face interview on a prespecified day and recorded the collected information through mobile phones. This information included the participants’ role in providing care to patients with COVID-19 and risk factors for SARS-CoV-2 infection (eg, history of travel since January 1, 2020, symptoms of an influenza-like illness [ILI] in the 4 weeks prior to the interview, close contact with a COVID-19 case). We defined close contact as an unmasked exposure within 6 feet of an infected individual for at least 15 minutes, irrespective of location (ie, community or the hospital).

Following the interview, trained phlebotomists collected 3 to 5 mL of venous blood under aseptic conditions. We strictly adhered to standard operating procedures during collection, transportation, and testing of blood samples. Following collection, the blood samples remained undisturbed for at least 30 minutes before centrifugation, which was performed at the collection site (or at the central laboratory for sites lacking the capability). The samples were then transported for further processing and testing through a cold chain supply line, using vaccine carriers with conditioned icepacks. All testing procedures were conducted with strict adherence to the manufacturers’ guidelines.

Laboratory Procedure

In accordance with the manufacturer’s recommendations, we used a chemiluminescent microparticle immunoassay to detect SARS-CoV-2–specific IgG antibodies in serum samples. The assay is an automated two-step immunoassay for the qualitative detection of IgG antibodies against the nucleocapsid of SARS-CoV-2 in human serum and plasma. The sensitivity and specificity of this test are 100% and 99%, respectively. The test result was considered positive for SARS-CoV-2 IgG if the index value was ≥1.4, consistent with guidance provided by the manufacturer.¹⁵

The IgG values were also entered into Epicollect5. Two trained medical interns independently entered the laboratory results in two separate forms. A third medical intern reviewed these forms for discrepancies, in response to which they referenced the source data for adjudication. The information gathered during the interview and the laboratory results were linked with the help of a unique identification number, which was generated at the time of the interview.

Statistical Analysis

We estimated the proportion (and logit-transformed 95% CI) of HCWs with a positive SARS-CoV-2–specific IgG antibody level, the primary outcome of interest. We compared seroprevalence rates by gender, age group, specific occupational group, and type of health facility (dedicated COVID hospital vs non-COVID hospital). Seroprevalence was also estimated separately for HCWs who reported symptoms in the past 4 weeks, had a history of exposure to a known case of COVID-19, or had undergone testing by reverse transcriptase-polymerase chain reaction (RT-PCR). In the case of zero seroprevalences, Jeffreys 95% CIs were reported. We used a chi-square test to report two-sided P values for comparison of seroprevalence between groups. When the expected frequency was <5 in more than 20% of the cells, the exact test was used instead of the chi-square test. We additionally performed multivariable logistic regression analysis to evaluate the independent association between place of work (primary independent variable) and seropositivity (dependent variable). We adjusted for the following observable covariates by including them as categorical variables: age, gender, occupational group, and history of close contact with a patient who was COVID-positive. We performed data analysis using Stata, version 15.1 (StataCorp LP). The Institutional Ethics Committee of Government Medical College, Srinagar, approved the study (Reference No. 1003/ETH/GMC dated 13-05-2020). We obtained written, informed consent from all participants.

Of the 7,346 HCWs we were granted permission to approach, 2,915 (39.7%) agreed to participate in the study. The participation rate was 49% at the dedicated COVID hospitals (57% physicians and 47% nonphysicians) and 39% at the non-COVID hospitals (46% physicians and 36% nonphysicians). We analyzed information gathered from 2,905 HCWs (Epicollect5 interview forms were missing for nine participants, and the laboratory report was missing for one participant).

The mean age of the participants was 38.6 years, and 35.8% of participants identified as female (Table 1). One third (33.7%) of the participants were physicians, nearly half of whom were residents. In our sample, the overall seroprevalence of SARS-CoV-2–specific antibodies was 2.5% (95% CI, 2.0%-3.1%).

Of the 2,905 participating HCWs, 123 (4.2%) reported an ILI (ie, fever and cough) in the 4 weeks preceding the interview, and 339 (11.7%) reported close contact with a person with COVID-19 (Table 2). A total of 760 (26.2%) HCWs had undergone RT-PCR testing, 29 (3.8%) of whom had a positive result. Stratifying by workplace, history of nasopharyngeal RT-PCR positivity was reported by 4 of 77 (5.1%) participants from dedicated COVID hospitals compared to (3.7%) participants from the non-COVID hospital (P = .528).

As Table 2 also demonstrates, we found a significantly higher seropositivity rate among HCWs who had a history of ILI (P < .001), a history of positive RT-PCR (P < .001), history of ever being put under quarantine (P = .009), and a self-reported history of close contact with a person with COVID-19 (P = .014). Healthcare workers who had ever worked at a dedicated COVID hospital had a significantly lower seroprevalence of infection (P = .004).

Among HCWs who reported no ILI symptoms in the 4 weeks prior to the interview but who had positive RT-PCR test, 20.8% were seropositive. Of HCWs who reported both ILI and a positive RT-PCR test result, 60.0% were seropositive. Compared to employment at a non-COVID hospital, HCWs working in dedicated COVID hospitals had a reduced multivariate-adjusted risk of seropositivity (odds ratio, 0.21; 95% CI, 0.06-0.66).

We aimed to estimate the seroprevalence of SARS-CoV-2 infection in HCWs in different hospital settings in the District Srinagar of Kashmir, India. In general, seroprevalence was low (2.5%), with little difference across gender or occupational group.

Seroprevalence studies of HCWs across divergent workplace environments have revealed estimates ranging from 1% to 10.2%.^16-19 Generally, the seroprevalence rates among HCWs are not significantly different from those of the general population, which reflects how different the dynamics of COVID-19 are compared to other infections in healthcare settings. The low seroprevalence observed in our study coincides with the overall low infection rate in the community population. During the study period, District Srinagar reported a median of 28 new infections daily (interquartile range, 17-46), which is indicative of the early phase of the pandemic in the population at the time of the study.²⁰

Among the HCW occupational groups, ambulance drivers and housekeeping staff had the highest seroprevalence rates, followed by nurses and physicians. Possible explanations for higher seropositivity in these groups are improper use or inadequate supply of protective gear and lack of training on the use of personal protective equipment (PPE), resulting in increased exposure risk.²¹ Concordance of HCW and community infection rates in specific geographic areas suggests that community exposure may be the dominant source of healthcare exposure and infection. Additionally, careful in-hospital behavior of HCWs in dedicated COVID hospitals may have had a spillover effect on their out-of-hospital behavior, which may partially explain our finding that employment at dedicated COVID hospitals was associated with a markedly lower chance of seropositivity. A study of 6,510 HCWs in Chicago, Illinois, showed high seropositivity rates among support service workers, medical assistants, and nurses, with nurses identified as having a markedly higher adjusted odds of seropositivity relative to administrators. The authors of the study concluded that exposure in the community setting plays a crucial role in transmission among HCWs.²² Similarly, higher seroprevalence among housekeeping, nonadministrative staff, and other support service staff has been reported elsewhere.²³ Certain underlying factors related to socioeconomic status and lifestyle may also contribute to higher seroprevalence in some occupational groups.²⁴ Nonadherence to masking, social distancing, and proper hand hygiene outside the hospital setting could result in community-acquired infection.

Interestingly, participants who were working in a dedicated COVID hospital or who had ever worked at one had a seroprevalence of 0.6%, much lower than the 2.8% observed among other participants. This difference remained statistically significant after controlling for age, sex, place of work, and occupational group. As these facilities were dedicated to the management and care of patients with COVID-19, the hospital staff strictly adhered to safety precautions, with particular vigilance during patient contact. These hospitals also strictly adhered to infection prevention and control practices based on the latest guidelines released by India’s Ministry of Health and Family Welfare.¹³

A commitment was made to provide adequate PPE to the dedicated COVID hospitals and staff, commensurate with expected infected patient volumes and associated exposure risks. Healthcare workers were specifically trained on proper donning and doffing of PPE, self-health monitoring, and protocols for reporting symptoms and PPE breaches during patient encounters. Healthcare workers were regularly tested for COVID-19 using nasopharyngeal RT-PCR. Of critical importance, these hospitals implemented a buddy system wherein a team of two or more staff members was responsible for ensuring each other’s safety, proper PPE use, conformance to other protective measures, and reporting breaches of PPE compliance.²⁵ Universal masking was mandatory for all hospital staff and patients at the COVID-focused facilities, with the additional use of N-95 masks, gloves, and face shields during times of patient contact. Administrative measures, including visitor restrictions and environmental sanitation, were rigorously enforced. Also, being a potentially high-risk area for transmission of infection, these facilities implemented staff-rationing to reduce the duration of exposure to the healthcare staff. Third, the HCWs of COVID-dedicated hospitals were provided with separate living accommodations during the period in which they were employed at a dedicated COVID hospital.

In contrast, in non-COVID hospitals, with the exception of HCWs, patients and the hospital visitors were not subject to a masking policy. Moreover, an adequate and timely supply of PPE was not prioritized at the non-COVID facilities due to resource constraints. Further, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. Though routine infection prevention and control activities were performed at non-COVID hospitals, we did not assess adherence to infection prevention and control guidelines in the two different categories of hospitals. Our results are also supported by evidence from studies conducted in different hospital settings, the findings of which reiterate the importance of fundamental principles of prevention (eg, proper masking, hand hygiene, and distancing) and are of particular importance in resource-limited settings.^17,26,27 The only published study quantifying seroprevalence among HCWs in India was performed in a single hospital setting with separate COVID and non-COVID units. The authors of that study reported a higher seroprevalence among HCWs in the COVID unit. However, this difference seems to be confounded by other factors as revealed by the multivariable analysis result.²³

We found a two-fold higher seroprevalence (4.4%) in HCWs who reported close contact with a patient with COVID-19. Respiratory infections pose a greater health risk to HCWs in an occupational setting. Substantial evidence has emerged demonstrating that the respiratory system is the dominant route of SARS-CoV-2 transmission, with proximity and ventilation as key predictive factors.²⁸ Globally, among thousands of HCWs infected with SARS-CoV-2, one of the leading risk factors identified was close contact with a patient with COVID-19; other identified risk factors were lack of PPE, poor infection prevention and control practices, work overload, and a preexisting health condition.²⁹

The seroprevalence estimate among participants who reported an ILI in the 4 weeks preceding the interview was only 12.2%, suggesting an alternative etiology of these symptoms. Among those who reported a previously positive RT-PCR for SARS-CoV-2, only 27.6% showed the presence of SARS-CoV-2–specific IgG antibodies. The inability to mount an antibody-mediated immune response or early conversion to seronegative status during the convalescence phase has been suggested as an explanation for such discordant findings.³⁰ On the contrary, seropositivity among participants who reported having a negative RT-PCR test was 1.9%. There are few plausible explanations for such observations. First, several studies have reported false-negative result rates from RT-PCR testing ranging from 2% to 29%.^31-33 Second, the sensitivity of the SARS-CoV-2 assay is influenced by the timing of the test after the onset of symptoms or RT-PCR positivity. The sensitivity of the assay we used varies from 53.1% at day 7 to 100% at day 17 postinfection.³⁴ Variable viral load and differences in duration of viral shedding are other possible reasons for false-negative RT-PCR results.^35,36

In our study, seroconversion among asymptomatic HCWs who were RT-PCR-positive was 20.8%. Among HCWs who reported an ILI and were RT-PCR-positive, seropositivity was 60%. In one study, 40% of asymptomatic and 13% of symptomatic patients who tested positive for COVID-19 became seronegative after initial seropositivity—that is, 8 weeks after hospital discharge.³⁷

Serological testing offers insight into both the exposure history and residual COVID-19 susceptibility of HCWs. However, current immunological knowledge does not allow us to conclude that seropositivity conveys high-level immunity against reinfection. As the epidemic evolves, HCWs will continue to be exposed to COVID-19 in the community and the workplace. Serial cross-sectional serosurveys can help monitor the progression of the pandemic within the healthcare setting and guide hospital authorities in resource allocation.

Strengths and Limitations

We used the Abbott Architect SARS-CoV-2 IgG assay, which has exhibited a high level of consistency and performance characteristics when tested in different patient populations. The participation rate was acceptable compared to similar studies, and we included all the major hospitals in the District Srinagar. The findings from our study can therefore be considered representative of the HCWs in the district.

The study results should be interpreted in the context of the following limitations. First, information on risk factors for seropositivity were based on participant report. Also, we did not collect information on the timing of symptoms or the date on which a participant became RT-PCR-positive. Second, information regarding place of exposure (ie, community or hospital setting) was not recorded, limiting conclusions regarding the effect of workplace exposures. Third, given the voluntary nature of participation in the study, there is a possibility of selection bias that may have limited the generalizability of our findings. For example, some HCWs with a recent exposure to COVID-19 or those who were symptomatic at the time of the study might not have participated based on the absence of an individual benefit from IgG testing in the early phase of infection. Conversely, some HCWs who had symptoms in the distant past might have been more likely to have participated in the study. However, we believe that selection bias does not vitiate the validity of the associations based on the plausible assumption that infection risk should be similar between respondents and nonrespondents due to comparable work environments. Finally, with a cross-sectional study design, we cannot ascertain the reconversion from an initial positive-IgG to negative-IgG status, which warrants a cohort study.

We conclude that the seroprevalence of SARS-CoV-2 infection was low among HCWs of District Srinagar at the time of the study. Healthcare workers in a dedicated COVID hospital or HCWs who had ever worked in such a facility had lower seroprevalence, suggesting both adherence to and effectiveness of standard protective measures during contact with patients who had COVID-19. Nonetheless, the careful in-hospital behavior of the HCWs at the COVID hospitals may have had a spillover effect on their out-of-hospital behaviors, which lead to community-acquired infection. On the contrary, lack of testing of asymptomatic patients at non-COVID hospitals may have resulted in nosocomial transmission from asymptomatic carriers. We believe that our findings highlight the value of implementing infection prevention and control measures in the hospital setting. Moreover, training and retraining of sanitation and other housekeeping staff on standard hygienic practices and appropriate use of the protective gear may further help reduce their rates of exposure.

The authors thank Principal and Dean of the Government Medical College, Srinagar, Professor Samia Rashid, and District Commissioner, Srinagar, Shahid Iqbal Chowdhary for their support. We also acknowledge the support rendered by the Directorate of Health Services, Kashmir; Chief Medical Officer Srinagar; Block Medical Officers; and Zonal Medical Officers of District Srinagar, Kashmir, and extend our appreciation to the medical interns for their efforts in data collection, and to laboratory in-charge Gulzar Ahmad Wani, PhD scholar, Biochemistry, and his staff, who were involved in this study. Finally, we thank the study participants for their understanding of the importance of this study and for their time and participation.

Data shall be made available on request through the corresponding author.

Febrile infants aged 0 to 60 days often undergo diagnostic testing to evaluate for invasive bacterial infections (IBI; ie, bacteremia and meningitis) and are subsequently hospitalized pending culture results. Only 1% to 2% of infants 0 to 60 days old have an IBI,^1-3 and most hospitalized infants are discharged once physicians feel confident that pathogens are unlikely to be isolated from blood and cerebrospinal fluid (CSF) cultures. Practice regarding duration of hospitalization while awaiting blood and CSF culture results is not standardized in this population. Longer hospitalizations can lead to increased costs and familial stress, including difficulty with breastfeeding and anxiety in newly postpartum mothers.^4,5

In 2010, an institutional evidence-based guideline for the management of febrile infants aged 0 to 60 days recommended discharge after 36 hours of observation if all cultures were negative.⁶ However, recent studies demonstrate that 85% to 93% of pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 Assuming a 2% prevalence of IBI, if 15% of pathogens were identified after 24 hours of incubation, only one out of 333 infants would have an IBI identified after 24 hours of hospital observation.⁷

Furthermore, a review of our institution’s electronic health records (EHR) over the past 5 years revealed that an observation period of 24 hours would have resulted in the discharge of three infants with an IBI. Two infants had bacteremia; both were discharged from the emergency department (ED) without antibiotics, returned to care after cultures were reported positive at 27 hours, and had no adverse outcomes. The third infant had meningitis, but also had an abnormal CSF Gram stain, which led to a longer hospitalization.

In 2019, our institution appraised the emerging literature and institutional data supporting the low absolute risk of missed IBI, and also leveraged local consensus among key stakeholders to update its evidence-based guideline for the evaluation and management of febrile infants aged 60 days and younger. The updated guideline recommends that clinicians consider discharging well-appearing neonates and infants if blood and CSF cultures remain negative at 24 hours.¹⁰ The objective of this study was to decrease the average hospital culture observation time (COT; culture incubation to hospital discharge) from 38 to 30 hours over a 12-month period in febrile infants aged 0 to 60 days.

Context

Improvement efforts were conducted at Cincinnati Children’s Hospital Medical Center (CCHMC), a large, urban, academic hospital that admitted more than 8,000 noncritically ill patients to the hospital medicine (HM) service from July 1, 2018, through June 30, 2019. Hospital medicine teams, located at both the main and satellite campuses, are staffed by attending physicians, fellows, residents, medical students, and nurse practitioners. The two campuses, which are about 20 miles apart, share clinician providers but have distinct nursing pools.

Microbiology services for all CCHMC patients are provided at the main campus. Blood and CSF cultures at the satellite campus are transported to the main campus for incubation and monitoring via an urgent courier service. The microbiology laboratory at CCHMC uses a continuous monitoring system for blood cultures (BACT/ALERT Virtuo, BioMérieux). The system automatically alerts laboratory technicians of positive cultures; these results are reported to clinical providers within 30 minutes of detection. Laboratory technicians manually evaluate CSF cultures once daily for 5 days.

Improvement Team

Our improvement team included three HM attending physicians; two HM fellows; a pediatric chief resident; two nurses, who represented nursing pools at the main and satellite campuses; and a clinical pharmacist, who is a co-leader of the antimicrobial stewardship program at CCHMC. Supporting members for the improvement team included the CCHMC laboratory director; the microbiology laboratory director; an infectious disease physician, who is a co-leader of the antimicrobial stewardship program; and nursing directors of the HM units at both campuses.

Evidence-Based Guideline

Our improvement initiative was based on recommendations from the updated CCHMC Evidence-Based Care Guideline for Management of Infants 0 to 60 days with Fever of Unknown Source.¹⁰ This guideline, published in May 2019, was developed by a multidisciplinary working group composed of key stakeholders from HM, community pediatrics, emergency medicine, the pediatric residency program, infectious disease, and laboratory medicine. Several improvement team members were participants on the committee that published the evidence-based guideline. The committee first performed a systematic literature review and critical appraisal of the literature. Care recommendations were formulated via a consensus process directed by best evidence, patient and family preferences, and clinical expertise; the recommendations were subsequently reviewed and approved by clinical experts who were not involved in the development process.

Based on evidence review and multistakeholder consensus, the updated guideline recommends clinicians consider discharging neonates and infants aged 60 days and younger if there is no culture growth after an observation period of 24 hours (as documented in the EHR) and patients are otherwise medically ready for discharge (ie, well appearing with adequate oral intake).^10,11 In addition, prior to discharge, there must be a documented working phone number on file for the patient’s parents/guardians, an established outpatient follow-up plan within 24 hours, and communication with the primary pediatrician who is in agreement with discharge at 24 hours.

Study Population

Infants 0 to 60 days old who had a documented or reported fever without an apparent source based on history and physical exam upon presentation to the ED, and who were subsequently admitted to the HM service at CCHMC between October 30, 2018, and July 10, 2020, were eligible for inclusion. We excluded infants who were admitted to other clinical services (eg, intensive care unit); had organisms identified on blood, urine, or CSF culture within 24 hours of incubation; had positive herpes simplex virus testing; had skin/soft tissue infections or another clearly documented source of bacterial infection; or had an alternative indication for hospitalization (eg, need for intravenous fluid or deep suctioning) after cultures had incubated for 24 hours. Infants who had a positive blood, urine, or CSF culture result after 24 hours of incubation were included in the study population. Organisms were classified as pathogen or contaminant based on treatment decisions made by the care team.

Improvement Activities

Key drivers critical to success of the improvement efforts were: (1) clearly defined standard of care for duration of observation in febrile infants 0 to 60 days old; (2) improved understanding of microbiology lab procedures; (3) effective communication of discharge criteria between providers and nurses; and (4) transparency of data with feedback (Figure 1).

Education and Structured Dissemination of Evidence-Based Guideline

The CCHMC febrile infant guideline¹⁰ was disseminated to HM physicians, residents, and nurses via the following means: (1) in-person announcements at staff meetings and educational conferences, (2) published highlights from the guideline in weekly newsletters, and (3) email announcements. Additionally, members of the study team educated HM attending physicians, nursing staff from the medical units at both campuses, and resident physicians about recent studies demonstrating safety of shorter length of stay (LOS) in febrile infants aged 0 to 60 days. The study team also provided residents, physicians, and nurses with data on the number of positive blood and CSF cultures and outcomes of patients at CCHMC within the past 5 years. In addition, team members led a journal club for residents discussing an article⁷ describing time-to-positivity of blood and CSF cultures in febrile infants. For ongoing engagement, the evidence-based guideline and a detailed explanation of microbiology procedures were published in the resident handbook, an internal resource that includes vital clinical pearls and practice guidelines across specialties. (Each resident receives an updated hard copy each year, and there is also an online link to the resource in the EHR.) Information about the guideline and COT was also included in the monthly chief resident’s orientation script, which is relayed to all residents on the first day of their HM rotation.

Clear Communication of Microbiology Procedures

Team members created a detailed process map describing the processing protocols for blood and CSF cultures collected at both CCHMC campuses. This information was shared with HM attending physicians and nurses via in-person announcements at staff meetings, flyers in team workrooms, and email communications. Residents received information on microbiology protocols via in-person announcements at educational conferences and dissemination in the weekly residency newsletter.Important information communicated included:

1. Definition of culture start time. We conveyed that there may be a delay of up to 4 hours between culture collection at the satellite campus and culture incubation at the main campus laboratory. As a result, the time of blood or CSF sample arrival to the main campus laboratory was a more accurate reflection of the culture incubation start time than the culture collection time.

2. Explanation of CSF culture processing. We discussed the process by which these cultures are plated upon arrival at the microbiology laboratory and read once per day in the morning. Therefore, a culture incubated at midnight would be evaluated once at 9 hours and not again until 33 hours.

Modification of Febrile Infant Order Set

Enhancements to the febrile infant order set improved communication and cultivated a shared mental model regarding discharge goals among all members of the care team. The EHR order set for febrile infants was updated as follows: (1) mandatory free-text fields that established the culture start time for blood and CSF cultures were added, (2) culture start time was clearly defined (ie, the time culture arrives at the main campus laboratory), and (3) a change was made in the default discharge criteria¹¹ to “culture observation for 24 hours,” with the ability to modify COT (Appendix Figure 1). We embedded hyperlinks to the guideline and microbiology process map within the updated order set, which allowed providers to easily access this information and refresh their knowledge of the recommendations (Appendix Figure 1).

Identification of Failures and Follow-up With Near-Time Feedback

All cases of febrile infants were tracked weekly. For infants hospitalized longer than 24 hours, the study team contacted the discharging clinicians to discuss reasons for prolonged hospitalization, with an emphasis on identifying system-level barriers to earlier discharge.

Study of the Interventions

The institutional microbiology database was queried weekly to identify all infants 0 to 60 days old who had a blood culture obtained and were hospitalized on the HM service. Study team members conducted targeted EHR review to determine whether patients met exclusion criteria and to identify reasons for prolonged COT. Baseline data were collected retrospectively for a 3-month period prior to initiation of improvement activities. During the study period, queries were conducted weekly and reviewed by study team members to evaluate the impact of improvement activities and to inform new interventions.

Measures

Our primary outcome measure was COT, defined as the hours between final culture incubation and hospital discharge. The operational definition for “final culture incubation” was the documented time of arrival of the last collected culture to the microbiology laboratory. Our goal COT was 30 hours to account for a subset of patients whose blood and/or CSF culture were obtained overnight (ie, after 9 pm), since subsequent discharge times would likely and practically be delayed beyond 24 hours. Our secondary outcome measure was LOS, defined as the time between ED arrival and hospital discharge. Process measures included the proportion of patients for whom the febrile infant EHR order set was used and the proportion of patients for whom medical discharge criteria (ie, blood and CSF culture observed for ”xx” hours) and culture incubation start times were entered using the order set. Balancing measures included identification of IBI after hospital discharge, 48-hour ED revisits, and 7-day hospital readmissions.

Analysis

Measures were evaluated using statistical process control charts and run charts, and Western Electric rules were employed to determine special cause variation.¹² Annotated X-bar S control charts tracked the impact of improvement activities on average COT and LOS for all infants. Given that a relatively small number of patients (ie, two to four) met inclusion criteria each week, average COT was calculated per five patients.

This study was considered exempt from review by the CCHMC Institutional Review Board.

Of the 184 infants in this study, 46 were included as part of baseline data collection, and 138 were included during the intervention period. The median age was 26.6 days (range, 3-59 days); 52% of patients were female; two-thirds were non-Hispanic White; 22% were Black, and 5% were Hispanic (Appendix Table).

Average COT decreased from 38 hours to 32 hours with improvement activities (Figure 2) and was sustained for a total of 17 months. There were small decreases in COT after initial education was provided to attendings, nurses, and residents.

After the launch of the updated order set, median usage of the EHR order set increased from 50% to 80%. Medical discharge criteria were entered for 80 (96%) of the 83 patients for whom the updated order set was applied; culture incubation start times were entered for 78 (94%) of these patients.

No infants in our cohort were found to have IBI after hospital discharge. There were no ED revisits within 48 hours of discharge, and there were no hospital readmissions within 7 days of index discharge. Furthermore, none of the patients included in the study had growth of a pathogenic organism after 24 hours.

Of the 138 infants hospitalized during the intervention period, 77 (56%) had a COT greater than 30 hours. Among these 77 patients, 49 (64%) had their final culture incubated between 9 pm and 4 am; Furthermore, 11 (14%) had missing, abnormal, pretreated, or uninterpretable CSF studies, 7 (9%) had ongoing fevers, and 4 (5%) remained hospitalized due to family preference or inability to obtain timely outpatient follow-up.

Our study aimed to decrease the average COT from 38 hours to 30 hours among hospitalized infants aged 60 days and younger over a period of 12 months. An intervention featuring implementation of an evidence-based guideline through education, laboratory procedure transparency, creation of a standardized EHR order set, and near-time feedback was associated with a shorter average COT of 32 hours, sustained over a 17-month period. No infants with bacteremia or meningitis were inappropriately discharged during this study.

Interpretation

Prior to our improvement efforts, most febrile infants at CCHMC were observed for at least 36 hours based on a prior institutional guideline,⁶ despite recent evidence suggesting that most pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 The goal of this improvement initiative was to bridge the gap between emerging evidence and clinical practice by developing and disseminating an updated evidence-based guideline to safely decrease the hospital observation time in febrile infants aged 60 days and younger.

Similar to previous studies aimed at improving diagnosis and management among febrile infants,^13-16 generation and structured dissemination of an institutional evidence-based guideline was crucial to safely shortening COT in our population. These prior studies established a goal COT of 36 to 42 hours for hospitalized febrile infants.^13,15,16 Our study incorporated emerging evidence and local experience into an updated evidence-based practice guideline to further reduce COT to 32 hours for hospitalized infants. Key factors contributing to our success included multidisciplinary engagement, specifically partnering with nurses and resident physicians in designing and implementing our initiatives. Furthermore, improved transparency of culture monitoring practices allowed clinicians to better understand the recommended observation periods. Finally, we employed a standardized EHR order set as a no-cost, one-time, high-reliability intervention to establish 24 hours of culture monitoring as the default and to enhance transparency around start time for culture incubation.

Average COT remained stable at 32 hours for 17 months after initiation of the intervention. During the intervention period, 64% patients with hospital stays longer than 30 hours had cultures obtained between 9 pm to 4 am. These patients often remained hospitalized for longer than 30 hours to allow for a daytime hospital discharge. Additionally, CSF cultures were only monitored manually once per day between 8 am and 10 am. As a result, CSF cultures obtained in the evening (eg, 9 pm) would be evaluated once at roughly 12 hours of incubation, and then the following morning at 36 hours of incubation. In cases where CSF studies (eg, cell count, protein, Gram stain) were abnormal, uninterpretable, or could not be obtained, clinicians monitored CSF cultures closer to 36 hours from incubation. While evidence-based guidelines and local data support safe early discharge of febrile infants, clinicians presented with incomplete or uninterpretable data were appropriately more likely to observe infants for longer periods to confirm negative cultures.

Limitations

The study has several limitations. First, this single-center study was conducted at a quaternary care medical center with a robust quality improvement infrastructure. Our interventions took advantage of the existing processes in place that ensure timely discharge of medically ready patients.¹¹ Furthermore, microbiology laboratory practices are unique to our institution. These factors limit the generalizability of this work. Second, due to small numbers of eligible infants, analyses were conducted per five patients. Infrequent hospitalizations limited our ability to learn quickly from PDSA cycles. Finally, we did not measure cost savings attributable to shorter hospital stays. However, in addition to financial savings from charges and decreased nonmedical costs such as lost earnings and childcare,¹⁷ shorter hospitalizations have many additional benefits, such as promoting bonding and breastfeeding and decreasing exposure to nosocomial infections. Shorter hospitalizations, with clearly communicated discharge times, also serve to optimize patient throughput.

Implementation of a clinical practice guideline resulted in reduction of average COT from 38 to 32 hours in febrile infants aged 60 days and younger, with no cases of missed IBI. Engagement of multidisciplinary stakeholders in the generation and structured dissemination of the evidence-based guideline, improved transparency of the microbiological blood and CSF culture process, and standardization of EHR order sets were crucial to the success of this work. Cultures incubated overnight and daily CSF culture-monitoring practices primarily contributed to an average LOS of more than 30 hours.

Future work will include collaboration with emergency physicians to improve evaluation efficiency and decrease LOS in the ED for febrile infants. Additionally, creation of an automated data dashboard of COT and LOS will provide clinicians with real-time feedback on hospitalization practices.

The authors thank Dr Jeffrey Simmons, MD, MSc, as well as the members of the 2019 Fever of Uncertain Source Evidence-Based Guideline Committee. We also thank the James M Anderson Center for Health System Excellence and the Rapid Cycle Improvement Collaborative for their support with guideline development as well as design and execution of our improvement efforts.

Febrile infants aged 0 to 60 days often undergo diagnostic testing to evaluate for invasive bacterial infections (IBI; ie, bacteremia and meningitis) and are subsequently hospitalized pending culture results. Only 1% to 2% of infants 0 to 60 days old have an IBI,^1-3 and most hospitalized infants are discharged once physicians feel confident that pathogens are unlikely to be isolated from blood and cerebrospinal fluid (CSF) cultures. Practice regarding duration of hospitalization while awaiting blood and CSF culture results is not standardized in this population. Longer hospitalizations can lead to increased costs and familial stress, including difficulty with breastfeeding and anxiety in newly postpartum mothers.^4,5

In 2010, an institutional evidence-based guideline for the management of febrile infants aged 0 to 60 days recommended discharge after 36 hours of observation if all cultures were negative.⁶ However, recent studies demonstrate that 85% to 93% of pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 Assuming a 2% prevalence of IBI, if 15% of pathogens were identified after 24 hours of incubation, only one out of 333 infants would have an IBI identified after 24 hours of hospital observation.⁷

Furthermore, a review of our institution’s electronic health records (EHR) over the past 5 years revealed that an observation period of 24 hours would have resulted in the discharge of three infants with an IBI. Two infants had bacteremia; both were discharged from the emergency department (ED) without antibiotics, returned to care after cultures were reported positive at 27 hours, and had no adverse outcomes. The third infant had meningitis, but also had an abnormal CSF Gram stain, which led to a longer hospitalization.

In 2019, our institution appraised the emerging literature and institutional data supporting the low absolute risk of missed IBI, and also leveraged local consensus among key stakeholders to update its evidence-based guideline for the evaluation and management of febrile infants aged 60 days and younger. The updated guideline recommends that clinicians consider discharging well-appearing neonates and infants if blood and CSF cultures remain negative at 24 hours.¹⁰ The objective of this study was to decrease the average hospital culture observation time (COT; culture incubation to hospital discharge) from 38 to 30 hours over a 12-month period in febrile infants aged 0 to 60 days.

Context

Improvement efforts were conducted at Cincinnati Children’s Hospital Medical Center (CCHMC), a large, urban, academic hospital that admitted more than 8,000 noncritically ill patients to the hospital medicine (HM) service from July 1, 2018, through June 30, 2019. Hospital medicine teams, located at both the main and satellite campuses, are staffed by attending physicians, fellows, residents, medical students, and nurse practitioners. The two campuses, which are about 20 miles apart, share clinician providers but have distinct nursing pools.

Microbiology services for all CCHMC patients are provided at the main campus. Blood and CSF cultures at the satellite campus are transported to the main campus for incubation and monitoring via an urgent courier service. The microbiology laboratory at CCHMC uses a continuous monitoring system for blood cultures (BACT/ALERT Virtuo, BioMérieux). The system automatically alerts laboratory technicians of positive cultures; these results are reported to clinical providers within 30 minutes of detection. Laboratory technicians manually evaluate CSF cultures once daily for 5 days.

Improvement Team

Our improvement team included three HM attending physicians; two HM fellows; a pediatric chief resident; two nurses, who represented nursing pools at the main and satellite campuses; and a clinical pharmacist, who is a co-leader of the antimicrobial stewardship program at CCHMC. Supporting members for the improvement team included the CCHMC laboratory director; the microbiology laboratory director; an infectious disease physician, who is a co-leader of the antimicrobial stewardship program; and nursing directors of the HM units at both campuses.

Evidence-Based Guideline

Our improvement initiative was based on recommendations from the updated CCHMC Evidence-Based Care Guideline for Management of Infants 0 to 60 days with Fever of Unknown Source.¹⁰ This guideline, published in May 2019, was developed by a multidisciplinary working group composed of key stakeholders from HM, community pediatrics, emergency medicine, the pediatric residency program, infectious disease, and laboratory medicine. Several improvement team members were participants on the committee that published the evidence-based guideline. The committee first performed a systematic literature review and critical appraisal of the literature. Care recommendations were formulated via a consensus process directed by best evidence, patient and family preferences, and clinical expertise; the recommendations were subsequently reviewed and approved by clinical experts who were not involved in the development process.

Based on evidence review and multistakeholder consensus, the updated guideline recommends clinicians consider discharging neonates and infants aged 60 days and younger if there is no culture growth after an observation period of 24 hours (as documented in the EHR) and patients are otherwise medically ready for discharge (ie, well appearing with adequate oral intake).^10,11 In addition, prior to discharge, there must be a documented working phone number on file for the patient’s parents/guardians, an established outpatient follow-up plan within 24 hours, and communication with the primary pediatrician who is in agreement with discharge at 24 hours.

Study Population

Infants 0 to 60 days old who had a documented or reported fever without an apparent source based on history and physical exam upon presentation to the ED, and who were subsequently admitted to the HM service at CCHMC between October 30, 2018, and July 10, 2020, were eligible for inclusion. We excluded infants who were admitted to other clinical services (eg, intensive care unit); had organisms identified on blood, urine, or CSF culture within 24 hours of incubation; had positive herpes simplex virus testing; had skin/soft tissue infections or another clearly documented source of bacterial infection; or had an alternative indication for hospitalization (eg, need for intravenous fluid or deep suctioning) after cultures had incubated for 24 hours. Infants who had a positive blood, urine, or CSF culture result after 24 hours of incubation were included in the study population. Organisms were classified as pathogen or contaminant based on treatment decisions made by the care team.

Improvement Activities

Key drivers critical to success of the improvement efforts were: (1) clearly defined standard of care for duration of observation in febrile infants 0 to 60 days old; (2) improved understanding of microbiology lab procedures; (3) effective communication of discharge criteria between providers and nurses; and (4) transparency of data with feedback (Figure 1).

Education and Structured Dissemination of Evidence-Based Guideline

The CCHMC febrile infant guideline¹⁰ was disseminated to HM physicians, residents, and nurses via the following means: (1) in-person announcements at staff meetings and educational conferences, (2) published highlights from the guideline in weekly newsletters, and (3) email announcements. Additionally, members of the study team educated HM attending physicians, nursing staff from the medical units at both campuses, and resident physicians about recent studies demonstrating safety of shorter length of stay (LOS) in febrile infants aged 0 to 60 days. The study team also provided residents, physicians, and nurses with data on the number of positive blood and CSF cultures and outcomes of patients at CCHMC within the past 5 years. In addition, team members led a journal club for residents discussing an article⁷ describing time-to-positivity of blood and CSF cultures in febrile infants. For ongoing engagement, the evidence-based guideline and a detailed explanation of microbiology procedures were published in the resident handbook, an internal resource that includes vital clinical pearls and practice guidelines across specialties. (Each resident receives an updated hard copy each year, and there is also an online link to the resource in the EHR.) Information about the guideline and COT was also included in the monthly chief resident’s orientation script, which is relayed to all residents on the first day of their HM rotation.

Clear Communication of Microbiology Procedures

Team members created a detailed process map describing the processing protocols for blood and CSF cultures collected at both CCHMC campuses. This information was shared with HM attending physicians and nurses via in-person announcements at staff meetings, flyers in team workrooms, and email communications. Residents received information on microbiology protocols via in-person announcements at educational conferences and dissemination in the weekly residency newsletter.Important information communicated included:

1. Definition of culture start time. We conveyed that there may be a delay of up to 4 hours between culture collection at the satellite campus and culture incubation at the main campus laboratory. As a result, the time of blood or CSF sample arrival to the main campus laboratory was a more accurate reflection of the culture incubation start time than the culture collection time.

2. Explanation of CSF culture processing. We discussed the process by which these cultures are plated upon arrival at the microbiology laboratory and read once per day in the morning. Therefore, a culture incubated at midnight would be evaluated once at 9 hours and not again until 33 hours.

Modification of Febrile Infant Order Set

Enhancements to the febrile infant order set improved communication and cultivated a shared mental model regarding discharge goals among all members of the care team. The EHR order set for febrile infants was updated as follows: (1) mandatory free-text fields that established the culture start time for blood and CSF cultures were added, (2) culture start time was clearly defined (ie, the time culture arrives at the main campus laboratory), and (3) a change was made in the default discharge criteria¹¹ to “culture observation for 24 hours,” with the ability to modify COT (Appendix Figure 1). We embedded hyperlinks to the guideline and microbiology process map within the updated order set, which allowed providers to easily access this information and refresh their knowledge of the recommendations (Appendix Figure 1).

Identification of Failures and Follow-up With Near-Time Feedback

All cases of febrile infants were tracked weekly. For infants hospitalized longer than 24 hours, the study team contacted the discharging clinicians to discuss reasons for prolonged hospitalization, with an emphasis on identifying system-level barriers to earlier discharge.

Study of the Interventions

The institutional microbiology database was queried weekly to identify all infants 0 to 60 days old who had a blood culture obtained and were hospitalized on the HM service. Study team members conducted targeted EHR review to determine whether patients met exclusion criteria and to identify reasons for prolonged COT. Baseline data were collected retrospectively for a 3-month period prior to initiation of improvement activities. During the study period, queries were conducted weekly and reviewed by study team members to evaluate the impact of improvement activities and to inform new interventions.

Measures

Our primary outcome measure was COT, defined as the hours between final culture incubation and hospital discharge. The operational definition for “final culture incubation” was the documented time of arrival of the last collected culture to the microbiology laboratory. Our goal COT was 30 hours to account for a subset of patients whose blood and/or CSF culture were obtained overnight (ie, after 9 pm), since subsequent discharge times would likely and practically be delayed beyond 24 hours. Our secondary outcome measure was LOS, defined as the time between ED arrival and hospital discharge. Process measures included the proportion of patients for whom the febrile infant EHR order set was used and the proportion of patients for whom medical discharge criteria (ie, blood and CSF culture observed for ”xx” hours) and culture incubation start times were entered using the order set. Balancing measures included identification of IBI after hospital discharge, 48-hour ED revisits, and 7-day hospital readmissions.

Analysis

Measures were evaluated using statistical process control charts and run charts, and Western Electric rules were employed to determine special cause variation.¹² Annotated X-bar S control charts tracked the impact of improvement activities on average COT and LOS for all infants. Given that a relatively small number of patients (ie, two to four) met inclusion criteria each week, average COT was calculated per five patients.

This study was considered exempt from review by the CCHMC Institutional Review Board.

Of the 184 infants in this study, 46 were included as part of baseline data collection, and 138 were included during the intervention period. The median age was 26.6 days (range, 3-59 days); 52% of patients were female; two-thirds were non-Hispanic White; 22% were Black, and 5% were Hispanic (Appendix Table).

Average COT decreased from 38 hours to 32 hours with improvement activities (Figure 2) and was sustained for a total of 17 months. There were small decreases in COT after initial education was provided to attendings, nurses, and residents.

After the launch of the updated order set, median usage of the EHR order set increased from 50% to 80%. Medical discharge criteria were entered for 80 (96%) of the 83 patients for whom the updated order set was applied; culture incubation start times were entered for 78 (94%) of these patients.

No infants in our cohort were found to have IBI after hospital discharge. There were no ED revisits within 48 hours of discharge, and there were no hospital readmissions within 7 days of index discharge. Furthermore, none of the patients included in the study had growth of a pathogenic organism after 24 hours.

Of the 138 infants hospitalized during the intervention period, 77 (56%) had a COT greater than 30 hours. Among these 77 patients, 49 (64%) had their final culture incubated between 9 pm and 4 am; Furthermore, 11 (14%) had missing, abnormal, pretreated, or uninterpretable CSF studies, 7 (9%) had ongoing fevers, and 4 (5%) remained hospitalized due to family preference or inability to obtain timely outpatient follow-up.

Our study aimed to decrease the average COT from 38 hours to 30 hours among hospitalized infants aged 60 days and younger over a period of 12 months. An intervention featuring implementation of an evidence-based guideline through education, laboratory procedure transparency, creation of a standardized EHR order set, and near-time feedback was associated with a shorter average COT of 32 hours, sustained over a 17-month period. No infants with bacteremia or meningitis were inappropriately discharged during this study.

Interpretation

Prior to our improvement efforts, most febrile infants at CCHMC were observed for at least 36 hours based on a prior institutional guideline,⁶ despite recent evidence suggesting that most pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 The goal of this improvement initiative was to bridge the gap between emerging evidence and clinical practice by developing and disseminating an updated evidence-based guideline to safely decrease the hospital observation time in febrile infants aged 60 days and younger.

Similar to previous studies aimed at improving diagnosis and management among febrile infants,^13-16 generation and structured dissemination of an institutional evidence-based guideline was crucial to safely shortening COT in our population. These prior studies established a goal COT of 36 to 42 hours for hospitalized febrile infants.^13,15,16 Our study incorporated emerging evidence and local experience into an updated evidence-based practice guideline to further reduce COT to 32 hours for hospitalized infants. Key factors contributing to our success included multidisciplinary engagement, specifically partnering with nurses and resident physicians in designing and implementing our initiatives. Furthermore, improved transparency of culture monitoring practices allowed clinicians to better understand the recommended observation periods. Finally, we employed a standardized EHR order set as a no-cost, one-time, high-reliability intervention to establish 24 hours of culture monitoring as the default and to enhance transparency around start time for culture incubation.

Average COT remained stable at 32 hours for 17 months after initiation of the intervention. During the intervention period, 64% patients with hospital stays longer than 30 hours had cultures obtained between 9 pm to 4 am. These patients often remained hospitalized for longer than 30 hours to allow for a daytime hospital discharge. Additionally, CSF cultures were only monitored manually once per day between 8 am and 10 am. As a result, CSF cultures obtained in the evening (eg, 9 pm) would be evaluated once at roughly 12 hours of incubation, and then the following morning at 36 hours of incubation. In cases where CSF studies (eg, cell count, protein, Gram stain) were abnormal, uninterpretable, or could not be obtained, clinicians monitored CSF cultures closer to 36 hours from incubation. While evidence-based guidelines and local data support safe early discharge of febrile infants, clinicians presented with incomplete or uninterpretable data were appropriately more likely to observe infants for longer periods to confirm negative cultures.

Limitations

The study has several limitations. First, this single-center study was conducted at a quaternary care medical center with a robust quality improvement infrastructure. Our interventions took advantage of the existing processes in place that ensure timely discharge of medically ready patients.¹¹ Furthermore, microbiology laboratory practices are unique to our institution. These factors limit the generalizability of this work. Second, due to small numbers of eligible infants, analyses were conducted per five patients. Infrequent hospitalizations limited our ability to learn quickly from PDSA cycles. Finally, we did not measure cost savings attributable to shorter hospital stays. However, in addition to financial savings from charges and decreased nonmedical costs such as lost earnings and childcare,¹⁷ shorter hospitalizations have many additional benefits, such as promoting bonding and breastfeeding and decreasing exposure to nosocomial infections. Shorter hospitalizations, with clearly communicated discharge times, also serve to optimize patient throughput.

Implementation of a clinical practice guideline resulted in reduction of average COT from 38 to 32 hours in febrile infants aged 60 days and younger, with no cases of missed IBI. Engagement of multidisciplinary stakeholders in the generation and structured dissemination of the evidence-based guideline, improved transparency of the microbiological blood and CSF culture process, and standardization of EHR order sets were crucial to the success of this work. Cultures incubated overnight and daily CSF culture-monitoring practices primarily contributed to an average LOS of more than 30 hours.

Future work will include collaboration with emergency physicians to improve evaluation efficiency and decrease LOS in the ED for febrile infants. Additionally, creation of an automated data dashboard of COT and LOS will provide clinicians with real-time feedback on hospitalization practices.

The authors thank Dr Jeffrey Simmons, MD, MSc, as well as the members of the 2019 Fever of Uncertain Source Evidence-Based Guideline Committee. We also thank the James M Anderson Center for Health System Excellence and the Rapid Cycle Improvement Collaborative for their support with guideline development as well as design and execution of our improvement efforts.

Febrile infants aged 0 to 60 days often undergo diagnostic testing to evaluate for invasive bacterial infections (IBI; ie, bacteremia and meningitis) and are subsequently hospitalized pending culture results. Only 1% to 2% of infants 0 to 60 days old have an IBI,^1-3 and most hospitalized infants are discharged once physicians feel confident that pathogens are unlikely to be isolated from blood and cerebrospinal fluid (CSF) cultures. Practice regarding duration of hospitalization while awaiting blood and CSF culture results is not standardized in this population. Longer hospitalizations can lead to increased costs and familial stress, including difficulty with breastfeeding and anxiety in newly postpartum mothers.^4,5

In 2010, an institutional evidence-based guideline for the management of febrile infants aged 0 to 60 days recommended discharge after 36 hours of observation if all cultures were negative.⁶ However, recent studies demonstrate that 85% to 93% of pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 Assuming a 2% prevalence of IBI, if 15% of pathogens were identified after 24 hours of incubation, only one out of 333 infants would have an IBI identified after 24 hours of hospital observation.⁷

Furthermore, a review of our institution’s electronic health records (EHR) over the past 5 years revealed that an observation period of 24 hours would have resulted in the discharge of three infants with an IBI. Two infants had bacteremia; both were discharged from the emergency department (ED) without antibiotics, returned to care after cultures were reported positive at 27 hours, and had no adverse outcomes. The third infant had meningitis, but also had an abnormal CSF Gram stain, which led to a longer hospitalization.

In 2019, our institution appraised the emerging literature and institutional data supporting the low absolute risk of missed IBI, and also leveraged local consensus among key stakeholders to update its evidence-based guideline for the evaluation and management of febrile infants aged 60 days and younger. The updated guideline recommends that clinicians consider discharging well-appearing neonates and infants if blood and CSF cultures remain negative at 24 hours.¹⁰ The objective of this study was to decrease the average hospital culture observation time (COT; culture incubation to hospital discharge) from 38 to 30 hours over a 12-month period in febrile infants aged 0 to 60 days.

Context

Improvement efforts were conducted at Cincinnati Children’s Hospital Medical Center (CCHMC), a large, urban, academic hospital that admitted more than 8,000 noncritically ill patients to the hospital medicine (HM) service from July 1, 2018, through June 30, 2019. Hospital medicine teams, located at both the main and satellite campuses, are staffed by attending physicians, fellows, residents, medical students, and nurse practitioners. The two campuses, which are about 20 miles apart, share clinician providers but have distinct nursing pools.

Microbiology services for all CCHMC patients are provided at the main campus. Blood and CSF cultures at the satellite campus are transported to the main campus for incubation and monitoring via an urgent courier service. The microbiology laboratory at CCHMC uses a continuous monitoring system for blood cultures (BACT/ALERT Virtuo, BioMérieux). The system automatically alerts laboratory technicians of positive cultures; these results are reported to clinical providers within 30 minutes of detection. Laboratory technicians manually evaluate CSF cultures once daily for 5 days.

Improvement Team

Our improvement team included three HM attending physicians; two HM fellows; a pediatric chief resident; two nurses, who represented nursing pools at the main and satellite campuses; and a clinical pharmacist, who is a co-leader of the antimicrobial stewardship program at CCHMC. Supporting members for the improvement team included the CCHMC laboratory director; the microbiology laboratory director; an infectious disease physician, who is a co-leader of the antimicrobial stewardship program; and nursing directors of the HM units at both campuses.

Evidence-Based Guideline

Our improvement initiative was based on recommendations from the updated CCHMC Evidence-Based Care Guideline for Management of Infants 0 to 60 days with Fever of Unknown Source.¹⁰ This guideline, published in May 2019, was developed by a multidisciplinary working group composed of key stakeholders from HM, community pediatrics, emergency medicine, the pediatric residency program, infectious disease, and laboratory medicine. Several improvement team members were participants on the committee that published the evidence-based guideline. The committee first performed a systematic literature review and critical appraisal of the literature. Care recommendations were formulated via a consensus process directed by best evidence, patient and family preferences, and clinical expertise; the recommendations were subsequently reviewed and approved by clinical experts who were not involved in the development process.

Based on evidence review and multistakeholder consensus, the updated guideline recommends clinicians consider discharging neonates and infants aged 60 days and younger if there is no culture growth after an observation period of 24 hours (as documented in the EHR) and patients are otherwise medically ready for discharge (ie, well appearing with adequate oral intake).^10,11 In addition, prior to discharge, there must be a documented working phone number on file for the patient’s parents/guardians, an established outpatient follow-up plan within 24 hours, and communication with the primary pediatrician who is in agreement with discharge at 24 hours.

Study Population

Infants 0 to 60 days old who had a documented or reported fever without an apparent source based on history and physical exam upon presentation to the ED, and who were subsequently admitted to the HM service at CCHMC between October 30, 2018, and July 10, 2020, were eligible for inclusion. We excluded infants who were admitted to other clinical services (eg, intensive care unit); had organisms identified on blood, urine, or CSF culture within 24 hours of incubation; had positive herpes simplex virus testing; had skin/soft tissue infections or another clearly documented source of bacterial infection; or had an alternative indication for hospitalization (eg, need for intravenous fluid or deep suctioning) after cultures had incubated for 24 hours. Infants who had a positive blood, urine, or CSF culture result after 24 hours of incubation were included in the study population. Organisms were classified as pathogen or contaminant based on treatment decisions made by the care team.

Improvement Activities

Key drivers critical to success of the improvement efforts were: (1) clearly defined standard of care for duration of observation in febrile infants 0 to 60 days old; (2) improved understanding of microbiology lab procedures; (3) effective communication of discharge criteria between providers and nurses; and (4) transparency of data with feedback (Figure 1).

Education and Structured Dissemination of Evidence-Based Guideline

The CCHMC febrile infant guideline¹⁰ was disseminated to HM physicians, residents, and nurses via the following means: (1) in-person announcements at staff meetings and educational conferences, (2) published highlights from the guideline in weekly newsletters, and (3) email announcements. Additionally, members of the study team educated HM attending physicians, nursing staff from the medical units at both campuses, and resident physicians about recent studies demonstrating safety of shorter length of stay (LOS) in febrile infants aged 0 to 60 days. The study team also provided residents, physicians, and nurses with data on the number of positive blood and CSF cultures and outcomes of patients at CCHMC within the past 5 years. In addition, team members led a journal club for residents discussing an article⁷ describing time-to-positivity of blood and CSF cultures in febrile infants. For ongoing engagement, the evidence-based guideline and a detailed explanation of microbiology procedures were published in the resident handbook, an internal resource that includes vital clinical pearls and practice guidelines across specialties. (Each resident receives an updated hard copy each year, and there is also an online link to the resource in the EHR.) Information about the guideline and COT was also included in the monthly chief resident’s orientation script, which is relayed to all residents on the first day of their HM rotation.

Clear Communication of Microbiology Procedures

Team members created a detailed process map describing the processing protocols for blood and CSF cultures collected at both CCHMC campuses. This information was shared with HM attending physicians and nurses via in-person announcements at staff meetings, flyers in team workrooms, and email communications. Residents received information on microbiology protocols via in-person announcements at educational conferences and dissemination in the weekly residency newsletter.Important information communicated included:

1. Definition of culture start time. We conveyed that there may be a delay of up to 4 hours between culture collection at the satellite campus and culture incubation at the main campus laboratory. As a result, the time of blood or CSF sample arrival to the main campus laboratory was a more accurate reflection of the culture incubation start time than the culture collection time.

2. Explanation of CSF culture processing. We discussed the process by which these cultures are plated upon arrival at the microbiology laboratory and read once per day in the morning. Therefore, a culture incubated at midnight would be evaluated once at 9 hours and not again until 33 hours.

Modification of Febrile Infant Order Set

Enhancements to the febrile infant order set improved communication and cultivated a shared mental model regarding discharge goals among all members of the care team. The EHR order set for febrile infants was updated as follows: (1) mandatory free-text fields that established the culture start time for blood and CSF cultures were added, (2) culture start time was clearly defined (ie, the time culture arrives at the main campus laboratory), and (3) a change was made in the default discharge criteria¹¹ to “culture observation for 24 hours,” with the ability to modify COT (Appendix Figure 1). We embedded hyperlinks to the guideline and microbiology process map within the updated order set, which allowed providers to easily access this information and refresh their knowledge of the recommendations (Appendix Figure 1).

Identification of Failures and Follow-up With Near-Time Feedback

All cases of febrile infants were tracked weekly. For infants hospitalized longer than 24 hours, the study team contacted the discharging clinicians to discuss reasons for prolonged hospitalization, with an emphasis on identifying system-level barriers to earlier discharge.

Study of the Interventions

The institutional microbiology database was queried weekly to identify all infants 0 to 60 days old who had a blood culture obtained and were hospitalized on the HM service. Study team members conducted targeted EHR review to determine whether patients met exclusion criteria and to identify reasons for prolonged COT. Baseline data were collected retrospectively for a 3-month period prior to initiation of improvement activities. During the study period, queries were conducted weekly and reviewed by study team members to evaluate the impact of improvement activities and to inform new interventions.

Measures

Our primary outcome measure was COT, defined as the hours between final culture incubation and hospital discharge. The operational definition for “final culture incubation” was the documented time of arrival of the last collected culture to the microbiology laboratory. Our goal COT was 30 hours to account for a subset of patients whose blood and/or CSF culture were obtained overnight (ie, after 9 pm), since subsequent discharge times would likely and practically be delayed beyond 24 hours. Our secondary outcome measure was LOS, defined as the time between ED arrival and hospital discharge. Process measures included the proportion of patients for whom the febrile infant EHR order set was used and the proportion of patients for whom medical discharge criteria (ie, blood and CSF culture observed for ”xx” hours) and culture incubation start times were entered using the order set. Balancing measures included identification of IBI after hospital discharge, 48-hour ED revisits, and 7-day hospital readmissions.

Analysis

Measures were evaluated using statistical process control charts and run charts, and Western Electric rules were employed to determine special cause variation.¹² Annotated X-bar S control charts tracked the impact of improvement activities on average COT and LOS for all infants. Given that a relatively small number of patients (ie, two to four) met inclusion criteria each week, average COT was calculated per five patients.

This study was considered exempt from review by the CCHMC Institutional Review Board.

Of the 184 infants in this study, 46 were included as part of baseline data collection, and 138 were included during the intervention period. The median age was 26.6 days (range, 3-59 days); 52% of patients were female; two-thirds were non-Hispanic White; 22% were Black, and 5% were Hispanic (Appendix Table).

Average COT decreased from 38 hours to 32 hours with improvement activities (Figure 2) and was sustained for a total of 17 months. There were small decreases in COT after initial education was provided to attendings, nurses, and residents.

After the launch of the updated order set, median usage of the EHR order set increased from 50% to 80%. Medical discharge criteria were entered for 80 (96%) of the 83 patients for whom the updated order set was applied; culture incubation start times were entered for 78 (94%) of these patients.

No infants in our cohort were found to have IBI after hospital discharge. There were no ED revisits within 48 hours of discharge, and there were no hospital readmissions within 7 days of index discharge. Furthermore, none of the patients included in the study had growth of a pathogenic organism after 24 hours.

Of the 138 infants hospitalized during the intervention period, 77 (56%) had a COT greater than 30 hours. Among these 77 patients, 49 (64%) had their final culture incubated between 9 pm and 4 am; Furthermore, 11 (14%) had missing, abnormal, pretreated, or uninterpretable CSF studies, 7 (9%) had ongoing fevers, and 4 (5%) remained hospitalized due to family preference or inability to obtain timely outpatient follow-up.

Our study aimed to decrease the average COT from 38 hours to 30 hours among hospitalized infants aged 60 days and younger over a period of 12 months. An intervention featuring implementation of an evidence-based guideline through education, laboratory procedure transparency, creation of a standardized EHR order set, and near-time feedback was associated with a shorter average COT of 32 hours, sustained over a 17-month period. No infants with bacteremia or meningitis were inappropriately discharged during this study.

Interpretation

Prior to our improvement efforts, most febrile infants at CCHMC were observed for at least 36 hours based on a prior institutional guideline,⁶ despite recent evidence suggesting that most pathogens in blood and CSF cultures grow within 24 hours of incubation.^7-9 The goal of this improvement initiative was to bridge the gap between emerging evidence and clinical practice by developing and disseminating an updated evidence-based guideline to safely decrease the hospital observation time in febrile infants aged 60 days and younger.

Similar to previous studies aimed at improving diagnosis and management among febrile infants,^13-16 generation and structured dissemination of an institutional evidence-based guideline was crucial to safely shortening COT in our population. These prior studies established a goal COT of 36 to 42 hours for hospitalized febrile infants.^13,15,16 Our study incorporated emerging evidence and local experience into an updated evidence-based practice guideline to further reduce COT to 32 hours for hospitalized infants. Key factors contributing to our success included multidisciplinary engagement, specifically partnering with nurses and resident physicians in designing and implementing our initiatives. Furthermore, improved transparency of culture monitoring practices allowed clinicians to better understand the recommended observation periods. Finally, we employed a standardized EHR order set as a no-cost, one-time, high-reliability intervention to establish 24 hours of culture monitoring as the default and to enhance transparency around start time for culture incubation.

Average COT remained stable at 32 hours for 17 months after initiation of the intervention. During the intervention period, 64% patients with hospital stays longer than 30 hours had cultures obtained between 9 pm to 4 am. These patients often remained hospitalized for longer than 30 hours to allow for a daytime hospital discharge. Additionally, CSF cultures were only monitored manually once per day between 8 am and 10 am. As a result, CSF cultures obtained in the evening (eg, 9 pm) would be evaluated once at roughly 12 hours of incubation, and then the following morning at 36 hours of incubation. In cases where CSF studies (eg, cell count, protein, Gram stain) were abnormal, uninterpretable, or could not be obtained, clinicians monitored CSF cultures closer to 36 hours from incubation. While evidence-based guidelines and local data support safe early discharge of febrile infants, clinicians presented with incomplete or uninterpretable data were appropriately more likely to observe infants for longer periods to confirm negative cultures.

Limitations

The study has several limitations. First, this single-center study was conducted at a quaternary care medical center with a robust quality improvement infrastructure. Our interventions took advantage of the existing processes in place that ensure timely discharge of medically ready patients.¹¹ Furthermore, microbiology laboratory practices are unique to our institution. These factors limit the generalizability of this work. Second, due to small numbers of eligible infants, analyses were conducted per five patients. Infrequent hospitalizations limited our ability to learn quickly from PDSA cycles. Finally, we did not measure cost savings attributable to shorter hospital stays. However, in addition to financial savings from charges and decreased nonmedical costs such as lost earnings and childcare,¹⁷ shorter hospitalizations have many additional benefits, such as promoting bonding and breastfeeding and decreasing exposure to nosocomial infections. Shorter hospitalizations, with clearly communicated discharge times, also serve to optimize patient throughput.

Implementation of a clinical practice guideline resulted in reduction of average COT from 38 to 32 hours in febrile infants aged 60 days and younger, with no cases of missed IBI. Engagement of multidisciplinary stakeholders in the generation and structured dissemination of the evidence-based guideline, improved transparency of the microbiological blood and CSF culture process, and standardization of EHR order sets were crucial to the success of this work. Cultures incubated overnight and daily CSF culture-monitoring practices primarily contributed to an average LOS of more than 30 hours.

Future work will include collaboration with emergency physicians to improve evaluation efficiency and decrease LOS in the ED for febrile infants. Additionally, creation of an automated data dashboard of COT and LOS will provide clinicians with real-time feedback on hospitalization practices.

The authors thank Dr Jeffrey Simmons, MD, MSc, as well as the members of the 2019 Fever of Uncertain Source Evidence-Based Guideline Committee. We also thank the James M Anderson Center for Health System Excellence and the Rapid Cycle Improvement Collaborative for their support with guideline development as well as design and execution of our improvement efforts.

Bronchiolitis is the leading cause of hospitalization in infants aged <1 year in the United States.^1-3 Estimates suggest that 1.5% to 2.0% of US infants require hospitalization every year, with a median (interquartile range) length of stay of 2 days (1-4),³ incurring direct medical costs of $555 million annually.¹ Evidence suggests that few interventions, aside from supportive care, are effective for bronchiolitis.^4-7 Adherence to standardized clinical guidelines could improve outcomes and resource use by streamlining care and limiting ineffective interventions, thereby decreasing hospital length of stay, which is a major medical cost.^8-13 For this reason, many hospitals have adopted bronchiolitis guidelines, although institutional practices vary.^14,15

Two relatively unexplored methods to reduce the inpatient burden of bronchiolitis are the use of observation units (OU) and home oxygen therapy (HOT). Motivated by research demonstrating the safety and effectiveness of an emergency department (ED)–based HOT protocol,¹⁶ where 36 of 37 patients with mild hypoxemia discharged on HOT avoided hospital admission, our institution implemented an observation unit and home oxygen therapy (OU-HOT) protocol designed to return children with bronchiolitis home earlier from the hospital. In the first winter season of implementation (2010 to 2011), the OU-HOT protocol was associated with significant reductions in length of stay and substantial cost savings, without an increase in return visits to the ED or inpatient readmissions.¹⁷ The objectives of this study were to determine whether these encouraging initial findings persisted and to measure the long-term impact of the OU-HOT protocol.

We conducted a retrospective cohort study of children hospitalized with bronchiolitis at Primary Children’s Hospital, a freestanding children’s hospital in Salt Lake City, Utah. Discharge diagnosis and procedures codes, as well as laboratory, imaging, pharmacy, and supply costs, were obtained from the Intermountain Healthcare enterprise data warehouse. A crosswalk available from the Centers for Medicare and Medicaid Services was used to convert International Classification of Diseases (ICD)-10 discharge diagnosis and procedure codes to ICD-9 equivalents.¹⁸ This study was approved by the University of Utah institutional review board (00110419).

Patients

Children aged 3 to 24 months who were discharged with a diagnosis of bronchiolitis (466.xx) during winter seasons from 2007 to 2019 were included. A winter season was defined as November 1 to April 30. Both observation and inpatient encounters were included in the cohort. We excluded patients with discharge diagnosis or procedure codes indicating tracheostomy (519.0-519.09, V44.0, V55.0, 31.1, 31.21, 31.41, 31.74, 97.23), ventilator dependence (V46.1x), chronic lung disease (518.83, 770.7), or pulmonary hypertension (416.xx). Patients with both bronchiolitis and a concurrent diagnosis, such as otitis media or pneumonia, were included unless exclusion criteria were met.

Intervention and Process Measures

Our institution implemented the OU-HOT protocol at the start of the 2010-2011 winter season.¹⁷ The aim of the OU-HOT protocol was to discharge children with bronchiolitis home sooner by increasing use of both an OU, with frequent assessment of discharge readiness, and HOT to help children become ready for discharge. Similar to most OUs, admission to our unit was limited to patients who met hospital admission criteria, and had a short anticipated length of stay (<48 hours). As a self-contained 20-bed unit providing 24-hour dedicated pediatrician/pediatric emergency medicine physician and nursing coverage, the OU actively monitored patients’ discharge readiness, with a goal to facilitate patient throughput more akin to an ED rather than a traditional inpatient unit. Patients who could not be discharged from the OU within 48 hours were transferred to the inpatient unit. Although the OU existed at the time of protocol implementation, its use for patients with bronchiolitis was not actively encouraged until implementation.

Hospitalized patients—in either inpatient or observation units—were eligible for discharge on HOT if they met the following criteria: hypoxemia was the only indication for continued hospitalization, the child’s oxygen requirement was <0.5 L/min for at least 6 hours (0.8 L/min for children aged >1 year), the child’s caregiver(s) were willing to manage oxygen at home, and the child had reliable access to primary care provider follow up. We used two process measures across winter seasons: (1) the percentage of patients discharged from the OU, and (2) the percentage of patients discharged with HOT. The percentage of patients discharged on HOT was estimated by a manual chart review and an electronic medical record (EMR) HOT flag that came into existence with our hospital system’s adoption of a new EMR (2017-2019). Chart review randomly sampled patients from 2007-2017, totaling 457 patients. To estimate the reliability of this method, we calculated the sensitivity, specificity, positive predictive value, and negative predictive value of the EMR HOT flag using chart review as the gold standard.

Outcome Measures

The main outcome measure was mean hospital length of stay. Balancing measures were revisit rates (stratified into ED visits and readmissions) and annual per-population bronchiolitis admission rates. Visits were considered revisits if they occurred within 7 days of initial hospital discharge, and included visits to Primary Children’s Hospital as well as 22 other Intermountain Healthcare hospitals. Population estimates from the Utah Department of Health were used to calculate the annual population-based rate of bronchiolitis admissions to Primary Children’s Hospital.¹⁹ Annual admission rates were calculated per 10,000 children aged 3 to 24 months who resided in Utah each year of the study period, and were evaluated to determine if patients were admitted more frequently after OU-HOT implementation. Secondary outcome measures included the percentage of patients discharged within 24 hours and mean inflation-adjusted cost per episode of care (in 2019 dollars). Hospitalization costs were determined using Intermountain Healthcare’s internal cost accounting system, an activity-based method that aggregates costs of individual resources according to date of service.²⁰ Costs were adjusted to 2019 dollars and were defined as the total costs of a patient’s initial hospitalization as well as any 7-day revisit encounters.

Data Analysis

Demographic data were compared before and after OU-HOT protocol implementation using Pearson chi-square tests. Multivariable linear or logistic regression models were used to compare measures before and after OU-HOT protocol implementation via an interrupted time-series approach. The interrupted time-series analysis measured two types of changes after protocol implementation during the 2010-2011 winter season: (1) any immediate change in the level of an outcome (immediate effect) and (2) any change of an outcome going forward over time (change in slope).²¹ Covariates in the regression models included patient age, sex, race, ethnicity, and insurance type, as well as presence of an underlying complex chronic condition, mechanical ventilation use, and pediatric intensive care unit (PICU) admission during hospitalization. Data were analyzed in STATA 15 (StataCorp LLC).²²

A total of 7,116 patients met inclusion criteria over the study period (2,061 pre-implementation, 5,055 post-implementation). A comparison of patient characteristics before and after HOT protocol implementation is presented in Table 1. Patients were similar in terms of age, sex, and insurance type. Patients in the postimplementation period were more likely to have a complex chronic condition, require admission to the PICU, and need mechanical ventilation (P < .01). Differences between cohorts with regard to race/ethnicity distribution largely were a result of improved capture of these data elements in the postimplementation period. For example, 30% of patients were classified as “race/ethnicity unknown” in the preimplementation cohort, compared with 4% of patients in the postimplementation period.

Process Measures

Figure 1 shows trends in OU and HOT use by winter season. The percentage of patients discharged from the OU increased immediately after OU-HOT protocol implementation (absolute 26.9% immediate increase; 95% CI, 21.9-42.2). The change in the proportion of OU use per season also increased (change in slope +3.9% per season; 95% CI, 3.4%-4.4%). The percentage of patients discharged with HOT increased immediately after OU-HOT protocol implementation (26.0% immediate change; 95% CI, 18.9%-33.1%); however, the immediate increase in HOT discharges was coupled with a declining rate of HOT discharges per season in the postprotocol period compared with the preprotocol period (change in slope –4.5% per season; 95% CI, –7.5% to –1.5%). Our chart review and EMR flag included 1,354 patients, or 19.0% of our cohort. Our EMR flag for HOT in the last two seasons of the study had a positive predictive value of 100% (5 of 5 identified by EMR flag as receiving HOT were confirmed by chart review) and negative predictive value of 89% (31 of 35 identified by EMR flag as not receiving HOT were confirmed by chart review). The specificity of the EMR flag was 100% (31 of 31 of those confirmed by chart review as not receiving HOT, who were correctly identified by EMR) and the sensitivity was 55% (5 of 9 of those confirmed by chart review as receiving HOT, who were correctly identified by EMR).

Primary and Secondary Outcomes

Trends in length of stay across winter seasons are presented in Figure 2. The OU-HOT protocol was associated with an immediate reduction of 30.6 hours in mean length of stay (95% CI, –37.1 to –24.2). The rate of change in length of stay postimplementation did not differ significantly from the rate of change preimplementation (change in slope –0.6 hours per season; 95% CI, –2.3 to 1.1 hours). The percentage of patients discharged within 24 hours of admission rose immediately after protocol implementation, by 23.8 absolute percentage points (95% CI, 11.7-28.8). Slopes of the preintervention and postintervention regression lines did not differ significantly (change in slope –0.1% per season; 95% CI, –1.4% to 1.1%). Immediate decreases in length of stay were accompanied by an immediate decrease in mean cost per episode of care (–$4,181; 95% CI, –$4,829 to –$3,533). Protocol implementation also was associated with a decreased slope in cost postimplementation (change in slope –$403 per season; 95% CI, –$543 to –$264). The total cost savings, estimated by the product of the average cost savings per episode of care and the number of bronchiolitis admissions included in the study after OU-HOT implementation, amounted to $21.1 million over the 9-year period, or $2.3 million per winter season.

Balancing Measures

We observed an immediate reduction in 7-day hospital revisits (–1.1% immediate change; 95% CI, –1.8% to –0.4%), but an increasing slope in revisits after implementation (change in slope 0.4% per season; 95% CI, 0.1%-0.8%) (Figure 3). Stratifying revisits into ED visits and readmissions revealed that the revisit findings reflected changes in ED return visits, for which there was an immediate reduction at the time of implementation (–1.0% immediate change; 95% CI, –1.6% to –0.4%), but an increasing slope postimplementation (change in slope 0.5% per season; 95% CI, 0.2-0.8). Neither an immediate intervention effect (0.0% immediate change; 95% CI, –0.5% to 0.4%) nor a change in slope (change in slope 0.0% per season; 95% CI, –0.1% to 0.1%) were observed for inpatient readmissions alone. The annual rate of bronchiolitis admissions to Primary Children’s Hospital per 10,000 children who reside in Utah decreased after implementation of the OU-HOT protocol (immediate intervention effect –6.2 admissions; 95% CI, –10.8 to –1.6; change in slope –1.8 admissions per season; 95% CI, –2.8 to –0.69).

Our OU-HOT protocol was associated with immediate improvements in care delivered to children hospitalized for bronchiolitis, including decreased length of stay and cost savings. These improvements in outcomes largely have been sustained over a 9-year period. The OU-HOT protocol also appears to be safe as evidenced by a stable rate of readmissions over the study period and only a small increase in revisits to EDs across Intermountain Healthcare facilities, which see most children in the catchment area. Our OU-HOT protocol represents a combination of two interventions: (1) the creation of an OU focused on discharge within 24 to 48 hours of admission and (2) encouragement to discharge children with HOT. We found that use of the OU and a commitment to timely discharges has been sustained in recent years, while the commitment to HOT has appeared to wane.

Earlier investigations have evaluated the efficacy of HOT in the ED setting to prevent hospital admissions, finding high levels of caregiver comfort, estimating $1,300 per patient cost savings, and reporting readmission rates of approximately 5%.^16,23-25 Our study is unique in addressing HOT among a population of patients already hospitalized with bronchiolitis. The cost reductions we observed with our OU-HOT protocol were similar to those noted in the ED-based HOT protocols. However, we recorded lower readmission rates, likely because of the additional time allotted to caregivers to better gauge illness trajectory in the inpatient setting vs the ED, as well as additional time for hospitalized patients to reach the plateau or convalescent phase of illness. The small increase in ED revisits that we measured in recent years might be related to the concurrent rise in patient acuity and complexity.

Considering that length of stay has remained low despite less commitment to HOT, our results suggest that the OU might be the more impactful of the two interventions, and these data support the use of such a unit for a subset of patients with bronchiolitis. However, it is important to note that while the EMR HOT flag demonstrated high specificity, positive predictive value, and negative predictive value, the sensitivity was low (56%). As a result, it is possible that we have underestimated HOT use in the 2017-2018 and 2018-2019 seasons, the final two years of the study. Alternatively, the discrepancy between sustained outcomes and lagging use of HOT could be explained by improved identification of patients who would experience the greatest benefit with oxygen in terms of length of stay reductions, with fewer patients discharged on HOT but greater per-patient benefit. Finally, in an era that encourages reduced monitor use and less aggressive response to transient mild desaturations,^13,26,27 it is possible that fewer patients are identified with clinically actionable hypoxemia around the time they would be otherwise discharged.

Our OU-HOT model is not unprecedented. Increasingly, other formerly inpatient indications are being successfully managed in the observation, outpatient, and home setting, such as parenteral antibiotic treatment^28,29 and chemotherapy administration.³⁰ Considering the inpatient burden of bronchiolitis, similar strategies to expedite discharge are needed. Although outpatient intravenous antibiotic and chemotherapy administration have been widely adopted, we are aware of only one other pediatric health care system in the United States (Children’s Hospital Colorado) that routinely discharges inpatients with bronchiolitis on HOT.

This study has several limitations. First, although the interrupted time-series analysis is designed to account for trends that precede an intervention and covariates that differ before and after the intervention, it is possible that important unmeasured patient factors or changes in practice patterns differed between the pre- and post-intervention cohorts. There were no major changes to the OU-HOT protocol or discharge criteria after implementation, but individual practice management of bronchiolitis during the study period likely has evolved as new evidence emerges. Second, one could postulate that the increase in discharges within 24 hours and accompanying decreases in average length of stay and cost could be achieved by hospitalizing healthier patients over time, which the presence of an OU might incentivize. To the contrary, we found that population-based bronchiolitis admission rates have declined and disease severity appears to be increased since implementation of the OU-HOT protocol. The increase in medically complex children and PICU use in our postimplementation cohort aligns with recently published data suggesting these are national trends.^3,31 Third, HOT use was estimated from a sample of the cohort using a chart review and a newly available EMR flag. A low sensitivity and a small sample for the positive predictive value are limitations of the EMR flag.

Additionally, there are almost certainly unmeasured ambulatory burdens of HOT not captured by this study. ED-based protocols have estimated that patients discharged with HOT have a median of two follow-up ambulatory visits before oxygen is discontinued³²; however, the ambulatory burden associated with discharge on HOT after a hospitalization and the extent to which demographic factors affect that burden is unknown. Furthermore, one insurance company charged $94 for a month of HOT in 2019; paying even a portion of this charge represents a nontrivial financial burden for many families, even considering inpatient cost savings. Although the decision to discharge on oxygen or remain hospitalized until the child did not need oxygen was left to the parents, their posthospitalization perspectives were not assessed in this study. Although reports indicate that families largely feel positive about HOT after discharge from an ED setting, with 90% of caregivers preferring HOT use to inpatient admission and most reporting no difficulty with home management,²³ it is uncertain whether this would also apply after inpatient hospitalization.

The OU-HOT bronchiolitis protocol was associated with decreases in inpatient length of stay and cost while appearing safe to implement. The sustained use of the OU combined with declining use of HOT suggests that the OU might be the more impactful intervention. As previously inpatient indications such as parenteral antibiotics and chemotherapy increasingly have been administered in observation and outpatient settings, bronchiolitis appears ideal for a similar strategy that allows patients to spend less time in the hospital. Studies are needed to understand the outpatient burden of HOT and the generalizability of our findings.

Bronchiolitis is the leading cause of hospitalization in infants aged <1 year in the United States.^1-3 Estimates suggest that 1.5% to 2.0% of US infants require hospitalization every year, with a median (interquartile range) length of stay of 2 days (1-4),³ incurring direct medical costs of $555 million annually.¹ Evidence suggests that few interventions, aside from supportive care, are effective for bronchiolitis.^4-7 Adherence to standardized clinical guidelines could improve outcomes and resource use by streamlining care and limiting ineffective interventions, thereby decreasing hospital length of stay, which is a major medical cost.^8-13 For this reason, many hospitals have adopted bronchiolitis guidelines, although institutional practices vary.^14,15

Two relatively unexplored methods to reduce the inpatient burden of bronchiolitis are the use of observation units (OU) and home oxygen therapy (HOT). Motivated by research demonstrating the safety and effectiveness of an emergency department (ED)–based HOT protocol,¹⁶ where 36 of 37 patients with mild hypoxemia discharged on HOT avoided hospital admission, our institution implemented an observation unit and home oxygen therapy (OU-HOT) protocol designed to return children with bronchiolitis home earlier from the hospital. In the first winter season of implementation (2010 to 2011), the OU-HOT protocol was associated with significant reductions in length of stay and substantial cost savings, without an increase in return visits to the ED or inpatient readmissions.¹⁷ The objectives of this study were to determine whether these encouraging initial findings persisted and to measure the long-term impact of the OU-HOT protocol.

We conducted a retrospective cohort study of children hospitalized with bronchiolitis at Primary Children’s Hospital, a freestanding children’s hospital in Salt Lake City, Utah. Discharge diagnosis and procedures codes, as well as laboratory, imaging, pharmacy, and supply costs, were obtained from the Intermountain Healthcare enterprise data warehouse. A crosswalk available from the Centers for Medicare and Medicaid Services was used to convert International Classification of Diseases (ICD)-10 discharge diagnosis and procedure codes to ICD-9 equivalents.¹⁸ This study was approved by the University of Utah institutional review board (00110419).

Patients

Children aged 3 to 24 months who were discharged with a diagnosis of bronchiolitis (466.xx) during winter seasons from 2007 to 2019 were included. A winter season was defined as November 1 to April 30. Both observation and inpatient encounters were included in the cohort. We excluded patients with discharge diagnosis or procedure codes indicating tracheostomy (519.0-519.09, V44.0, V55.0, 31.1, 31.21, 31.41, 31.74, 97.23), ventilator dependence (V46.1x), chronic lung disease (518.83, 770.7), or pulmonary hypertension (416.xx). Patients with both bronchiolitis and a concurrent diagnosis, such as otitis media or pneumonia, were included unless exclusion criteria were met.

Intervention and Process Measures

Our institution implemented the OU-HOT protocol at the start of the 2010-2011 winter season.¹⁷ The aim of the OU-HOT protocol was to discharge children with bronchiolitis home sooner by increasing use of both an OU, with frequent assessment of discharge readiness, and HOT to help children become ready for discharge. Similar to most OUs, admission to our unit was limited to patients who met hospital admission criteria, and had a short anticipated length of stay (<48 hours). As a self-contained 20-bed unit providing 24-hour dedicated pediatrician/pediatric emergency medicine physician and nursing coverage, the OU actively monitored patients’ discharge readiness, with a goal to facilitate patient throughput more akin to an ED rather than a traditional inpatient unit. Patients who could not be discharged from the OU within 48 hours were transferred to the inpatient unit. Although the OU existed at the time of protocol implementation, its use for patients with bronchiolitis was not actively encouraged until implementation.

Hospitalized patients—in either inpatient or observation units—were eligible for discharge on HOT if they met the following criteria: hypoxemia was the only indication for continued hospitalization, the child’s oxygen requirement was <0.5 L/min for at least 6 hours (0.8 L/min for children aged >1 year), the child’s caregiver(s) were willing to manage oxygen at home, and the child had reliable access to primary care provider follow up. We used two process measures across winter seasons: (1) the percentage of patients discharged from the OU, and (2) the percentage of patients discharged with HOT. The percentage of patients discharged on HOT was estimated by a manual chart review and an electronic medical record (EMR) HOT flag that came into existence with our hospital system’s adoption of a new EMR (2017-2019). Chart review randomly sampled patients from 2007-2017, totaling 457 patients. To estimate the reliability of this method, we calculated the sensitivity, specificity, positive predictive value, and negative predictive value of the EMR HOT flag using chart review as the gold standard.

Outcome Measures

The main outcome measure was mean hospital length of stay. Balancing measures were revisit rates (stratified into ED visits and readmissions) and annual per-population bronchiolitis admission rates. Visits were considered revisits if they occurred within 7 days of initial hospital discharge, and included visits to Primary Children’s Hospital as well as 22 other Intermountain Healthcare hospitals. Population estimates from the Utah Department of Health were used to calculate the annual population-based rate of bronchiolitis admissions to Primary Children’s Hospital.¹⁹ Annual admission rates were calculated per 10,000 children aged 3 to 24 months who resided in Utah each year of the study period, and were evaluated to determine if patients were admitted more frequently after OU-HOT implementation. Secondary outcome measures included the percentage of patients discharged within 24 hours and mean inflation-adjusted cost per episode of care (in 2019 dollars). Hospitalization costs were determined using Intermountain Healthcare’s internal cost accounting system, an activity-based method that aggregates costs of individual resources according to date of service.²⁰ Costs were adjusted to 2019 dollars and were defined as the total costs of a patient’s initial hospitalization as well as any 7-day revisit encounters.

Data Analysis

Demographic data were compared before and after OU-HOT protocol implementation using Pearson chi-square tests. Multivariable linear or logistic regression models were used to compare measures before and after OU-HOT protocol implementation via an interrupted time-series approach. The interrupted time-series analysis measured two types of changes after protocol implementation during the 2010-2011 winter season: (1) any immediate change in the level of an outcome (immediate effect) and (2) any change of an outcome going forward over time (change in slope).²¹ Covariates in the regression models included patient age, sex, race, ethnicity, and insurance type, as well as presence of an underlying complex chronic condition, mechanical ventilation use, and pediatric intensive care unit (PICU) admission during hospitalization. Data were analyzed in STATA 15 (StataCorp LLC).²²

A total of 7,116 patients met inclusion criteria over the study period (2,061 pre-implementation, 5,055 post-implementation). A comparison of patient characteristics before and after HOT protocol implementation is presented in Table 1. Patients were similar in terms of age, sex, and insurance type. Patients in the postimplementation period were more likely to have a complex chronic condition, require admission to the PICU, and need mechanical ventilation (P < .01). Differences between cohorts with regard to race/ethnicity distribution largely were a result of improved capture of these data elements in the postimplementation period. For example, 30% of patients were classified as “race/ethnicity unknown” in the preimplementation cohort, compared with 4% of patients in the postimplementation period.

Process Measures

Figure 1 shows trends in OU and HOT use by winter season. The percentage of patients discharged from the OU increased immediately after OU-HOT protocol implementation (absolute 26.9% immediate increase; 95% CI, 21.9-42.2). The change in the proportion of OU use per season also increased (change in slope +3.9% per season; 95% CI, 3.4%-4.4%). The percentage of patients discharged with HOT increased immediately after OU-HOT protocol implementation (26.0% immediate change; 95% CI, 18.9%-33.1%); however, the immediate increase in HOT discharges was coupled with a declining rate of HOT discharges per season in the postprotocol period compared with the preprotocol period (change in slope –4.5% per season; 95% CI, –7.5% to –1.5%). Our chart review and EMR flag included 1,354 patients, or 19.0% of our cohort. Our EMR flag for HOT in the last two seasons of the study had a positive predictive value of 100% (5 of 5 identified by EMR flag as receiving HOT were confirmed by chart review) and negative predictive value of 89% (31 of 35 identified by EMR flag as not receiving HOT were confirmed by chart review). The specificity of the EMR flag was 100% (31 of 31 of those confirmed by chart review as not receiving HOT, who were correctly identified by EMR) and the sensitivity was 55% (5 of 9 of those confirmed by chart review as receiving HOT, who were correctly identified by EMR).

Primary and Secondary Outcomes

Trends in length of stay across winter seasons are presented in Figure 2. The OU-HOT protocol was associated with an immediate reduction of 30.6 hours in mean length of stay (95% CI, –37.1 to –24.2). The rate of change in length of stay postimplementation did not differ significantly from the rate of change preimplementation (change in slope –0.6 hours per season; 95% CI, –2.3 to 1.1 hours). The percentage of patients discharged within 24 hours of admission rose immediately after protocol implementation, by 23.8 absolute percentage points (95% CI, 11.7-28.8). Slopes of the preintervention and postintervention regression lines did not differ significantly (change in slope –0.1% per season; 95% CI, –1.4% to 1.1%). Immediate decreases in length of stay were accompanied by an immediate decrease in mean cost per episode of care (–$4,181; 95% CI, –$4,829 to –$3,533). Protocol implementation also was associated with a decreased slope in cost postimplementation (change in slope –$403 per season; 95% CI, –$543 to –$264). The total cost savings, estimated by the product of the average cost savings per episode of care and the number of bronchiolitis admissions included in the study after OU-HOT implementation, amounted to $21.1 million over the 9-year period, or $2.3 million per winter season.

Balancing Measures

We observed an immediate reduction in 7-day hospital revisits (–1.1% immediate change; 95% CI, –1.8% to –0.4%), but an increasing slope in revisits after implementation (change in slope 0.4% per season; 95% CI, 0.1%-0.8%) (Figure 3). Stratifying revisits into ED visits and readmissions revealed that the revisit findings reflected changes in ED return visits, for which there was an immediate reduction at the time of implementation (–1.0% immediate change; 95% CI, –1.6% to –0.4%), but an increasing slope postimplementation (change in slope 0.5% per season; 95% CI, 0.2-0.8). Neither an immediate intervention effect (0.0% immediate change; 95% CI, –0.5% to 0.4%) nor a change in slope (change in slope 0.0% per season; 95% CI, –0.1% to 0.1%) were observed for inpatient readmissions alone. The annual rate of bronchiolitis admissions to Primary Children’s Hospital per 10,000 children who reside in Utah decreased after implementation of the OU-HOT protocol (immediate intervention effect –6.2 admissions; 95% CI, –10.8 to –1.6; change in slope –1.8 admissions per season; 95% CI, –2.8 to –0.69).

Our OU-HOT protocol was associated with immediate improvements in care delivered to children hospitalized for bronchiolitis, including decreased length of stay and cost savings. These improvements in outcomes largely have been sustained over a 9-year period. The OU-HOT protocol also appears to be safe as evidenced by a stable rate of readmissions over the study period and only a small increase in revisits to EDs across Intermountain Healthcare facilities, which see most children in the catchment area. Our OU-HOT protocol represents a combination of two interventions: (1) the creation of an OU focused on discharge within 24 to 48 hours of admission and (2) encouragement to discharge children with HOT. We found that use of the OU and a commitment to timely discharges has been sustained in recent years, while the commitment to HOT has appeared to wane.

Earlier investigations have evaluated the efficacy of HOT in the ED setting to prevent hospital admissions, finding high levels of caregiver comfort, estimating $1,300 per patient cost savings, and reporting readmission rates of approximately 5%.^16,23-25 Our study is unique in addressing HOT among a population of patients already hospitalized with bronchiolitis. The cost reductions we observed with our OU-HOT protocol were similar to those noted in the ED-based HOT protocols. However, we recorded lower readmission rates, likely because of the additional time allotted to caregivers to better gauge illness trajectory in the inpatient setting vs the ED, as well as additional time for hospitalized patients to reach the plateau or convalescent phase of illness. The small increase in ED revisits that we measured in recent years might be related to the concurrent rise in patient acuity and complexity.

Considering that length of stay has remained low despite less commitment to HOT, our results suggest that the OU might be the more impactful of the two interventions, and these data support the use of such a unit for a subset of patients with bronchiolitis. However, it is important to note that while the EMR HOT flag demonstrated high specificity, positive predictive value, and negative predictive value, the sensitivity was low (56%). As a result, it is possible that we have underestimated HOT use in the 2017-2018 and 2018-2019 seasons, the final two years of the study. Alternatively, the discrepancy between sustained outcomes and lagging use of HOT could be explained by improved identification of patients who would experience the greatest benefit with oxygen in terms of length of stay reductions, with fewer patients discharged on HOT but greater per-patient benefit. Finally, in an era that encourages reduced monitor use and less aggressive response to transient mild desaturations,^13,26,27 it is possible that fewer patients are identified with clinically actionable hypoxemia around the time they would be otherwise discharged.

Our OU-HOT model is not unprecedented. Increasingly, other formerly inpatient indications are being successfully managed in the observation, outpatient, and home setting, such as parenteral antibiotic treatment^28,29 and chemotherapy administration.³⁰ Considering the inpatient burden of bronchiolitis, similar strategies to expedite discharge are needed. Although outpatient intravenous antibiotic and chemotherapy administration have been widely adopted, we are aware of only one other pediatric health care system in the United States (Children’s Hospital Colorado) that routinely discharges inpatients with bronchiolitis on HOT.

This study has several limitations. First, although the interrupted time-series analysis is designed to account for trends that precede an intervention and covariates that differ before and after the intervention, it is possible that important unmeasured patient factors or changes in practice patterns differed between the pre- and post-intervention cohorts. There were no major changes to the OU-HOT protocol or discharge criteria after implementation, but individual practice management of bronchiolitis during the study period likely has evolved as new evidence emerges. Second, one could postulate that the increase in discharges within 24 hours and accompanying decreases in average length of stay and cost could be achieved by hospitalizing healthier patients over time, which the presence of an OU might incentivize. To the contrary, we found that population-based bronchiolitis admission rates have declined and disease severity appears to be increased since implementation of the OU-HOT protocol. The increase in medically complex children and PICU use in our postimplementation cohort aligns with recently published data suggesting these are national trends.^3,31 Third, HOT use was estimated from a sample of the cohort using a chart review and a newly available EMR flag. A low sensitivity and a small sample for the positive predictive value are limitations of the EMR flag.

Additionally, there are almost certainly unmeasured ambulatory burdens of HOT not captured by this study. ED-based protocols have estimated that patients discharged with HOT have a median of two follow-up ambulatory visits before oxygen is discontinued³²; however, the ambulatory burden associated with discharge on HOT after a hospitalization and the extent to which demographic factors affect that burden is unknown. Furthermore, one insurance company charged $94 for a month of HOT in 2019; paying even a portion of this charge represents a nontrivial financial burden for many families, even considering inpatient cost savings. Although the decision to discharge on oxygen or remain hospitalized until the child did not need oxygen was left to the parents, their posthospitalization perspectives were not assessed in this study. Although reports indicate that families largely feel positive about HOT after discharge from an ED setting, with 90% of caregivers preferring HOT use to inpatient admission and most reporting no difficulty with home management,²³ it is uncertain whether this would also apply after inpatient hospitalization.

The OU-HOT bronchiolitis protocol was associated with decreases in inpatient length of stay and cost while appearing safe to implement. The sustained use of the OU combined with declining use of HOT suggests that the OU might be the more impactful intervention. As previously inpatient indications such as parenteral antibiotics and chemotherapy increasingly have been administered in observation and outpatient settings, bronchiolitis appears ideal for a similar strategy that allows patients to spend less time in the hospital. Studies are needed to understand the outpatient burden of HOT and the generalizability of our findings.

Bronchiolitis is the leading cause of hospitalization in infants aged <1 year in the United States.^1-3 Estimates suggest that 1.5% to 2.0% of US infants require hospitalization every year, with a median (interquartile range) length of stay of 2 days (1-4),³ incurring direct medical costs of $555 million annually.¹ Evidence suggests that few interventions, aside from supportive care, are effective for bronchiolitis.^4-7 Adherence to standardized clinical guidelines could improve outcomes and resource use by streamlining care and limiting ineffective interventions, thereby decreasing hospital length of stay, which is a major medical cost.^8-13 For this reason, many hospitals have adopted bronchiolitis guidelines, although institutional practices vary.^14,15

Two relatively unexplored methods to reduce the inpatient burden of bronchiolitis are the use of observation units (OU) and home oxygen therapy (HOT). Motivated by research demonstrating the safety and effectiveness of an emergency department (ED)–based HOT protocol,¹⁶ where 36 of 37 patients with mild hypoxemia discharged on HOT avoided hospital admission, our institution implemented an observation unit and home oxygen therapy (OU-HOT) protocol designed to return children with bronchiolitis home earlier from the hospital. In the first winter season of implementation (2010 to 2011), the OU-HOT protocol was associated with significant reductions in length of stay and substantial cost savings, without an increase in return visits to the ED or inpatient readmissions.¹⁷ The objectives of this study were to determine whether these encouraging initial findings persisted and to measure the long-term impact of the OU-HOT protocol.

We conducted a retrospective cohort study of children hospitalized with bronchiolitis at Primary Children’s Hospital, a freestanding children’s hospital in Salt Lake City, Utah. Discharge diagnosis and procedures codes, as well as laboratory, imaging, pharmacy, and supply costs, were obtained from the Intermountain Healthcare enterprise data warehouse. A crosswalk available from the Centers for Medicare and Medicaid Services was used to convert International Classification of Diseases (ICD)-10 discharge diagnosis and procedure codes to ICD-9 equivalents.¹⁸ This study was approved by the University of Utah institutional review board (00110419).

Patients

Children aged 3 to 24 months who were discharged with a diagnosis of bronchiolitis (466.xx) during winter seasons from 2007 to 2019 were included. A winter season was defined as November 1 to April 30. Both observation and inpatient encounters were included in the cohort. We excluded patients with discharge diagnosis or procedure codes indicating tracheostomy (519.0-519.09, V44.0, V55.0, 31.1, 31.21, 31.41, 31.74, 97.23), ventilator dependence (V46.1x), chronic lung disease (518.83, 770.7), or pulmonary hypertension (416.xx). Patients with both bronchiolitis and a concurrent diagnosis, such as otitis media or pneumonia, were included unless exclusion criteria were met.

Intervention and Process Measures

Our institution implemented the OU-HOT protocol at the start of the 2010-2011 winter season.¹⁷ The aim of the OU-HOT protocol was to discharge children with bronchiolitis home sooner by increasing use of both an OU, with frequent assessment of discharge readiness, and HOT to help children become ready for discharge. Similar to most OUs, admission to our unit was limited to patients who met hospital admission criteria, and had a short anticipated length of stay (<48 hours). As a self-contained 20-bed unit providing 24-hour dedicated pediatrician/pediatric emergency medicine physician and nursing coverage, the OU actively monitored patients’ discharge readiness, with a goal to facilitate patient throughput more akin to an ED rather than a traditional inpatient unit. Patients who could not be discharged from the OU within 48 hours were transferred to the inpatient unit. Although the OU existed at the time of protocol implementation, its use for patients with bronchiolitis was not actively encouraged until implementation.

Hospitalized patients—in either inpatient or observation units—were eligible for discharge on HOT if they met the following criteria: hypoxemia was the only indication for continued hospitalization, the child’s oxygen requirement was <0.5 L/min for at least 6 hours (0.8 L/min for children aged >1 year), the child’s caregiver(s) were willing to manage oxygen at home, and the child had reliable access to primary care provider follow up. We used two process measures across winter seasons: (1) the percentage of patients discharged from the OU, and (2) the percentage of patients discharged with HOT. The percentage of patients discharged on HOT was estimated by a manual chart review and an electronic medical record (EMR) HOT flag that came into existence with our hospital system’s adoption of a new EMR (2017-2019). Chart review randomly sampled patients from 2007-2017, totaling 457 patients. To estimate the reliability of this method, we calculated the sensitivity, specificity, positive predictive value, and negative predictive value of the EMR HOT flag using chart review as the gold standard.

Outcome Measures

The main outcome measure was mean hospital length of stay. Balancing measures were revisit rates (stratified into ED visits and readmissions) and annual per-population bronchiolitis admission rates. Visits were considered revisits if they occurred within 7 days of initial hospital discharge, and included visits to Primary Children’s Hospital as well as 22 other Intermountain Healthcare hospitals. Population estimates from the Utah Department of Health were used to calculate the annual population-based rate of bronchiolitis admissions to Primary Children’s Hospital.¹⁹ Annual admission rates were calculated per 10,000 children aged 3 to 24 months who resided in Utah each year of the study period, and were evaluated to determine if patients were admitted more frequently after OU-HOT implementation. Secondary outcome measures included the percentage of patients discharged within 24 hours and mean inflation-adjusted cost per episode of care (in 2019 dollars). Hospitalization costs were determined using Intermountain Healthcare’s internal cost accounting system, an activity-based method that aggregates costs of individual resources according to date of service.²⁰ Costs were adjusted to 2019 dollars and were defined as the total costs of a patient’s initial hospitalization as well as any 7-day revisit encounters.

Data Analysis

Demographic data were compared before and after OU-HOT protocol implementation using Pearson chi-square tests. Multivariable linear or logistic regression models were used to compare measures before and after OU-HOT protocol implementation via an interrupted time-series approach. The interrupted time-series analysis measured two types of changes after protocol implementation during the 2010-2011 winter season: (1) any immediate change in the level of an outcome (immediate effect) and (2) any change of an outcome going forward over time (change in slope).²¹ Covariates in the regression models included patient age, sex, race, ethnicity, and insurance type, as well as presence of an underlying complex chronic condition, mechanical ventilation use, and pediatric intensive care unit (PICU) admission during hospitalization. Data were analyzed in STATA 15 (StataCorp LLC).²²

A total of 7,116 patients met inclusion criteria over the study period (2,061 pre-implementation, 5,055 post-implementation). A comparison of patient characteristics before and after HOT protocol implementation is presented in Table 1. Patients were similar in terms of age, sex, and insurance type. Patients in the postimplementation period were more likely to have a complex chronic condition, require admission to the PICU, and need mechanical ventilation (P < .01). Differences between cohorts with regard to race/ethnicity distribution largely were a result of improved capture of these data elements in the postimplementation period. For example, 30% of patients were classified as “race/ethnicity unknown” in the preimplementation cohort, compared with 4% of patients in the postimplementation period.

Process Measures

Figure 1 shows trends in OU and HOT use by winter season. The percentage of patients discharged from the OU increased immediately after OU-HOT protocol implementation (absolute 26.9% immediate increase; 95% CI, 21.9-42.2). The change in the proportion of OU use per season also increased (change in slope +3.9% per season; 95% CI, 3.4%-4.4%). The percentage of patients discharged with HOT increased immediately after OU-HOT protocol implementation (26.0% immediate change; 95% CI, 18.9%-33.1%); however, the immediate increase in HOT discharges was coupled with a declining rate of HOT discharges per season in the postprotocol period compared with the preprotocol period (change in slope –4.5% per season; 95% CI, –7.5% to –1.5%). Our chart review and EMR flag included 1,354 patients, or 19.0% of our cohort. Our EMR flag for HOT in the last two seasons of the study had a positive predictive value of 100% (5 of 5 identified by EMR flag as receiving HOT were confirmed by chart review) and negative predictive value of 89% (31 of 35 identified by EMR flag as not receiving HOT were confirmed by chart review). The specificity of the EMR flag was 100% (31 of 31 of those confirmed by chart review as not receiving HOT, who were correctly identified by EMR) and the sensitivity was 55% (5 of 9 of those confirmed by chart review as receiving HOT, who were correctly identified by EMR).

Primary and Secondary Outcomes

Trends in length of stay across winter seasons are presented in Figure 2. The OU-HOT protocol was associated with an immediate reduction of 30.6 hours in mean length of stay (95% CI, –37.1 to –24.2). The rate of change in length of stay postimplementation did not differ significantly from the rate of change preimplementation (change in slope –0.6 hours per season; 95% CI, –2.3 to 1.1 hours). The percentage of patients discharged within 24 hours of admission rose immediately after protocol implementation, by 23.8 absolute percentage points (95% CI, 11.7-28.8). Slopes of the preintervention and postintervention regression lines did not differ significantly (change in slope –0.1% per season; 95% CI, –1.4% to 1.1%). Immediate decreases in length of stay were accompanied by an immediate decrease in mean cost per episode of care (–$4,181; 95% CI, –$4,829 to –$3,533). Protocol implementation also was associated with a decreased slope in cost postimplementation (change in slope –$403 per season; 95% CI, –$543 to –$264). The total cost savings, estimated by the product of the average cost savings per episode of care and the number of bronchiolitis admissions included in the study after OU-HOT implementation, amounted to $21.1 million over the 9-year period, or $2.3 million per winter season.

Balancing Measures

We observed an immediate reduction in 7-day hospital revisits (–1.1% immediate change; 95% CI, –1.8% to –0.4%), but an increasing slope in revisits after implementation (change in slope 0.4% per season; 95% CI, 0.1%-0.8%) (Figure 3). Stratifying revisits into ED visits and readmissions revealed that the revisit findings reflected changes in ED return visits, for which there was an immediate reduction at the time of implementation (–1.0% immediate change; 95% CI, –1.6% to –0.4%), but an increasing slope postimplementation (change in slope 0.5% per season; 95% CI, 0.2-0.8). Neither an immediate intervention effect (0.0% immediate change; 95% CI, –0.5% to 0.4%) nor a change in slope (change in slope 0.0% per season; 95% CI, –0.1% to 0.1%) were observed for inpatient readmissions alone. The annual rate of bronchiolitis admissions to Primary Children’s Hospital per 10,000 children who reside in Utah decreased after implementation of the OU-HOT protocol (immediate intervention effect –6.2 admissions; 95% CI, –10.8 to –1.6; change in slope –1.8 admissions per season; 95% CI, –2.8 to –0.69).

Our OU-HOT protocol was associated with immediate improvements in care delivered to children hospitalized for bronchiolitis, including decreased length of stay and cost savings. These improvements in outcomes largely have been sustained over a 9-year period. The OU-HOT protocol also appears to be safe as evidenced by a stable rate of readmissions over the study period and only a small increase in revisits to EDs across Intermountain Healthcare facilities, which see most children in the catchment area. Our OU-HOT protocol represents a combination of two interventions: (1) the creation of an OU focused on discharge within 24 to 48 hours of admission and (2) encouragement to discharge children with HOT. We found that use of the OU and a commitment to timely discharges has been sustained in recent years, while the commitment to HOT has appeared to wane.

Earlier investigations have evaluated the efficacy of HOT in the ED setting to prevent hospital admissions, finding high levels of caregiver comfort, estimating $1,300 per patient cost savings, and reporting readmission rates of approximately 5%.^16,23-25 Our study is unique in addressing HOT among a population of patients already hospitalized with bronchiolitis. The cost reductions we observed with our OU-HOT protocol were similar to those noted in the ED-based HOT protocols. However, we recorded lower readmission rates, likely because of the additional time allotted to caregivers to better gauge illness trajectory in the inpatient setting vs the ED, as well as additional time for hospitalized patients to reach the plateau or convalescent phase of illness. The small increase in ED revisits that we measured in recent years might be related to the concurrent rise in patient acuity and complexity.

Considering that length of stay has remained low despite less commitment to HOT, our results suggest that the OU might be the more impactful of the two interventions, and these data support the use of such a unit for a subset of patients with bronchiolitis. However, it is important to note that while the EMR HOT flag demonstrated high specificity, positive predictive value, and negative predictive value, the sensitivity was low (56%). As a result, it is possible that we have underestimated HOT use in the 2017-2018 and 2018-2019 seasons, the final two years of the study. Alternatively, the discrepancy between sustained outcomes and lagging use of HOT could be explained by improved identification of patients who would experience the greatest benefit with oxygen in terms of length of stay reductions, with fewer patients discharged on HOT but greater per-patient benefit. Finally, in an era that encourages reduced monitor use and less aggressive response to transient mild desaturations,^13,26,27 it is possible that fewer patients are identified with clinically actionable hypoxemia around the time they would be otherwise discharged.

Our OU-HOT model is not unprecedented. Increasingly, other formerly inpatient indications are being successfully managed in the observation, outpatient, and home setting, such as parenteral antibiotic treatment^28,29 and chemotherapy administration.³⁰ Considering the inpatient burden of bronchiolitis, similar strategies to expedite discharge are needed. Although outpatient intravenous antibiotic and chemotherapy administration have been widely adopted, we are aware of only one other pediatric health care system in the United States (Children’s Hospital Colorado) that routinely discharges inpatients with bronchiolitis on HOT.

This study has several limitations. First, although the interrupted time-series analysis is designed to account for trends that precede an intervention and covariates that differ before and after the intervention, it is possible that important unmeasured patient factors or changes in practice patterns differed between the pre- and post-intervention cohorts. There were no major changes to the OU-HOT protocol or discharge criteria after implementation, but individual practice management of bronchiolitis during the study period likely has evolved as new evidence emerges. Second, one could postulate that the increase in discharges within 24 hours and accompanying decreases in average length of stay and cost could be achieved by hospitalizing healthier patients over time, which the presence of an OU might incentivize. To the contrary, we found that population-based bronchiolitis admission rates have declined and disease severity appears to be increased since implementation of the OU-HOT protocol. The increase in medically complex children and PICU use in our postimplementation cohort aligns with recently published data suggesting these are national trends.^3,31 Third, HOT use was estimated from a sample of the cohort using a chart review and a newly available EMR flag. A low sensitivity and a small sample for the positive predictive value are limitations of the EMR flag.

Additionally, there are almost certainly unmeasured ambulatory burdens of HOT not captured by this study. ED-based protocols have estimated that patients discharged with HOT have a median of two follow-up ambulatory visits before oxygen is discontinued³²; however, the ambulatory burden associated with discharge on HOT after a hospitalization and the extent to which demographic factors affect that burden is unknown. Furthermore, one insurance company charged $94 for a month of HOT in 2019; paying even a portion of this charge represents a nontrivial financial burden for many families, even considering inpatient cost savings. Although the decision to discharge on oxygen or remain hospitalized until the child did not need oxygen was left to the parents, their posthospitalization perspectives were not assessed in this study. Although reports indicate that families largely feel positive about HOT after discharge from an ED setting, with 90% of caregivers preferring HOT use to inpatient admission and most reporting no difficulty with home management,²³ it is uncertain whether this would also apply after inpatient hospitalization.

The OU-HOT bronchiolitis protocol was associated with decreases in inpatient length of stay and cost while appearing safe to implement. The sustained use of the OU combined with declining use of HOT suggests that the OU might be the more impactful intervention. As previously inpatient indications such as parenteral antibiotics and chemotherapy increasingly have been administered in observation and outpatient settings, bronchiolitis appears ideal for a similar strategy that allows patients to spend less time in the hospital. Studies are needed to understand the outpatient burden of HOT and the generalizability of our findings.

Across the country, residents are bound to a set of rules from the Accreditation Council for Graduate Medical Education (ACGME) designed to mini mize fatigue, maintain quality of life, and reduce fatigue-related patient safety events. Adherence to work hours regulations is required to maintain accreditation. Among other guidelines, residents are required to work fewer than 80 hours per week on average over 4 consecutive weeks.¹ When work hour violations occur, programs risk citation, penalties, and harm to the program’s reputation.

Residents self-report their adherence to program regulations in an annual survey conducted by the ACGME.² To collect more frequent data, most training programs monitor resident work hours through self-report on an electronic tracking platform.³ These data generally are used internally to identify problems and opportunities for improvement. However, self-report approaches are subject to imperfect recall and incomplete reporting, and require time and effort to complete.⁴

The widespread adoption of electronic health records (EHRs) brings new opportunity to measure and promote adherence to work hours. EHR log data capture when users log in and out of the system, along with their location and specific actions. These data offer a compelling alternative to self-report because they are already being collected and can be analyzed almost immediately. Recent studies using EHR log data to approximate resident work hours in a pediatric hospital successfully approximated scheduled hours, but the approach was customized to their hospital’s workflows and might not generalize to other settings.⁵ Furthermore, earlier studies have not captured evening out-of-hospital work, which contributes to total work hours and is associated with physician burnout.⁶

We developed a computational method that sought to accurately capture work hours, including out-of-hospital work, which could be used as a screening tool to identify at-risk residents and rotations in near real-time. We estimated work hours, including EHR and non-EHR work, from these EHR data and compared these daily estimations to self-report. We then used a heuristic to estimate the frequency of exceeding the 80-hour workweek in a large internal medicine residency program.

The population included 82 internal medicine interns (PGY-1) and 121 residents (PGY-2 = 60, PGY-3 = 61) who rotated through University of California, San Francisco Medical Center (UCSFMC) between July 1, 2018, and June 30, 2019, on inpatient rotations. In the UCSF internal medicine residency program, interns spend an average of 5 months per year and residents spend an average of 2 months per year on inpatient rotations at UCSFMC. Scheduled inpatient rotations generally are in 1-month blocks and include general medical wards, cardiology, liver transplant, night-float, and a procedures and jeopardy rotation where interns perform procedures at UCSFMC and serve as backup for their colleagues across sites. Although expected shift duration differs by rotation, types of shifts include regular length days, call days that are not overnight (but expected duration of work is into the late evening), 28-hour overnight call (PGY-2 and PGY-3), and night-float.

Data Source

This computational method was developed at UCSFMC. This study was approved by the University of California, San Francisco institutional review board. Using the UCSF Epic Clarity database, EHR access log data were obtained, including all Epic logins/logoffs, times, and access devices. Access devices identified included medical center computers, personal computers, and mobile devices.

Trainees self-report their work hours in MedHub, a widely used electronic tracking platform for self-report of resident work hours.⁷ Data were extracted from this database for interns and residents who matched the criteria above. The self-report data were considered the gold standard for comparison, because it is the best available despite its known limitations.

We used data collected from UCSF’s physician scheduling platform, AMiON, to identify interns and residents assigned to rotations at UCSF hospitals.⁸ AMiON also was used to capture half-days of off-site scheduled clinics and teaching, which count toward the workday but would not be associated with on-campus logins.

Developing a Computational Method to Measure Work Hours

We developed a heuristic to accomplish two goals: (1) infer the duration of continuous in-hospital work hours while providing clinical care and (2) measure “out-of-hospital” work. Logins from medical center computers were considered to be “on-campus” work. Logins from personal computers were considered to be “out-of-hospital.” “Out-of-hospital” login sessions were further subdivided into “out-of-hospital work” and “out-of-hospital study” based on activity during the session; if any work activities listed in Appendix Table 1 were performed, the session was attributed to work. If only chart review was performed, the session was attributed to study and did not count towards total hours worked. Logins from mobile devices also did not count towards total hours worked.

We inferred continuous in-hospital work by linking on-campus EHR sessions from the first on-campus login until the last on-campus logoff (Figure 1).

If there was overlapping time measurement between on-campus work and personal computer logins (for example, a resident was inferred to be doing on-campus work based on frequent medical center computer logins but there were also logins from personal computers), we inferred this to indicate that a personal device had been brought on-campus and the time was only attributed to on-campus work and was not double counted as out-of-hospital work. Out-of-hospital work that did not overlap with inferred on-campus work time contributed to the total hours worked in a week, consistent with ACGME guidelines.

Our internal medicine residents work at three hospitals: UCSFMC and two affiliated teaching hospitals. Although this study measured work hours while the residents were on an inpatient rotation at UCSFMC, trainees also might have occasional half-day clinics or teaching activities at other sites not captured by these EHR log data. The allocated time for that scheduled activity (extracted from AMiON) was counted as work hours. If the trainee was assigned to a morning half-day of off-site work (eg, didactics), this was counted the same as an 8 am to noon on-campus EHR session. If a trainee was assigned an afternoon half-day of off-site work (eg, a non-UCSF clinic), this was counted the same as a 1 pm to 5 pm on-campus EHR session. Counting this scheduled time as an on-campus EHR session allowed half-days of off-site work to be linked with inferred in-hospital work.

Comparison of EHR-Derived Work Hours Heuristic to Self-Report

Because resident adherence with daily self-report is imperfect, we compared EHR-derived work to self-report on days when both were available. We generated scatter plots of EHR-derived work hours compared with self-report and calculated the mean absolute error of estimation. We fit a linear mixed-effect model for each PGY, modeling self-reported hours as a linear function of estimated hours (fixed effect) with a random intercept (random effect) for each trainee to account for variations among individuals. StatsModels, version 0.11.1, was used for statistical analyses.⁹

We reviewed detailed data from outlier clusters to understand situations where the heuristic might not perform optimally. To assess whether EHR-derived work hours reasonably overlapped with expected shifts, 20 8-day blocks from separate interns and residents were randomly selected for qualitative detail review in comparison with AMiON schedule data.

Estimating Hours Worked and Work Hours Violations

After validating against self-report on a daily basis, we used our heuristic to infer the average rate at which the 80-hour workweek was exceeded across all inpatient rotations at UCSFMC. This was determined both including “out-of-hospital” work as derived from logins on personal computers and excluding it. Using the estimated daily hours worked, we built a near real-time dashboard to assist program leadership with identifying at-risk trainees and trends across the program.

Data from 82 interns (PGY-1) and 121 internal medicine residents (PGY-2 and PGY-3) who rotated at UCSFMC between July 1, 2018, and June 30, 2019, were included in the study. Table 1 shows the number of days and rotations worked at UCSFMC as well as the frequency of self-report of work hours according to program year.

Qualitative review of EHR-derived data compared with schedule data showed that, although residents often reported homogenous daily work hours, EHR-derived work hours often varied as expected on a day-to-day basis according to the schedule (Appendix Table 2).

Because out-of-hospital EHR use does not count as work if done for educational purposes, we evaluated the proportion of out-of-hospital EHR use that is considered work and found that 67% of PGY-1, 50% of PGY-2, and 53% of PGY-3 out-of-hospital sessions included at least one work activity, as denoted in Appendix Table 1. Out-of-hospital work therefore represented 85% of PGY-1, 66% of PGY-2, and 73% of PGY-3 time spent in the EHR out-of-hospital. These sessions were counted towards work hours in accordance with ACGME rules and included 29% of PGY-1 workdays and 21% of PGY-2 and PGY-3 workdays. This amounted to a median of 1.0 hours per day (95% CI, 0.1-4.6 hours) of out-of-hospital work for PGY-1, 0.9 hours per day (95% CI, 0.1-4.1 hours) for PGY-2, and 0.8 hours per day (95% CI, 0.1-4.7 hours) for PGY-3 residents. Out-of-hospital logins that did not include work activities, as denoted in Appendix Table 1, were labeled out-of-hospital study and did not count towards work hours; this amounted to a median of 0.3 hours per day (95% CI, 0.02-1.6 hours) for PGY-1, 0.5 hours per day (95% CI, 0.04-0.25 hours) for PGY-2, and 0.3 hours per day (95% CI, 0.03-1.7 hours) for PGY-3. Mobile device logins also were not counted towards total work hours, with a median of 3 minutes per day for PGY-1, 6 minutes per day for PGY-2, and 5 minutes per day for PGY-3.

The percentage of rotation months where average hours worked exceeded 80 hours weekly is shown in Table 2. Inclusion of out-of-hospital work hours substantially increased the frequency at which the 80-hour workweek was exceeded. The frequency of individual residents working more than 80 hours weekly on average is shown in Appendix Figure 3. A narrow majority of PGY-1 and PGY-2 trainees and a larger majority of PGY-3 trainees never worked in excess of 80 hours per week when averaged over the course of a rotation, but several trainees did on several occasions.

Estimations from the computational method were built into a dashboard for use as screening tool by residency program directors (Appendix Figure 4).

EHR log data can be used to automate measurement of trainee work hours, providing timely data to program directors for identifying residents at risk of exceeding work hours limits. We demonstrated this by developing a data-driven approach to link on-campus logins that can be replicated in other training programs. We further demonstrated that out-of-hospital work substantially contributed to resident work hours and the frequency with which they exceed the 80-hour workweek, making it a critical component of any work hour estimation approach. Inclusive of out-of-hospital work, our computational method found that residents exceeded the 80-hour workweek 10% to 21% of the time, depending on their year in residency, with a small majority of residents never exceeding the 80-hour workweek.

Historically, most ACGME residency programs have relied on resident self-report to determine work hours.³ The validity of this method has been extensively studied and results remain mixed; in some surveys, residents admit to underreporting their hours while other validation studies, including the use of clock-in and clock-out or time-stamped parking data, align with self-report relatively well.^10-12 Regardless of the reliability of self-report, it is a cumbersome task that residents have difficulty adhering to, as shown in our study, where only slightly more than one-half of the days worked had associated self-report. By relying on resident self-report, we are adding to the burden of clerical work, which is associated with physician burnout.¹³ Furthermore, because self-report typically does not happen in real-time, it limits a program’s ability to intervene on recent or impending work-hour violations. Our computational method enabled us to build a dashboard that is updated daily and provides critical insight into resident work hours at any time, without waiting for retrospective self-report.

Our study builds on previous work by Dziorny et al using EHR log data to algorithmically measure in-hospital work.⁵ In their study, the authors isolated shifts with a login gap of 4 hours and then combined shifts according to a set of heuristics. However, their logic integrated an extensive workflow analysis of trainee shifts, which might limit generalizability.⁵ Our approach computationally derives the temporal threshold for linking EHR sessions, which in our data was 5 hours but might differ at other sites. Automated derivation of this threshold will support generalizability to other programs and sites, although programs will still need to manually account for off-site work such as didactics. In a subsequent study evaluating the 80-hour workweek, Dziorny et al evaluated shift duration and appropriate time-off between shifts and found systematic underreporting of work.¹⁴ In our study, we prioritized evaluation of the 80-hour workweek and found general alignment between self-report and EHR-derived work-hour estimates, with a tendency to underestimate at lower reported work hours and overestimate at higher reported work hours (potentially because of underreporting as illustrated by Dziorny et al). We included the important out-of-hospital logins as discrete work events because out-of-hospital work contributes to the total hours worked and to the number of workweeks that exceed the 80-hour workweek, and might contribute to burnout.¹⁵ The incidence of exceeding the 80-hour workweek increased by 7% to 8% across all residents when out-of-hospital work was included, demonstrating that tools such as ResQ (ResQ Medical) that rely primarily on geolocation data might not sufficiently capture the ways in which residents spend their time working.¹⁶

Our approach has limitations. We determined on-campus vs out-of-hospital locations based on whether the login device belonged to the medical center or was a personal computer. Consequently, if trainees exclusively used a personal computer while on-campus and never used a medical center computer, we would have captured this work done while logged into the EHR but would not have inferred on-campus work. Although nearly all trainees in our organization use medical center computers throughout the day, this might impact generalizability for programs where trainees use personal computers exclusively in the hospital. Our approach also assumes trainees will use the EHR at the beginning and end of their workdays, which could lead to underestimation of work hours in trainees who do not employ this practice. With regards to work done on personal computers, our heuristic required that at least one work activity (as denoted in Appendix Table 1) be included in the session in order for it to count as work. Although this approach allows us to exclude sessions where trainees might be reviewing charts exclusively for educational purposes, it is difficult to infer the true intent of chart review.

There might be periods of time where residents are doing in-hospital work but more than 5 hours elapsed between EHR user sessions. As we have started adapting this computational method for other residency programs, we have added logic that allows for long periods of time in the operating room to be considered part of a continuous workday. There also are limitations to assigning blocks of time to off-site clinics; clinics that are associated with after-hours work but use a different EHR would not be captured in total out-of-hospital work.

Although correlation with self-report was good, we identified clusters of inaccuracy. This likely resulted from our residency program covering three medical centers, two of which were not included in the data set. For example, if a resident had an off-site clinic that was not accounted for in AMiON, EHR-derived work hours might have been underestimated relative to self-report. Operationally leveraging an automated system for measuring work hours in the form of dashboards and other tools could provide the impetus to ensure accurate documentation of schedule anomalies.

Implementation of our EHR-derived work-hour model will allow ACGME residency programs to understand and act upon trainee work-hour violations closer to real time, as the data extraction is daily and automated. Automation will save busy residents a cumbersome task, provide more complete data than self-report, and empower residency programs to intervene quickly to support overworked trainees.

The authors thank Drs Bradley Monash, Larissa Thomas, and Rebecca Berman for providing residency program input.

Across the country, residents are bound to a set of rules from the Accreditation Council for Graduate Medical Education (ACGME) designed to mini mize fatigue, maintain quality of life, and reduce fatigue-related patient safety events. Adherence to work hours regulations is required to maintain accreditation. Among other guidelines, residents are required to work fewer than 80 hours per week on average over 4 consecutive weeks.¹ When work hour violations occur, programs risk citation, penalties, and harm to the program’s reputation.

Residents self-report their adherence to program regulations in an annual survey conducted by the ACGME.² To collect more frequent data, most training programs monitor resident work hours through self-report on an electronic tracking platform.³ These data generally are used internally to identify problems and opportunities for improvement. However, self-report approaches are subject to imperfect recall and incomplete reporting, and require time and effort to complete.⁴

The widespread adoption of electronic health records (EHRs) brings new opportunity to measure and promote adherence to work hours. EHR log data capture when users log in and out of the system, along with their location and specific actions. These data offer a compelling alternative to self-report because they are already being collected and can be analyzed almost immediately. Recent studies using EHR log data to approximate resident work hours in a pediatric hospital successfully approximated scheduled hours, but the approach was customized to their hospital’s workflows and might not generalize to other settings.⁵ Furthermore, earlier studies have not captured evening out-of-hospital work, which contributes to total work hours and is associated with physician burnout.⁶

We developed a computational method that sought to accurately capture work hours, including out-of-hospital work, which could be used as a screening tool to identify at-risk residents and rotations in near real-time. We estimated work hours, including EHR and non-EHR work, from these EHR data and compared these daily estimations to self-report. We then used a heuristic to estimate the frequency of exceeding the 80-hour workweek in a large internal medicine residency program.

The population included 82 internal medicine interns (PGY-1) and 121 residents (PGY-2 = 60, PGY-3 = 61) who rotated through University of California, San Francisco Medical Center (UCSFMC) between July 1, 2018, and June 30, 2019, on inpatient rotations. In the UCSF internal medicine residency program, interns spend an average of 5 months per year and residents spend an average of 2 months per year on inpatient rotations at UCSFMC. Scheduled inpatient rotations generally are in 1-month blocks and include general medical wards, cardiology, liver transplant, night-float, and a procedures and jeopardy rotation where interns perform procedures at UCSFMC and serve as backup for their colleagues across sites. Although expected shift duration differs by rotation, types of shifts include regular length days, call days that are not overnight (but expected duration of work is into the late evening), 28-hour overnight call (PGY-2 and PGY-3), and night-float.

Data Source

This computational method was developed at UCSFMC. This study was approved by the University of California, San Francisco institutional review board. Using the UCSF Epic Clarity database, EHR access log data were obtained, including all Epic logins/logoffs, times, and access devices. Access devices identified included medical center computers, personal computers, and mobile devices.

Trainees self-report their work hours in MedHub, a widely used electronic tracking platform for self-report of resident work hours.⁷ Data were extracted from this database for interns and residents who matched the criteria above. The self-report data were considered the gold standard for comparison, because it is the best available despite its known limitations.

We used data collected from UCSF’s physician scheduling platform, AMiON, to identify interns and residents assigned to rotations at UCSF hospitals.⁸ AMiON also was used to capture half-days of off-site scheduled clinics and teaching, which count toward the workday but would not be associated with on-campus logins.

Developing a Computational Method to Measure Work Hours

We developed a heuristic to accomplish two goals: (1) infer the duration of continuous in-hospital work hours while providing clinical care and (2) measure “out-of-hospital” work. Logins from medical center computers were considered to be “on-campus” work. Logins from personal computers were considered to be “out-of-hospital.” “Out-of-hospital” login sessions were further subdivided into “out-of-hospital work” and “out-of-hospital study” based on activity during the session; if any work activities listed in Appendix Table 1 were performed, the session was attributed to work. If only chart review was performed, the session was attributed to study and did not count towards total hours worked. Logins from mobile devices also did not count towards total hours worked.

We inferred continuous in-hospital work by linking on-campus EHR sessions from the first on-campus login until the last on-campus logoff (Figure 1).

If there was overlapping time measurement between on-campus work and personal computer logins (for example, a resident was inferred to be doing on-campus work based on frequent medical center computer logins but there were also logins from personal computers), we inferred this to indicate that a personal device had been brought on-campus and the time was only attributed to on-campus work and was not double counted as out-of-hospital work. Out-of-hospital work that did not overlap with inferred on-campus work time contributed to the total hours worked in a week, consistent with ACGME guidelines.

Our internal medicine residents work at three hospitals: UCSFMC and two affiliated teaching hospitals. Although this study measured work hours while the residents were on an inpatient rotation at UCSFMC, trainees also might have occasional half-day clinics or teaching activities at other sites not captured by these EHR log data. The allocated time for that scheduled activity (extracted from AMiON) was counted as work hours. If the trainee was assigned to a morning half-day of off-site work (eg, didactics), this was counted the same as an 8 am to noon on-campus EHR session. If a trainee was assigned an afternoon half-day of off-site work (eg, a non-UCSF clinic), this was counted the same as a 1 pm to 5 pm on-campus EHR session. Counting this scheduled time as an on-campus EHR session allowed half-days of off-site work to be linked with inferred in-hospital work.

Comparison of EHR-Derived Work Hours Heuristic to Self-Report

Because resident adherence with daily self-report is imperfect, we compared EHR-derived work to self-report on days when both were available. We generated scatter plots of EHR-derived work hours compared with self-report and calculated the mean absolute error of estimation. We fit a linear mixed-effect model for each PGY, modeling self-reported hours as a linear function of estimated hours (fixed effect) with a random intercept (random effect) for each trainee to account for variations among individuals. StatsModels, version 0.11.1, was used for statistical analyses.⁹

We reviewed detailed data from outlier clusters to understand situations where the heuristic might not perform optimally. To assess whether EHR-derived work hours reasonably overlapped with expected shifts, 20 8-day blocks from separate interns and residents were randomly selected for qualitative detail review in comparison with AMiON schedule data.

Estimating Hours Worked and Work Hours Violations

After validating against self-report on a daily basis, we used our heuristic to infer the average rate at which the 80-hour workweek was exceeded across all inpatient rotations at UCSFMC. This was determined both including “out-of-hospital” work as derived from logins on personal computers and excluding it. Using the estimated daily hours worked, we built a near real-time dashboard to assist program leadership with identifying at-risk trainees and trends across the program.

Data from 82 interns (PGY-1) and 121 internal medicine residents (PGY-2 and PGY-3) who rotated at UCSFMC between July 1, 2018, and June 30, 2019, were included in the study. Table 1 shows the number of days and rotations worked at UCSFMC as well as the frequency of self-report of work hours according to program year.

Qualitative review of EHR-derived data compared with schedule data showed that, although residents often reported homogenous daily work hours, EHR-derived work hours often varied as expected on a day-to-day basis according to the schedule (Appendix Table 2).

Because out-of-hospital EHR use does not count as work if done for educational purposes, we evaluated the proportion of out-of-hospital EHR use that is considered work and found that 67% of PGY-1, 50% of PGY-2, and 53% of PGY-3 out-of-hospital sessions included at least one work activity, as denoted in Appendix Table 1. Out-of-hospital work therefore represented 85% of PGY-1, 66% of PGY-2, and 73% of PGY-3 time spent in the EHR out-of-hospital. These sessions were counted towards work hours in accordance with ACGME rules and included 29% of PGY-1 workdays and 21% of PGY-2 and PGY-3 workdays. This amounted to a median of 1.0 hours per day (95% CI, 0.1-4.6 hours) of out-of-hospital work for PGY-1, 0.9 hours per day (95% CI, 0.1-4.1 hours) for PGY-2, and 0.8 hours per day (95% CI, 0.1-4.7 hours) for PGY-3 residents. Out-of-hospital logins that did not include work activities, as denoted in Appendix Table 1, were labeled out-of-hospital study and did not count towards work hours; this amounted to a median of 0.3 hours per day (95% CI, 0.02-1.6 hours) for PGY-1, 0.5 hours per day (95% CI, 0.04-0.25 hours) for PGY-2, and 0.3 hours per day (95% CI, 0.03-1.7 hours) for PGY-3. Mobile device logins also were not counted towards total work hours, with a median of 3 minutes per day for PGY-1, 6 minutes per day for PGY-2, and 5 minutes per day for PGY-3.

The percentage of rotation months where average hours worked exceeded 80 hours weekly is shown in Table 2. Inclusion of out-of-hospital work hours substantially increased the frequency at which the 80-hour workweek was exceeded. The frequency of individual residents working more than 80 hours weekly on average is shown in Appendix Figure 3. A narrow majority of PGY-1 and PGY-2 trainees and a larger majority of PGY-3 trainees never worked in excess of 80 hours per week when averaged over the course of a rotation, but several trainees did on several occasions.

Estimations from the computational method were built into a dashboard for use as screening tool by residency program directors (Appendix Figure 4).

EHR log data can be used to automate measurement of trainee work hours, providing timely data to program directors for identifying residents at risk of exceeding work hours limits. We demonstrated this by developing a data-driven approach to link on-campus logins that can be replicated in other training programs. We further demonstrated that out-of-hospital work substantially contributed to resident work hours and the frequency with which they exceed the 80-hour workweek, making it a critical component of any work hour estimation approach. Inclusive of out-of-hospital work, our computational method found that residents exceeded the 80-hour workweek 10% to 21% of the time, depending on their year in residency, with a small majority of residents never exceeding the 80-hour workweek.

Historically, most ACGME residency programs have relied on resident self-report to determine work hours.³ The validity of this method has been extensively studied and results remain mixed; in some surveys, residents admit to underreporting their hours while other validation studies, including the use of clock-in and clock-out or time-stamped parking data, align with self-report relatively well.^10-12 Regardless of the reliability of self-report, it is a cumbersome task that residents have difficulty adhering to, as shown in our study, where only slightly more than one-half of the days worked had associated self-report. By relying on resident self-report, we are adding to the burden of clerical work, which is associated with physician burnout.¹³ Furthermore, because self-report typically does not happen in real-time, it limits a program’s ability to intervene on recent or impending work-hour violations. Our computational method enabled us to build a dashboard that is updated daily and provides critical insight into resident work hours at any time, without waiting for retrospective self-report.

Our study builds on previous work by Dziorny et al using EHR log data to algorithmically measure in-hospital work.⁵ In their study, the authors isolated shifts with a login gap of 4 hours and then combined shifts according to a set of heuristics. However, their logic integrated an extensive workflow analysis of trainee shifts, which might limit generalizability.⁵ Our approach computationally derives the temporal threshold for linking EHR sessions, which in our data was 5 hours but might differ at other sites. Automated derivation of this threshold will support generalizability to other programs and sites, although programs will still need to manually account for off-site work such as didactics. In a subsequent study evaluating the 80-hour workweek, Dziorny et al evaluated shift duration and appropriate time-off between shifts and found systematic underreporting of work.¹⁴ In our study, we prioritized evaluation of the 80-hour workweek and found general alignment between self-report and EHR-derived work-hour estimates, with a tendency to underestimate at lower reported work hours and overestimate at higher reported work hours (potentially because of underreporting as illustrated by Dziorny et al). We included the important out-of-hospital logins as discrete work events because out-of-hospital work contributes to the total hours worked and to the number of workweeks that exceed the 80-hour workweek, and might contribute to burnout.¹⁵ The incidence of exceeding the 80-hour workweek increased by 7% to 8% across all residents when out-of-hospital work was included, demonstrating that tools such as ResQ (ResQ Medical) that rely primarily on geolocation data might not sufficiently capture the ways in which residents spend their time working.¹⁶

Our approach has limitations. We determined on-campus vs out-of-hospital locations based on whether the login device belonged to the medical center or was a personal computer. Consequently, if trainees exclusively used a personal computer while on-campus and never used a medical center computer, we would have captured this work done while logged into the EHR but would not have inferred on-campus work. Although nearly all trainees in our organization use medical center computers throughout the day, this might impact generalizability for programs where trainees use personal computers exclusively in the hospital. Our approach also assumes trainees will use the EHR at the beginning and end of their workdays, which could lead to underestimation of work hours in trainees who do not employ this practice. With regards to work done on personal computers, our heuristic required that at least one work activity (as denoted in Appendix Table 1) be included in the session in order for it to count as work. Although this approach allows us to exclude sessions where trainees might be reviewing charts exclusively for educational purposes, it is difficult to infer the true intent of chart review.

There might be periods of time where residents are doing in-hospital work but more than 5 hours elapsed between EHR user sessions. As we have started adapting this computational method for other residency programs, we have added logic that allows for long periods of time in the operating room to be considered part of a continuous workday. There also are limitations to assigning blocks of time to off-site clinics; clinics that are associated with after-hours work but use a different EHR would not be captured in total out-of-hospital work.

Although correlation with self-report was good, we identified clusters of inaccuracy. This likely resulted from our residency program covering three medical centers, two of which were not included in the data set. For example, if a resident had an off-site clinic that was not accounted for in AMiON, EHR-derived work hours might have been underestimated relative to self-report. Operationally leveraging an automated system for measuring work hours in the form of dashboards and other tools could provide the impetus to ensure accurate documentation of schedule anomalies.

Implementation of our EHR-derived work-hour model will allow ACGME residency programs to understand and act upon trainee work-hour violations closer to real time, as the data extraction is daily and automated. Automation will save busy residents a cumbersome task, provide more complete data than self-report, and empower residency programs to intervene quickly to support overworked trainees.

The authors thank Drs Bradley Monash, Larissa Thomas, and Rebecca Berman for providing residency program input.

Across the country, residents are bound to a set of rules from the Accreditation Council for Graduate Medical Education (ACGME) designed to mini mize fatigue, maintain quality of life, and reduce fatigue-related patient safety events. Adherence to work hours regulations is required to maintain accreditation. Among other guidelines, residents are required to work fewer than 80 hours per week on average over 4 consecutive weeks.¹ When work hour violations occur, programs risk citation, penalties, and harm to the program’s reputation.

Residents self-report their adherence to program regulations in an annual survey conducted by the ACGME.² To collect more frequent data, most training programs monitor resident work hours through self-report on an electronic tracking platform.³ These data generally are used internally to identify problems and opportunities for improvement. However, self-report approaches are subject to imperfect recall and incomplete reporting, and require time and effort to complete.⁴

The widespread adoption of electronic health records (EHRs) brings new opportunity to measure and promote adherence to work hours. EHR log data capture when users log in and out of the system, along with their location and specific actions. These data offer a compelling alternative to self-report because they are already being collected and can be analyzed almost immediately. Recent studies using EHR log data to approximate resident work hours in a pediatric hospital successfully approximated scheduled hours, but the approach was customized to their hospital’s workflows and might not generalize to other settings.⁵ Furthermore, earlier studies have not captured evening out-of-hospital work, which contributes to total work hours and is associated with physician burnout.⁶

We developed a computational method that sought to accurately capture work hours, including out-of-hospital work, which could be used as a screening tool to identify at-risk residents and rotations in near real-time. We estimated work hours, including EHR and non-EHR work, from these EHR data and compared these daily estimations to self-report. We then used a heuristic to estimate the frequency of exceeding the 80-hour workweek in a large internal medicine residency program.

The population included 82 internal medicine interns (PGY-1) and 121 residents (PGY-2 = 60, PGY-3 = 61) who rotated through University of California, San Francisco Medical Center (UCSFMC) between July 1, 2018, and June 30, 2019, on inpatient rotations. In the UCSF internal medicine residency program, interns spend an average of 5 months per year and residents spend an average of 2 months per year on inpatient rotations at UCSFMC. Scheduled inpatient rotations generally are in 1-month blocks and include general medical wards, cardiology, liver transplant, night-float, and a procedures and jeopardy rotation where interns perform procedures at UCSFMC and serve as backup for their colleagues across sites. Although expected shift duration differs by rotation, types of shifts include regular length days, call days that are not overnight (but expected duration of work is into the late evening), 28-hour overnight call (PGY-2 and PGY-3), and night-float.

Data Source

This computational method was developed at UCSFMC. This study was approved by the University of California, San Francisco institutional review board. Using the UCSF Epic Clarity database, EHR access log data were obtained, including all Epic logins/logoffs, times, and access devices. Access devices identified included medical center computers, personal computers, and mobile devices.

Trainees self-report their work hours in MedHub, a widely used electronic tracking platform for self-report of resident work hours.⁷ Data were extracted from this database for interns and residents who matched the criteria above. The self-report data were considered the gold standard for comparison, because it is the best available despite its known limitations.

We used data collected from UCSF’s physician scheduling platform, AMiON, to identify interns and residents assigned to rotations at UCSF hospitals.⁸ AMiON also was used to capture half-days of off-site scheduled clinics and teaching, which count toward the workday but would not be associated with on-campus logins.

Developing a Computational Method to Measure Work Hours

We developed a heuristic to accomplish two goals: (1) infer the duration of continuous in-hospital work hours while providing clinical care and (2) measure “out-of-hospital” work. Logins from medical center computers were considered to be “on-campus” work. Logins from personal computers were considered to be “out-of-hospital.” “Out-of-hospital” login sessions were further subdivided into “out-of-hospital work” and “out-of-hospital study” based on activity during the session; if any work activities listed in Appendix Table 1 were performed, the session was attributed to work. If only chart review was performed, the session was attributed to study and did not count towards total hours worked. Logins from mobile devices also did not count towards total hours worked.

We inferred continuous in-hospital work by linking on-campus EHR sessions from the first on-campus login until the last on-campus logoff (Figure 1).

If there was overlapping time measurement between on-campus work and personal computer logins (for example, a resident was inferred to be doing on-campus work based on frequent medical center computer logins but there were also logins from personal computers), we inferred this to indicate that a personal device had been brought on-campus and the time was only attributed to on-campus work and was not double counted as out-of-hospital work. Out-of-hospital work that did not overlap with inferred on-campus work time contributed to the total hours worked in a week, consistent with ACGME guidelines.

Our internal medicine residents work at three hospitals: UCSFMC and two affiliated teaching hospitals. Although this study measured work hours while the residents were on an inpatient rotation at UCSFMC, trainees also might have occasional half-day clinics or teaching activities at other sites not captured by these EHR log data. The allocated time for that scheduled activity (extracted from AMiON) was counted as work hours. If the trainee was assigned to a morning half-day of off-site work (eg, didactics), this was counted the same as an 8 am to noon on-campus EHR session. If a trainee was assigned an afternoon half-day of off-site work (eg, a non-UCSF clinic), this was counted the same as a 1 pm to 5 pm on-campus EHR session. Counting this scheduled time as an on-campus EHR session allowed half-days of off-site work to be linked with inferred in-hospital work.

Comparison of EHR-Derived Work Hours Heuristic to Self-Report

Because resident adherence with daily self-report is imperfect, we compared EHR-derived work to self-report on days when both were available. We generated scatter plots of EHR-derived work hours compared with self-report and calculated the mean absolute error of estimation. We fit a linear mixed-effect model for each PGY, modeling self-reported hours as a linear function of estimated hours (fixed effect) with a random intercept (random effect) for each trainee to account for variations among individuals. StatsModels, version 0.11.1, was used for statistical analyses.⁹

We reviewed detailed data from outlier clusters to understand situations where the heuristic might not perform optimally. To assess whether EHR-derived work hours reasonably overlapped with expected shifts, 20 8-day blocks from separate interns and residents were randomly selected for qualitative detail review in comparison with AMiON schedule data.

Estimating Hours Worked and Work Hours Violations

After validating against self-report on a daily basis, we used our heuristic to infer the average rate at which the 80-hour workweek was exceeded across all inpatient rotations at UCSFMC. This was determined both including “out-of-hospital” work as derived from logins on personal computers and excluding it. Using the estimated daily hours worked, we built a near real-time dashboard to assist program leadership with identifying at-risk trainees and trends across the program.

Data from 82 interns (PGY-1) and 121 internal medicine residents (PGY-2 and PGY-3) who rotated at UCSFMC between July 1, 2018, and June 30, 2019, were included in the study. Table 1 shows the number of days and rotations worked at UCSFMC as well as the frequency of self-report of work hours according to program year.

Qualitative review of EHR-derived data compared with schedule data showed that, although residents often reported homogenous daily work hours, EHR-derived work hours often varied as expected on a day-to-day basis according to the schedule (Appendix Table 2).

Because out-of-hospital EHR use does not count as work if done for educational purposes, we evaluated the proportion of out-of-hospital EHR use that is considered work and found that 67% of PGY-1, 50% of PGY-2, and 53% of PGY-3 out-of-hospital sessions included at least one work activity, as denoted in Appendix Table 1. Out-of-hospital work therefore represented 85% of PGY-1, 66% of PGY-2, and 73% of PGY-3 time spent in the EHR out-of-hospital. These sessions were counted towards work hours in accordance with ACGME rules and included 29% of PGY-1 workdays and 21% of PGY-2 and PGY-3 workdays. This amounted to a median of 1.0 hours per day (95% CI, 0.1-4.6 hours) of out-of-hospital work for PGY-1, 0.9 hours per day (95% CI, 0.1-4.1 hours) for PGY-2, and 0.8 hours per day (95% CI, 0.1-4.7 hours) for PGY-3 residents. Out-of-hospital logins that did not include work activities, as denoted in Appendix Table 1, were labeled out-of-hospital study and did not count towards work hours; this amounted to a median of 0.3 hours per day (95% CI, 0.02-1.6 hours) for PGY-1, 0.5 hours per day (95% CI, 0.04-0.25 hours) for PGY-2, and 0.3 hours per day (95% CI, 0.03-1.7 hours) for PGY-3. Mobile device logins also were not counted towards total work hours, with a median of 3 minutes per day for PGY-1, 6 minutes per day for PGY-2, and 5 minutes per day for PGY-3.

The percentage of rotation months where average hours worked exceeded 80 hours weekly is shown in Table 2. Inclusion of out-of-hospital work hours substantially increased the frequency at which the 80-hour workweek was exceeded. The frequency of individual residents working more than 80 hours weekly on average is shown in Appendix Figure 3. A narrow majority of PGY-1 and PGY-2 trainees and a larger majority of PGY-3 trainees never worked in excess of 80 hours per week when averaged over the course of a rotation, but several trainees did on several occasions.

Estimations from the computational method were built into a dashboard for use as screening tool by residency program directors (Appendix Figure 4).

EHR log data can be used to automate measurement of trainee work hours, providing timely data to program directors for identifying residents at risk of exceeding work hours limits. We demonstrated this by developing a data-driven approach to link on-campus logins that can be replicated in other training programs. We further demonstrated that out-of-hospital work substantially contributed to resident work hours and the frequency with which they exceed the 80-hour workweek, making it a critical component of any work hour estimation approach. Inclusive of out-of-hospital work, our computational method found that residents exceeded the 80-hour workweek 10% to 21% of the time, depending on their year in residency, with a small majority of residents never exceeding the 80-hour workweek.

Historically, most ACGME residency programs have relied on resident self-report to determine work hours.³ The validity of this method has been extensively studied and results remain mixed; in some surveys, residents admit to underreporting their hours while other validation studies, including the use of clock-in and clock-out or time-stamped parking data, align with self-report relatively well.^10-12 Regardless of the reliability of self-report, it is a cumbersome task that residents have difficulty adhering to, as shown in our study, where only slightly more than one-half of the days worked had associated self-report. By relying on resident self-report, we are adding to the burden of clerical work, which is associated with physician burnout.¹³ Furthermore, because self-report typically does not happen in real-time, it limits a program’s ability to intervene on recent or impending work-hour violations. Our computational method enabled us to build a dashboard that is updated daily and provides critical insight into resident work hours at any time, without waiting for retrospective self-report.

Our study builds on previous work by Dziorny et al using EHR log data to algorithmically measure in-hospital work.⁵ In their study, the authors isolated shifts with a login gap of 4 hours and then combined shifts according to a set of heuristics. However, their logic integrated an extensive workflow analysis of trainee shifts, which might limit generalizability.⁵ Our approach computationally derives the temporal threshold for linking EHR sessions, which in our data was 5 hours but might differ at other sites. Automated derivation of this threshold will support generalizability to other programs and sites, although programs will still need to manually account for off-site work such as didactics. In a subsequent study evaluating the 80-hour workweek, Dziorny et al evaluated shift duration and appropriate time-off between shifts and found systematic underreporting of work.¹⁴ In our study, we prioritized evaluation of the 80-hour workweek and found general alignment between self-report and EHR-derived work-hour estimates, with a tendency to underestimate at lower reported work hours and overestimate at higher reported work hours (potentially because of underreporting as illustrated by Dziorny et al). We included the important out-of-hospital logins as discrete work events because out-of-hospital work contributes to the total hours worked and to the number of workweeks that exceed the 80-hour workweek, and might contribute to burnout.¹⁵ The incidence of exceeding the 80-hour workweek increased by 7% to 8% across all residents when out-of-hospital work was included, demonstrating that tools such as ResQ (ResQ Medical) that rely primarily on geolocation data might not sufficiently capture the ways in which residents spend their time working.¹⁶

Our approach has limitations. We determined on-campus vs out-of-hospital locations based on whether the login device belonged to the medical center or was a personal computer. Consequently, if trainees exclusively used a personal computer while on-campus and never used a medical center computer, we would have captured this work done while logged into the EHR but would not have inferred on-campus work. Although nearly all trainees in our organization use medical center computers throughout the day, this might impact generalizability for programs where trainees use personal computers exclusively in the hospital. Our approach also assumes trainees will use the EHR at the beginning and end of their workdays, which could lead to underestimation of work hours in trainees who do not employ this practice. With regards to work done on personal computers, our heuristic required that at least one work activity (as denoted in Appendix Table 1) be included in the session in order for it to count as work. Although this approach allows us to exclude sessions where trainees might be reviewing charts exclusively for educational purposes, it is difficult to infer the true intent of chart review.

There might be periods of time where residents are doing in-hospital work but more than 5 hours elapsed between EHR user sessions. As we have started adapting this computational method for other residency programs, we have added logic that allows for long periods of time in the operating room to be considered part of a continuous workday. There also are limitations to assigning blocks of time to off-site clinics; clinics that are associated with after-hours work but use a different EHR would not be captured in total out-of-hospital work.

Although correlation with self-report was good, we identified clusters of inaccuracy. This likely resulted from our residency program covering three medical centers, two of which were not included in the data set. For example, if a resident had an off-site clinic that was not accounted for in AMiON, EHR-derived work hours might have been underestimated relative to self-report. Operationally leveraging an automated system for measuring work hours in the form of dashboards and other tools could provide the impetus to ensure accurate documentation of schedule anomalies.

Implementation of our EHR-derived work-hour model will allow ACGME residency programs to understand and act upon trainee work-hour violations closer to real time, as the data extraction is daily and automated. Automation will save busy residents a cumbersome task, provide more complete data than self-report, and empower residency programs to intervene quickly to support overworked trainees.

The authors thank Drs Bradley Monash, Larissa Thomas, and Rebecca Berman for providing residency program input.

Effective transitions of care (TOC) are essential to ensure quality continuity of care after hospital discharge. About 20 to 30% of patients experience an adverse event (AE) in the peridischarge period when discharged to the community.¹ Additionally, about two-thirds of AEs are preventable.¹ The Joint Commission has identified various breakdowns in care that are associated with poor outcomes, including a lack of standardized discharge procedures, limited time dedicated to discharge planning and processes, and patients who lack the necessary resources or skills to implement discharge care plans.²

Background

The most impactful TOC programs are those that target patients who are at high risk for readmission or adverse outcomes.³ Factors such as advanced age, polypharmacy, cognitive impairment, and lack of social support are patient characteristics that have been associated with unfavorable outcomes after discharge.⁴ To identify this subset of high-risk individuals, various risk assessment scores have been developed, ranging from those that are used locally at the facility level to those that are nationally validated. The LACE score (Length of hospital stay; Acuity of the admission; Comorbidities measured with the Charlson comorbidity index score; and Emergency department visits within the past 6 months) is a validated index scoring tool that is used to identify medical and surgical patients at risk for readmission or death within 30 days of hospital discharge. On a 19-point scale, a score of ≥ 10 is considered high risk.⁵ Specific to the US Department of Veterans Affairs (VA), the Care Assessment Needs (CAN) score was developed to risk stratify the veteran population. The CAN score is generated using information including patient demographics, medical conditions, VA health care utilization, vital signs, laboratory values, medications, and socioeconomic status. This score is expressed as a percentile that compares the probability of death or admission among veterans at 90 days and 1 year postdischarge. Veterans in the 99th percentile have a 74% risk for these adverse outcomes at 1 year.⁶

The Joint Commission states that a fundamental component to assuring safe and effective TOC is medication management, which includes the involvement of pharmacists.² TOC programs with pharmacist involvement have shown significant improvements related to reduced 30-day hospital readmissions and health care costs in addition to significant medication-related interventions.^7-9 While this body of evidence continues to grow and demonstrates that pharmacists are an integral component of the TOC process, there is no gold standard program. Brantley and colleagues noted that a weakness of many TOC programs is that they are one dimensional, meaning that they focus on only 1 element of care transitions or 1 specific patient population or disease.¹⁰

There is well-supported evidence of high-impact interventions for pharmacists involved early in the admission process, but data are less robust on the discharge process. ^11,12 Therefore, the primary focus of this project was to develop a pharmacist-based TOC program and implement a process for communicating high-risk patients who are discharging from our hospital across the continuum of care.

Setting

The Richard L. Roudebush VA Medical Center (RLRVAMC) is a tertiary care referral center for veterans in Indiana and eastern Illinois. Acute care clinical pharmacists are fully integrated into the acute care teams and practice under a comprehensive care model. Pharmacists attend daily patient care rounds and conduct discharge medication reconciliation for all patients with additional bedside counseling for patients who are being discharged home.

Primary care services are provided by patient aligned care teams (PACTs), multidisciplinary teams composed of physicians, advanced practice nurses, pharmacists, mental health care providers, registered nurses, dieticians, and care coordinators. Ambulatory Care or PACT clinical pharmacists are established within each RLRVAMC PACT clinic and provide comprehensive care management through an independent scope of practice for several chronic diseases, including hypertension, type 2 diabetes mellitus (T2DM), dyslipidemia, hypothyroidism, and tobacco cessation. Prior to this project implementation, there was no formalized or standardized method for facilitating routine communication of patients between acute care and PACT pharmacists in the TOC process.

Pilot Study

In 2017, RLRVAMC implemented a TOC pharmacy program pilot. A pharmacy resident and both acute care and PACT clinical pharmacy specialists (CPSs) developed the service. The pilot program was conducted from September 1, 2017 to March 1, 2018. The initial phase consisted of the development of an electronic TOC tool to standardize communication between acute care and PACT pharmacists. The TOC tool was created on a secure site accessible only to pharmacy personnel and not part of the formal medical record. (Figure 1).

The acute care pharmacist identified high-risk patients through calculated CAN and LACE scores during the discharge process and offered PACT pharmacist follow-up to the patient during bedside discharge counseling. Information was then entered into the TOC tool, including patient identifiers and a message with specific information outlining the reason for referral. PACT pharmacists routinely reviewed the tool and attempted to phone each patient within 7 days of discharge. Follow-up included medication reconciliation and chronic disease management as warranted at the discretion of the PACT pharmacist. All postdischarge follow-up appointments were created and documented in the electronic health record. A retrospective chart review was completed on patients who were entered into the TOC tool.

Patients were eligible for referral if they were discharged during the study period with primary care established in one of the facility’s PACT clinics. Additionally, patients had to meet ≥ 1 of the following criteria, deeming them a high risk for readmission: LACE score ≥ 10, CAN score ≥ 90th percentile, or be considered high risk based on the discretion of the acute care pharmacist. Patients were included in the analysis if they met the CAN or LACE score requirement. Patients were excluded if they received primary care from a site other than a RLRVAMC PACT clinic. This included non-VA primary care, home-based primary care, or VA community-based outpatient clinics (CBOCs). Patients also were excluded if they required further institutional care postdischarge (ie, subacute rehabilitation, extended care facility, etc), discharged to hospice, or against medical advice.

The average referral rate per month during the pilot study was 19 patients, with 113 total referrals during the 6-month study period. Lower rates of index emergency department (ED) visits (5.3% vs 23.3%) and readmissions (1% vs 6.7%) were seen in the group of patients who received PACT pharmacist follow-up postdischarge compared with those who did not. Additionally, PACT pharmacists were able to make > 120 interventions, averaging 1.7 interventions per patient. Of note, these results were not statistically analyzed and were assessed as observational data to determine whether the program had the potential to be impactful. The results of the pilot study demonstrated positive outcomes associated with having a pharmacist-based TOC process and led to the desire for further development and implementation of the TOC program at the RLRVAMC. These positive results prompted a second phase project to address barriers, make improvements, and ensure sustainability.

Methods

Phase 2 was a quality improvement initiative; therefore, institutional review board approval was not needed. The aim of phase 2 was to improve, expand, and sustain the TOC program that was implemented in the pilot study. Barriers identified after discussion with acute care and PACT pharmacists included difficulty in making referrals due to required entry of cumbersome readmission risk factor calculations, limiting inclusion to patients who receive primary care at the main hospital facility, and the expansion of pharmacy staff with new pharmacists who were not knowledgeable of the referral process.

Design

To overcome barriers, 4 main targeted interventions were needed: streamlining the referral process, enhancing pharmacy staff education, updating the discharge note template, and expanding the criteria to include patients who receive care at VA CBOCs. The referral process was streamlined by removing required calculated readmission risk scores, allowing pharmacist judgement to take precedence for referrals. Focused face-to-face education was provided to acute care and PACT pharmacists about the referral process and inclusion criteria to increase awareness and provide guidance of who may benefit from entry into the tool. Unlike the first phase of the study, education was provided for outpatient staff pharmacists responsible for discharging patients on the weekends. Additionally, the pharmacists received a printed quick reference guide of the information covered during the education sessions (Figure 2). Referral prompts were embedded into the standard pharmacy discharge note template to serve as a reminder to discharging pharmacists to assess patients for inclusion into the tool and provided a direct link to the tool. Expansion to include VA CBOCs occurred postpilot study, allowing increased patient access to this TOC service. All other aspects of the program were continued from the pilot phase.

Patients were eligible if they were discharged from RLRVAMC between October 1, 2018 and February 28, 2019. Additionally, the patient had to be established in a PACT clinic for primary care and have been referred to the tool based on the discretion of an acute care pharmacist. Patients were excluded if they were discharged against medical advice or to any facility where the patient and/or caregiver would not be responsible for medication administration (eg, subacute rehabilitation, extended care facility), or if the patient refused pharmacy follow-up.

Outcomes

The primary outcomes assessed were all-cause and index ED visits and readmissions within 30 days of discharge. All-cause ED visits and readmissions were defined as a second visit to RLRVAMC , regardless of readmission diagnosis. Index ED visits and readmissions were defined as those that were related to the initial admission diagnosis. Additional data collected and analyzed included the number of patients referred by pharmacists, number and type of medication discrepancies, medication changes, counseling interventions, time to follow-up postdischarge, and number of patients added to the PACT pharmacist’s clinic schedule for further management. A discrepancy identified by a PACT pharmacist was defined as a difference between the discharge medication list and the patient-reported medication list at the time of follow-up. Patients who were referred to the TOC tool but were unable to be reached by telephone served as the control group for this study.

Data Collection

A retrospective chart review was completed on patients entered into the tool. Data were collected and kept in a secured Microsoft Excel workbook. Baseline characteristics were analyzed using either a χ² for nominal data or Student t test for continuous data. The primary outcomes were analyzed using a χ² test. All statistical tests were analyzed using MiniTab 19 Statistical Software.

Results

Pharmacists added 172 patients into the TOC tool; 139 patients met inclusion criteria. Of those excluded, most were because the PACT pharmacist did not attempt to contact the patient since they already had a primary care visit scheduled postdischarge (Table 1). Of the 139 patients who met the inclusion criteria, 99 were successfully contacted by a PACT pharmacist. Most patients were aged in their 60s, male, and white. Both groups had a similar quantity of outpatient medications on admission and medication changes made at discharge. Additionally, both groups had a similar number of patients with hospitalizations and/or ED visits in the 3 months before hospital admission that resulted in TOC tool referral (Table 2).

Hospital Readmission

Hospital 30-day readmission rates for patients who were successfully followed by pharmacy compared with those who were not were 5.1% vs 15.0% (P = .049) for index readmissions and 8.1% vs 27.5% (P = .03) for all-cause readmissions. No statistically significant difference existed between those patients with follow-up compared with those without follow-up for either index (10.1% vs 12.5%, respectively; P = .68) or for all-cause ED visit rates (15.2% vs 20.0%, respectively; P = .49).

Patient Encounters

The average time to follow-up was 8.8 days, which was above the predetermined goal of contact within 7 days. Additionally, this was a decline from the initial pilot study, which had an average time to reach of 4.7 days. All patients reached by a pharmacist received medication reconciliation, with ≥ 28% of patients having ≥ 1 discrepancy. There were 43 discrepancies among all patients. Of the discrepancies, 25 were reported as errors performed by the patient, and 18 were from an error during the discharge process. The discrepancies that resulted from patient error were primarily patients who took the wrong dose of prescribed medications. Other patient discrepancies included taking medications not as scheduled, omitting medications (both intentionally and mistakenly), continuing to take medications that had been discontinued by a health care provider and improper administration technique. Examples of provider errors that occurred during the discharge process included not ordering medications for patient to pick up at discharge, not discontinuing a medication from the patient’s profile, and failure to renew expired prescriptions.

Additional counseling was provided to 75% of patients: The most common reason for counseling was T2DM, hypertension, and dyslipidemia management. PACT pharmacists changed medication regimens for 27.3% of patients for improved control of chronic diseases or relief of medication AEs.

At the end of each visit, patients were assessed to determine whether they could benefit from additional pharmacy follow-up. Thirty-seven patients were added to the pharmacist schedules for disease management appointments. The most common conditions for these appointments were T2DM, hypertension, tobacco cessation, and hyperlipidemia. Among the 37 patients who had pharmacy follow-up, there were 137 additional pharmacy appointments within the study period.

Program Referrals

After expansion to include the VA CBOCs, elimination of the elevated LACE or CAN score requirement, and additional staff education, the rate of referrals per month increased during phase 2 in comparison to the pilot study (Figure 3). There were a mean (SD) of 34 (10) referrals per month. Although not statistically analyzed, it is an objective increase in comparison to a mean 19 referrals per month in the pilot study.

Discussion

The continued development and use of a pharmacist-driven TOC tool at RLRVAMC increased communication and follow-up of high-risk patients, demonstrated the ability of pharmacists to identify and intervene in medication-related issues postdischarge, and successfully reduce 30-day readmissions. This program emphasized pharmacist involvement during the discharge process and created a standardized mechanism for TOC follow-up, addressing multiple areas that were identified by The Joint Commission as being associated with poor outcomes. The advanced pharmacy practice model at RLRVAMC allowed for a multidimensional program, including prospective patient identification and multiple pharmacy touchpoints. This is unique in comparison to many of the one-dimensional programs described in the literature.

Polypharmacy has been identified as a major predictor of medication discrepancies postdischarge, and patients with ≥ 10 active medications have been found to be at highest risk.^13,14 Patients in this study had a mean 13 active medications on admission, with a mean 5 medication changes at discharge. PACT pharmacists documented 28 of 99 patients with ≥ 1 medication-related discrepancy at postdischarge reconciliation. This 28% discrepancy rate is consistent with discrepancy rates previously reported in the literature, which ranged from 14 to 45% in large meta-analyses.^14,15 The majority of these discrepancies (58%) were related to patients who took the wrong dose of a prescribed medication.

Targeted interventions to overcome barriers in the pilot study increased the referral rates to the TOC tool; however, the increase in referral rate was associated with increased time to follow up by ambulatory care pharmacists. The extended follow-up times were seen most often in the 2 busiest primary care clinics, one of which is considered a teaching clinic for medical residents. Pharmacists were required to integrate these calls into their normal work schedule and were not provided additional time for calling, allowing for an increased follow-up time. The increased follow-up time likely contributed to the increased number of patients excluded due to already having PACT follow-up, giving more time for the primary care provider to have an appointment with the patient. The ambulatory care pharmacist could then determine whether further intervention was needed. In the summer of 2018, a decrease in referral rates occurred for a short time, but this is likely explained by incoming new residents and staff within the pharmacy department and decreased awareness among the new staff. The enhanced staff education took place during September 2018 and lead to increased referral rates compared with those seen in months prior.

PACT pharmacists were not only able to identify discrepancies, but also provide timely intervention on a multitude of medication-related issues by using their scope of practice (SOP). Most interventions were related to medication or disease counseling, including lifestyle, device, and disease education. The independent SOP of our PACT pharmacists is a unique aspect of this program and allowed pharmacists to independently adjust many aspects of a patient’s medication regimen during follow-up visits.

The outcomes of 30-day index and all-cause readmissions, as well as index and all-cause ED visit rates, were lower in the subset of patients who received PACT pharmacist follow-up after discharge (Table 3). The difference was most pronounced in the all-cause readmission rates: Only 8.1% of patients who received PACT follow-up experienced a readmission compared with 27.5% of those who did not. The difference between the groups regarding ED visit rates were not as pronounced, but this may be attributed to a limited sample size. These data indicate that the role of the pharmacist in identifying discrepancies and performing interventions at follow-up may play a clinically significant part in reducing both ED visit rates and hospital readmissions.

Limitations

There are some limitations identified within this study. Although the referral criteria were relaxed from the pilot study and enhanced education was created, continued education regarding appropriate referral of TOC patients continues to be necessary given intermittent staff changeover, incorporation of pharmacy trainees, and modifications to clinic workflow. Patients who were discharged to facilities were not included. This ensured that appropriate and consistent PACT pharmacist follow-up would be available, but likely reduced our sample size.

Although performing this study in a closed health care system with pharmacists who have independent SOPs is a strength of our study, also it can limit generalizability. Not all facilities house both acute care and ambulatory care in one location with wide SOPs to allow for comprehensive and continued care. Last, this study used convenience sampling, potentially introducing selection bias, as patients unable to be reached by PACT pharmacists may inherently be at increased risk for hospital readmission. However, in the 3 months preceding the hospital admission that resulted in TOC tool referral, both groups had a similar number of patients with hospital admissions and ED visits.

The TOC tool has become fully integrated into the daily workflow for both acute care and PACT pharmacists. After the conclusion of the study period, the referral rates into the tool have been maintained at a steady level, even surpassing the rates seen during the study period. In comparison with the pilot study, PACT pharmacists reported a subjective increase in referrals placed for procedures such as medication reconciliation or adherence checks. This is likely because acute care pharmacists were able to use their clinical judgement rather than to rely solely on calculated readmission risk scores for TOC tool referral.

The success of the TOC program led to the expansion to other specialty areas. ED pharmacists now refer patients from the ED who were not admitted to the hospital but would benefit from PACT follow-up. Additionally, the option to refer hematology and oncology patients was added to allow these patients to be followed up by our hematology/oncology CPSs by phone appointments. Unique reasons for follow-up for this patient population include concerns about delayed chemotherapy cycles or chemotherapy-associated AEs.

Conclusions

This study outlines the creation and continued improvement of a pharmacist-based TOC program. The program was designed as a method of communication between acute care and PACT pharmacists about high-risk patients. The creation of this program allowed PACT pharmacists not only to identify discrepancies and make interventions on high-risk patients, but also demonstrate that having pharmacists involved in these programs may have a positive impact on readmissions and ED visits. The success of the TOC tool at the RLRVAMC has led to its expansion and is now an integral part of the daily workflow for both acute care and PACT pharmacists.

Effective transitions of care (TOC) are essential to ensure quality continuity of care after hospital discharge. About 20 to 30% of patients experience an adverse event (AE) in the peridischarge period when discharged to the community.¹ Additionally, about two-thirds of AEs are preventable.¹ The Joint Commission has identified various breakdowns in care that are associated with poor outcomes, including a lack of standardized discharge procedures, limited time dedicated to discharge planning and processes, and patients who lack the necessary resources or skills to implement discharge care plans.²

Background

The most impactful TOC programs are those that target patients who are at high risk for readmission or adverse outcomes.³ Factors such as advanced age, polypharmacy, cognitive impairment, and lack of social support are patient characteristics that have been associated with unfavorable outcomes after discharge.⁴ To identify this subset of high-risk individuals, various risk assessment scores have been developed, ranging from those that are used locally at the facility level to those that are nationally validated. The LACE score (Length of hospital stay; Acuity of the admission; Comorbidities measured with the Charlson comorbidity index score; and Emergency department visits within the past 6 months) is a validated index scoring tool that is used to identify medical and surgical patients at risk for readmission or death within 30 days of hospital discharge. On a 19-point scale, a score of ≥ 10 is considered high risk.⁵ Specific to the US Department of Veterans Affairs (VA), the Care Assessment Needs (CAN) score was developed to risk stratify the veteran population. The CAN score is generated using information including patient demographics, medical conditions, VA health care utilization, vital signs, laboratory values, medications, and socioeconomic status. This score is expressed as a percentile that compares the probability of death or admission among veterans at 90 days and 1 year postdischarge. Veterans in the 99th percentile have a 74% risk for these adverse outcomes at 1 year.⁶

The Joint Commission states that a fundamental component to assuring safe and effective TOC is medication management, which includes the involvement of pharmacists.² TOC programs with pharmacist involvement have shown significant improvements related to reduced 30-day hospital readmissions and health care costs in addition to significant medication-related interventions.^7-9 While this body of evidence continues to grow and demonstrates that pharmacists are an integral component of the TOC process, there is no gold standard program. Brantley and colleagues noted that a weakness of many TOC programs is that they are one dimensional, meaning that they focus on only 1 element of care transitions or 1 specific patient population or disease.¹⁰

There is well-supported evidence of high-impact interventions for pharmacists involved early in the admission process, but data are less robust on the discharge process. ^11,12 Therefore, the primary focus of this project was to develop a pharmacist-based TOC program and implement a process for communicating high-risk patients who are discharging from our hospital across the continuum of care.

Setting

The Richard L. Roudebush VA Medical Center (RLRVAMC) is a tertiary care referral center for veterans in Indiana and eastern Illinois. Acute care clinical pharmacists are fully integrated into the acute care teams and practice under a comprehensive care model. Pharmacists attend daily patient care rounds and conduct discharge medication reconciliation for all patients with additional bedside counseling for patients who are being discharged home.

Primary care services are provided by patient aligned care teams (PACTs), multidisciplinary teams composed of physicians, advanced practice nurses, pharmacists, mental health care providers, registered nurses, dieticians, and care coordinators. Ambulatory Care or PACT clinical pharmacists are established within each RLRVAMC PACT clinic and provide comprehensive care management through an independent scope of practice for several chronic diseases, including hypertension, type 2 diabetes mellitus (T2DM), dyslipidemia, hypothyroidism, and tobacco cessation. Prior to this project implementation, there was no formalized or standardized method for facilitating routine communication of patients between acute care and PACT pharmacists in the TOC process.

Pilot Study

In 2017, RLRVAMC implemented a TOC pharmacy program pilot. A pharmacy resident and both acute care and PACT clinical pharmacy specialists (CPSs) developed the service. The pilot program was conducted from September 1, 2017 to March 1, 2018. The initial phase consisted of the development of an electronic TOC tool to standardize communication between acute care and PACT pharmacists. The TOC tool was created on a secure site accessible only to pharmacy personnel and not part of the formal medical record. (Figure 1).

The acute care pharmacist identified high-risk patients through calculated CAN and LACE scores during the discharge process and offered PACT pharmacist follow-up to the patient during bedside discharge counseling. Information was then entered into the TOC tool, including patient identifiers and a message with specific information outlining the reason for referral. PACT pharmacists routinely reviewed the tool and attempted to phone each patient within 7 days of discharge. Follow-up included medication reconciliation and chronic disease management as warranted at the discretion of the PACT pharmacist. All postdischarge follow-up appointments were created and documented in the electronic health record. A retrospective chart review was completed on patients who were entered into the TOC tool.

Patients were eligible for referral if they were discharged during the study period with primary care established in one of the facility’s PACT clinics. Additionally, patients had to meet ≥ 1 of the following criteria, deeming them a high risk for readmission: LACE score ≥ 10, CAN score ≥ 90th percentile, or be considered high risk based on the discretion of the acute care pharmacist. Patients were included in the analysis if they met the CAN or LACE score requirement. Patients were excluded if they received primary care from a site other than a RLRVAMC PACT clinic. This included non-VA primary care, home-based primary care, or VA community-based outpatient clinics (CBOCs). Patients also were excluded if they required further institutional care postdischarge (ie, subacute rehabilitation, extended care facility, etc), discharged to hospice, or against medical advice.

The average referral rate per month during the pilot study was 19 patients, with 113 total referrals during the 6-month study period. Lower rates of index emergency department (ED) visits (5.3% vs 23.3%) and readmissions (1% vs 6.7%) were seen in the group of patients who received PACT pharmacist follow-up postdischarge compared with those who did not. Additionally, PACT pharmacists were able to make > 120 interventions, averaging 1.7 interventions per patient. Of note, these results were not statistically analyzed and were assessed as observational data to determine whether the program had the potential to be impactful. The results of the pilot study demonstrated positive outcomes associated with having a pharmacist-based TOC process and led to the desire for further development and implementation of the TOC program at the RLRVAMC. These positive results prompted a second phase project to address barriers, make improvements, and ensure sustainability.

Methods

Phase 2 was a quality improvement initiative; therefore, institutional review board approval was not needed. The aim of phase 2 was to improve, expand, and sustain the TOC program that was implemented in the pilot study. Barriers identified after discussion with acute care and PACT pharmacists included difficulty in making referrals due to required entry of cumbersome readmission risk factor calculations, limiting inclusion to patients who receive primary care at the main hospital facility, and the expansion of pharmacy staff with new pharmacists who were not knowledgeable of the referral process.

Design

To overcome barriers, 4 main targeted interventions were needed: streamlining the referral process, enhancing pharmacy staff education, updating the discharge note template, and expanding the criteria to include patients who receive care at VA CBOCs. The referral process was streamlined by removing required calculated readmission risk scores, allowing pharmacist judgement to take precedence for referrals. Focused face-to-face education was provided to acute care and PACT pharmacists about the referral process and inclusion criteria to increase awareness and provide guidance of who may benefit from entry into the tool. Unlike the first phase of the study, education was provided for outpatient staff pharmacists responsible for discharging patients on the weekends. Additionally, the pharmacists received a printed quick reference guide of the information covered during the education sessions (Figure 2). Referral prompts were embedded into the standard pharmacy discharge note template to serve as a reminder to discharging pharmacists to assess patients for inclusion into the tool and provided a direct link to the tool. Expansion to include VA CBOCs occurred postpilot study, allowing increased patient access to this TOC service. All other aspects of the program were continued from the pilot phase.

Patients were eligible if they were discharged from RLRVAMC between October 1, 2018 and February 28, 2019. Additionally, the patient had to be established in a PACT clinic for primary care and have been referred to the tool based on the discretion of an acute care pharmacist. Patients were excluded if they were discharged against medical advice or to any facility where the patient and/or caregiver would not be responsible for medication administration (eg, subacute rehabilitation, extended care facility), or if the patient refused pharmacy follow-up.

Outcomes

The primary outcomes assessed were all-cause and index ED visits and readmissions within 30 days of discharge. All-cause ED visits and readmissions were defined as a second visit to RLRVAMC , regardless of readmission diagnosis. Index ED visits and readmissions were defined as those that were related to the initial admission diagnosis. Additional data collected and analyzed included the number of patients referred by pharmacists, number and type of medication discrepancies, medication changes, counseling interventions, time to follow-up postdischarge, and number of patients added to the PACT pharmacist’s clinic schedule for further management. A discrepancy identified by a PACT pharmacist was defined as a difference between the discharge medication list and the patient-reported medication list at the time of follow-up. Patients who were referred to the TOC tool but were unable to be reached by telephone served as the control group for this study.

Data Collection

A retrospective chart review was completed on patients entered into the tool. Data were collected and kept in a secured Microsoft Excel workbook. Baseline characteristics were analyzed using either a χ² for nominal data or Student t test for continuous data. The primary outcomes were analyzed using a χ² test. All statistical tests were analyzed using MiniTab 19 Statistical Software.

Results

Pharmacists added 172 patients into the TOC tool; 139 patients met inclusion criteria. Of those excluded, most were because the PACT pharmacist did not attempt to contact the patient since they already had a primary care visit scheduled postdischarge (Table 1). Of the 139 patients who met the inclusion criteria, 99 were successfully contacted by a PACT pharmacist. Most patients were aged in their 60s, male, and white. Both groups had a similar quantity of outpatient medications on admission and medication changes made at discharge. Additionally, both groups had a similar number of patients with hospitalizations and/or ED visits in the 3 months before hospital admission that resulted in TOC tool referral (Table 2).

Hospital Readmission

Hospital 30-day readmission rates for patients who were successfully followed by pharmacy compared with those who were not were 5.1% vs 15.0% (P = .049) for index readmissions and 8.1% vs 27.5% (P = .03) for all-cause readmissions. No statistically significant difference existed between those patients with follow-up compared with those without follow-up for either index (10.1% vs 12.5%, respectively; P = .68) or for all-cause ED visit rates (15.2% vs 20.0%, respectively; P = .49).

Patient Encounters

The average time to follow-up was 8.8 days, which was above the predetermined goal of contact within 7 days. Additionally, this was a decline from the initial pilot study, which had an average time to reach of 4.7 days. All patients reached by a pharmacist received medication reconciliation, with ≥ 28% of patients having ≥ 1 discrepancy. There were 43 discrepancies among all patients. Of the discrepancies, 25 were reported as errors performed by the patient, and 18 were from an error during the discharge process. The discrepancies that resulted from patient error were primarily patients who took the wrong dose of prescribed medications. Other patient discrepancies included taking medications not as scheduled, omitting medications (both intentionally and mistakenly), continuing to take medications that had been discontinued by a health care provider and improper administration technique. Examples of provider errors that occurred during the discharge process included not ordering medications for patient to pick up at discharge, not discontinuing a medication from the patient’s profile, and failure to renew expired prescriptions.

Additional counseling was provided to 75% of patients: The most common reason for counseling was T2DM, hypertension, and dyslipidemia management. PACT pharmacists changed medication regimens for 27.3% of patients for improved control of chronic diseases or relief of medication AEs.

At the end of each visit, patients were assessed to determine whether they could benefit from additional pharmacy follow-up. Thirty-seven patients were added to the pharmacist schedules for disease management appointments. The most common conditions for these appointments were T2DM, hypertension, tobacco cessation, and hyperlipidemia. Among the 37 patients who had pharmacy follow-up, there were 137 additional pharmacy appointments within the study period.

Program Referrals

After expansion to include the VA CBOCs, elimination of the elevated LACE or CAN score requirement, and additional staff education, the rate of referrals per month increased during phase 2 in comparison to the pilot study (Figure 3). There were a mean (SD) of 34 (10) referrals per month. Although not statistically analyzed, it is an objective increase in comparison to a mean 19 referrals per month in the pilot study.

Discussion

The continued development and use of a pharmacist-driven TOC tool at RLRVAMC increased communication and follow-up of high-risk patients, demonstrated the ability of pharmacists to identify and intervene in medication-related issues postdischarge, and successfully reduce 30-day readmissions. This program emphasized pharmacist involvement during the discharge process and created a standardized mechanism for TOC follow-up, addressing multiple areas that were identified by The Joint Commission as being associated with poor outcomes. The advanced pharmacy practice model at RLRVAMC allowed for a multidimensional program, including prospective patient identification and multiple pharmacy touchpoints. This is unique in comparison to many of the one-dimensional programs described in the literature.

Polypharmacy has been identified as a major predictor of medication discrepancies postdischarge, and patients with ≥ 10 active medications have been found to be at highest risk.^13,14 Patients in this study had a mean 13 active medications on admission, with a mean 5 medication changes at discharge. PACT pharmacists documented 28 of 99 patients with ≥ 1 medication-related discrepancy at postdischarge reconciliation. This 28% discrepancy rate is consistent with discrepancy rates previously reported in the literature, which ranged from 14 to 45% in large meta-analyses.^14,15 The majority of these discrepancies (58%) were related to patients who took the wrong dose of a prescribed medication.

Targeted interventions to overcome barriers in the pilot study increased the referral rates to the TOC tool; however, the increase in referral rate was associated with increased time to follow up by ambulatory care pharmacists. The extended follow-up times were seen most often in the 2 busiest primary care clinics, one of which is considered a teaching clinic for medical residents. Pharmacists were required to integrate these calls into their normal work schedule and were not provided additional time for calling, allowing for an increased follow-up time. The increased follow-up time likely contributed to the increased number of patients excluded due to already having PACT follow-up, giving more time for the primary care provider to have an appointment with the patient. The ambulatory care pharmacist could then determine whether further intervention was needed. In the summer of 2018, a decrease in referral rates occurred for a short time, but this is likely explained by incoming new residents and staff within the pharmacy department and decreased awareness among the new staff. The enhanced staff education took place during September 2018 and lead to increased referral rates compared with those seen in months prior.

PACT pharmacists were not only able to identify discrepancies, but also provide timely intervention on a multitude of medication-related issues by using their scope of practice (SOP). Most interventions were related to medication or disease counseling, including lifestyle, device, and disease education. The independent SOP of our PACT pharmacists is a unique aspect of this program and allowed pharmacists to independently adjust many aspects of a patient’s medication regimen during follow-up visits.

The outcomes of 30-day index and all-cause readmissions, as well as index and all-cause ED visit rates, were lower in the subset of patients who received PACT pharmacist follow-up after discharge (Table 3). The difference was most pronounced in the all-cause readmission rates: Only 8.1% of patients who received PACT follow-up experienced a readmission compared with 27.5% of those who did not. The difference between the groups regarding ED visit rates were not as pronounced, but this may be attributed to a limited sample size. These data indicate that the role of the pharmacist in identifying discrepancies and performing interventions at follow-up may play a clinically significant part in reducing both ED visit rates and hospital readmissions.

Limitations

There are some limitations identified within this study. Although the referral criteria were relaxed from the pilot study and enhanced education was created, continued education regarding appropriate referral of TOC patients continues to be necessary given intermittent staff changeover, incorporation of pharmacy trainees, and modifications to clinic workflow. Patients who were discharged to facilities were not included. This ensured that appropriate and consistent PACT pharmacist follow-up would be available, but likely reduced our sample size.

Although performing this study in a closed health care system with pharmacists who have independent SOPs is a strength of our study, also it can limit generalizability. Not all facilities house both acute care and ambulatory care in one location with wide SOPs to allow for comprehensive and continued care. Last, this study used convenience sampling, potentially introducing selection bias, as patients unable to be reached by PACT pharmacists may inherently be at increased risk for hospital readmission. However, in the 3 months preceding the hospital admission that resulted in TOC tool referral, both groups had a similar number of patients with hospital admissions and ED visits.

The TOC tool has become fully integrated into the daily workflow for both acute care and PACT pharmacists. After the conclusion of the study period, the referral rates into the tool have been maintained at a steady level, even surpassing the rates seen during the study period. In comparison with the pilot study, PACT pharmacists reported a subjective increase in referrals placed for procedures such as medication reconciliation or adherence checks. This is likely because acute care pharmacists were able to use their clinical judgement rather than to rely solely on calculated readmission risk scores for TOC tool referral.

The success of the TOC program led to the expansion to other specialty areas. ED pharmacists now refer patients from the ED who were not admitted to the hospital but would benefit from PACT follow-up. Additionally, the option to refer hematology and oncology patients was added to allow these patients to be followed up by our hematology/oncology CPSs by phone appointments. Unique reasons for follow-up for this patient population include concerns about delayed chemotherapy cycles or chemotherapy-associated AEs.

Conclusions

This study outlines the creation and continued improvement of a pharmacist-based TOC program. The program was designed as a method of communication between acute care and PACT pharmacists about high-risk patients. The creation of this program allowed PACT pharmacists not only to identify discrepancies and make interventions on high-risk patients, but also demonstrate that having pharmacists involved in these programs may have a positive impact on readmissions and ED visits. The success of the TOC tool at the RLRVAMC has led to its expansion and is now an integral part of the daily workflow for both acute care and PACT pharmacists.

Effective transitions of care (TOC) are essential to ensure quality continuity of care after hospital discharge. About 20 to 30% of patients experience an adverse event (AE) in the peridischarge period when discharged to the community.¹ Additionally, about two-thirds of AEs are preventable.¹ The Joint Commission has identified various breakdowns in care that are associated with poor outcomes, including a lack of standardized discharge procedures, limited time dedicated to discharge planning and processes, and patients who lack the necessary resources or skills to implement discharge care plans.²

Background

The most impactful TOC programs are those that target patients who are at high risk for readmission or adverse outcomes.³ Factors such as advanced age, polypharmacy, cognitive impairment, and lack of social support are patient characteristics that have been associated with unfavorable outcomes after discharge.⁴ To identify this subset of high-risk individuals, various risk assessment scores have been developed, ranging from those that are used locally at the facility level to those that are nationally validated. The LACE score (Length of hospital stay; Acuity of the admission; Comorbidities measured with the Charlson comorbidity index score; and Emergency department visits within the past 6 months) is a validated index scoring tool that is used to identify medical and surgical patients at risk for readmission or death within 30 days of hospital discharge. On a 19-point scale, a score of ≥ 10 is considered high risk.⁵ Specific to the US Department of Veterans Affairs (VA), the Care Assessment Needs (CAN) score was developed to risk stratify the veteran population. The CAN score is generated using information including patient demographics, medical conditions, VA health care utilization, vital signs, laboratory values, medications, and socioeconomic status. This score is expressed as a percentile that compares the probability of death or admission among veterans at 90 days and 1 year postdischarge. Veterans in the 99th percentile have a 74% risk for these adverse outcomes at 1 year.⁶

The Joint Commission states that a fundamental component to assuring safe and effective TOC is medication management, which includes the involvement of pharmacists.² TOC programs with pharmacist involvement have shown significant improvements related to reduced 30-day hospital readmissions and health care costs in addition to significant medication-related interventions.^7-9 While this body of evidence continues to grow and demonstrates that pharmacists are an integral component of the TOC process, there is no gold standard program. Brantley and colleagues noted that a weakness of many TOC programs is that they are one dimensional, meaning that they focus on only 1 element of care transitions or 1 specific patient population or disease.¹⁰

There is well-supported evidence of high-impact interventions for pharmacists involved early in the admission process, but data are less robust on the discharge process. ^11,12 Therefore, the primary focus of this project was to develop a pharmacist-based TOC program and implement a process for communicating high-risk patients who are discharging from our hospital across the continuum of care.

Setting

The Richard L. Roudebush VA Medical Center (RLRVAMC) is a tertiary care referral center for veterans in Indiana and eastern Illinois. Acute care clinical pharmacists are fully integrated into the acute care teams and practice under a comprehensive care model. Pharmacists attend daily patient care rounds and conduct discharge medication reconciliation for all patients with additional bedside counseling for patients who are being discharged home.

Primary care services are provided by patient aligned care teams (PACTs), multidisciplinary teams composed of physicians, advanced practice nurses, pharmacists, mental health care providers, registered nurses, dieticians, and care coordinators. Ambulatory Care or PACT clinical pharmacists are established within each RLRVAMC PACT clinic and provide comprehensive care management through an independent scope of practice for several chronic diseases, including hypertension, type 2 diabetes mellitus (T2DM), dyslipidemia, hypothyroidism, and tobacco cessation. Prior to this project implementation, there was no formalized or standardized method for facilitating routine communication of patients between acute care and PACT pharmacists in the TOC process.

Pilot Study

In 2017, RLRVAMC implemented a TOC pharmacy program pilot. A pharmacy resident and both acute care and PACT clinical pharmacy specialists (CPSs) developed the service. The pilot program was conducted from September 1, 2017 to March 1, 2018. The initial phase consisted of the development of an electronic TOC tool to standardize communication between acute care and PACT pharmacists. The TOC tool was created on a secure site accessible only to pharmacy personnel and not part of the formal medical record. (Figure 1).

The acute care pharmacist identified high-risk patients through calculated CAN and LACE scores during the discharge process and offered PACT pharmacist follow-up to the patient during bedside discharge counseling. Information was then entered into the TOC tool, including patient identifiers and a message with specific information outlining the reason for referral. PACT pharmacists routinely reviewed the tool and attempted to phone each patient within 7 days of discharge. Follow-up included medication reconciliation and chronic disease management as warranted at the discretion of the PACT pharmacist. All postdischarge follow-up appointments were created and documented in the electronic health record. A retrospective chart review was completed on patients who were entered into the TOC tool.

Patients were eligible for referral if they were discharged during the study period with primary care established in one of the facility’s PACT clinics. Additionally, patients had to meet ≥ 1 of the following criteria, deeming them a high risk for readmission: LACE score ≥ 10, CAN score ≥ 90th percentile, or be considered high risk based on the discretion of the acute care pharmacist. Patients were included in the analysis if they met the CAN or LACE score requirement. Patients were excluded if they received primary care from a site other than a RLRVAMC PACT clinic. This included non-VA primary care, home-based primary care, or VA community-based outpatient clinics (CBOCs). Patients also were excluded if they required further institutional care postdischarge (ie, subacute rehabilitation, extended care facility, etc), discharged to hospice, or against medical advice.

The average referral rate per month during the pilot study was 19 patients, with 113 total referrals during the 6-month study period. Lower rates of index emergency department (ED) visits (5.3% vs 23.3%) and readmissions (1% vs 6.7%) were seen in the group of patients who received PACT pharmacist follow-up postdischarge compared with those who did not. Additionally, PACT pharmacists were able to make > 120 interventions, averaging 1.7 interventions per patient. Of note, these results were not statistically analyzed and were assessed as observational data to determine whether the program had the potential to be impactful. The results of the pilot study demonstrated positive outcomes associated with having a pharmacist-based TOC process and led to the desire for further development and implementation of the TOC program at the RLRVAMC. These positive results prompted a second phase project to address barriers, make improvements, and ensure sustainability.

Methods

Phase 2 was a quality improvement initiative; therefore, institutional review board approval was not needed. The aim of phase 2 was to improve, expand, and sustain the TOC program that was implemented in the pilot study. Barriers identified after discussion with acute care and PACT pharmacists included difficulty in making referrals due to required entry of cumbersome readmission risk factor calculations, limiting inclusion to patients who receive primary care at the main hospital facility, and the expansion of pharmacy staff with new pharmacists who were not knowledgeable of the referral process.

Design

To overcome barriers, 4 main targeted interventions were needed: streamlining the referral process, enhancing pharmacy staff education, updating the discharge note template, and expanding the criteria to include patients who receive care at VA CBOCs. The referral process was streamlined by removing required calculated readmission risk scores, allowing pharmacist judgement to take precedence for referrals. Focused face-to-face education was provided to acute care and PACT pharmacists about the referral process and inclusion criteria to increase awareness and provide guidance of who may benefit from entry into the tool. Unlike the first phase of the study, education was provided for outpatient staff pharmacists responsible for discharging patients on the weekends. Additionally, the pharmacists received a printed quick reference guide of the information covered during the education sessions (Figure 2). Referral prompts were embedded into the standard pharmacy discharge note template to serve as a reminder to discharging pharmacists to assess patients for inclusion into the tool and provided a direct link to the tool. Expansion to include VA CBOCs occurred postpilot study, allowing increased patient access to this TOC service. All other aspects of the program were continued from the pilot phase.

Patients were eligible if they were discharged from RLRVAMC between October 1, 2018 and February 28, 2019. Additionally, the patient had to be established in a PACT clinic for primary care and have been referred to the tool based on the discretion of an acute care pharmacist. Patients were excluded if they were discharged against medical advice or to any facility where the patient and/or caregiver would not be responsible for medication administration (eg, subacute rehabilitation, extended care facility), or if the patient refused pharmacy follow-up.

Outcomes

The primary outcomes assessed were all-cause and index ED visits and readmissions within 30 days of discharge. All-cause ED visits and readmissions were defined as a second visit to RLRVAMC , regardless of readmission diagnosis. Index ED visits and readmissions were defined as those that were related to the initial admission diagnosis. Additional data collected and analyzed included the number of patients referred by pharmacists, number and type of medication discrepancies, medication changes, counseling interventions, time to follow-up postdischarge, and number of patients added to the PACT pharmacist’s clinic schedule for further management. A discrepancy identified by a PACT pharmacist was defined as a difference between the discharge medication list and the patient-reported medication list at the time of follow-up. Patients who were referred to the TOC tool but were unable to be reached by telephone served as the control group for this study.

Data Collection

A retrospective chart review was completed on patients entered into the tool. Data were collected and kept in a secured Microsoft Excel workbook. Baseline characteristics were analyzed using either a χ² for nominal data or Student t test for continuous data. The primary outcomes were analyzed using a χ² test. All statistical tests were analyzed using MiniTab 19 Statistical Software.

Results

Pharmacists added 172 patients into the TOC tool; 139 patients met inclusion criteria. Of those excluded, most were because the PACT pharmacist did not attempt to contact the patient since they already had a primary care visit scheduled postdischarge (Table 1). Of the 139 patients who met the inclusion criteria, 99 were successfully contacted by a PACT pharmacist. Most patients were aged in their 60s, male, and white. Both groups had a similar quantity of outpatient medications on admission and medication changes made at discharge. Additionally, both groups had a similar number of patients with hospitalizations and/or ED visits in the 3 months before hospital admission that resulted in TOC tool referral (Table 2).

Hospital Readmission

Hospital 30-day readmission rates for patients who were successfully followed by pharmacy compared with those who were not were 5.1% vs 15.0% (P = .049) for index readmissions and 8.1% vs 27.5% (P = .03) for all-cause readmissions. No statistically significant difference existed between those patients with follow-up compared with those without follow-up for either index (10.1% vs 12.5%, respectively; P = .68) or for all-cause ED visit rates (15.2% vs 20.0%, respectively; P = .49).

Patient Encounters

The average time to follow-up was 8.8 days, which was above the predetermined goal of contact within 7 days. Additionally, this was a decline from the initial pilot study, which had an average time to reach of 4.7 days. All patients reached by a pharmacist received medication reconciliation, with ≥ 28% of patients having ≥ 1 discrepancy. There were 43 discrepancies among all patients. Of the discrepancies, 25 were reported as errors performed by the patient, and 18 were from an error during the discharge process. The discrepancies that resulted from patient error were primarily patients who took the wrong dose of prescribed medications. Other patient discrepancies included taking medications not as scheduled, omitting medications (both intentionally and mistakenly), continuing to take medications that had been discontinued by a health care provider and improper administration technique. Examples of provider errors that occurred during the discharge process included not ordering medications for patient to pick up at discharge, not discontinuing a medication from the patient’s profile, and failure to renew expired prescriptions.

Additional counseling was provided to 75% of patients: The most common reason for counseling was T2DM, hypertension, and dyslipidemia management. PACT pharmacists changed medication regimens for 27.3% of patients for improved control of chronic diseases or relief of medication AEs.

At the end of each visit, patients were assessed to determine whether they could benefit from additional pharmacy follow-up. Thirty-seven patients were added to the pharmacist schedules for disease management appointments. The most common conditions for these appointments were T2DM, hypertension, tobacco cessation, and hyperlipidemia. Among the 37 patients who had pharmacy follow-up, there were 137 additional pharmacy appointments within the study period.

Program Referrals

After expansion to include the VA CBOCs, elimination of the elevated LACE or CAN score requirement, and additional staff education, the rate of referrals per month increased during phase 2 in comparison to the pilot study (Figure 3). There were a mean (SD) of 34 (10) referrals per month. Although not statistically analyzed, it is an objective increase in comparison to a mean 19 referrals per month in the pilot study.

Discussion

The continued development and use of a pharmacist-driven TOC tool at RLRVAMC increased communication and follow-up of high-risk patients, demonstrated the ability of pharmacists to identify and intervene in medication-related issues postdischarge, and successfully reduce 30-day readmissions. This program emphasized pharmacist involvement during the discharge process and created a standardized mechanism for TOC follow-up, addressing multiple areas that were identified by The Joint Commission as being associated with poor outcomes. The advanced pharmacy practice model at RLRVAMC allowed for a multidimensional program, including prospective patient identification and multiple pharmacy touchpoints. This is unique in comparison to many of the one-dimensional programs described in the literature.

Polypharmacy has been identified as a major predictor of medication discrepancies postdischarge, and patients with ≥ 10 active medications have been found to be at highest risk.^13,14 Patients in this study had a mean 13 active medications on admission, with a mean 5 medication changes at discharge. PACT pharmacists documented 28 of 99 patients with ≥ 1 medication-related discrepancy at postdischarge reconciliation. This 28% discrepancy rate is consistent with discrepancy rates previously reported in the literature, which ranged from 14 to 45% in large meta-analyses.^14,15 The majority of these discrepancies (58%) were related to patients who took the wrong dose of a prescribed medication.

Targeted interventions to overcome barriers in the pilot study increased the referral rates to the TOC tool; however, the increase in referral rate was associated with increased time to follow up by ambulatory care pharmacists. The extended follow-up times were seen most often in the 2 busiest primary care clinics, one of which is considered a teaching clinic for medical residents. Pharmacists were required to integrate these calls into their normal work schedule and were not provided additional time for calling, allowing for an increased follow-up time. The increased follow-up time likely contributed to the increased number of patients excluded due to already having PACT follow-up, giving more time for the primary care provider to have an appointment with the patient. The ambulatory care pharmacist could then determine whether further intervention was needed. In the summer of 2018, a decrease in referral rates occurred for a short time, but this is likely explained by incoming new residents and staff within the pharmacy department and decreased awareness among the new staff. The enhanced staff education took place during September 2018 and lead to increased referral rates compared with those seen in months prior.

PACT pharmacists were not only able to identify discrepancies, but also provide timely intervention on a multitude of medication-related issues by using their scope of practice (SOP). Most interventions were related to medication or disease counseling, including lifestyle, device, and disease education. The independent SOP of our PACT pharmacists is a unique aspect of this program and allowed pharmacists to independently adjust many aspects of a patient’s medication regimen during follow-up visits.

The outcomes of 30-day index and all-cause readmissions, as well as index and all-cause ED visit rates, were lower in the subset of patients who received PACT pharmacist follow-up after discharge (Table 3). The difference was most pronounced in the all-cause readmission rates: Only 8.1% of patients who received PACT follow-up experienced a readmission compared with 27.5% of those who did not. The difference between the groups regarding ED visit rates were not as pronounced, but this may be attributed to a limited sample size. These data indicate that the role of the pharmacist in identifying discrepancies and performing interventions at follow-up may play a clinically significant part in reducing both ED visit rates and hospital readmissions.

Limitations

There are some limitations identified within this study. Although the referral criteria were relaxed from the pilot study and enhanced education was created, continued education regarding appropriate referral of TOC patients continues to be necessary given intermittent staff changeover, incorporation of pharmacy trainees, and modifications to clinic workflow. Patients who were discharged to facilities were not included. This ensured that appropriate and consistent PACT pharmacist follow-up would be available, but likely reduced our sample size.

Although performing this study in a closed health care system with pharmacists who have independent SOPs is a strength of our study, also it can limit generalizability. Not all facilities house both acute care and ambulatory care in one location with wide SOPs to allow for comprehensive and continued care. Last, this study used convenience sampling, potentially introducing selection bias, as patients unable to be reached by PACT pharmacists may inherently be at increased risk for hospital readmission. However, in the 3 months preceding the hospital admission that resulted in TOC tool referral, both groups had a similar number of patients with hospital admissions and ED visits.

The TOC tool has become fully integrated into the daily workflow for both acute care and PACT pharmacists. After the conclusion of the study period, the referral rates into the tool have been maintained at a steady level, even surpassing the rates seen during the study period. In comparison with the pilot study, PACT pharmacists reported a subjective increase in referrals placed for procedures such as medication reconciliation or adherence checks. This is likely because acute care pharmacists were able to use their clinical judgement rather than to rely solely on calculated readmission risk scores for TOC tool referral.

The success of the TOC program led to the expansion to other specialty areas. ED pharmacists now refer patients from the ED who were not admitted to the hospital but would benefit from PACT follow-up. Additionally, the option to refer hematology and oncology patients was added to allow these patients to be followed up by our hematology/oncology CPSs by phone appointments. Unique reasons for follow-up for this patient population include concerns about delayed chemotherapy cycles or chemotherapy-associated AEs.

Conclusions

This study outlines the creation and continued improvement of a pharmacist-based TOC program. The program was designed as a method of communication between acute care and PACT pharmacists about high-risk patients. The creation of this program allowed PACT pharmacists not only to identify discrepancies and make interventions on high-risk patients, but also demonstrate that having pharmacists involved in these programs may have a positive impact on readmissions and ED visits. The success of the TOC tool at the RLRVAMC has led to its expansion and is now an integral part of the daily workflow for both acute care and PACT pharmacists.

The Veterans Health Administration (VHA), 1 of 3 administrative branches in the US Department of Veterans Affairs (VA), is the largest integrated health care system in the United States.¹ The VHA has 4 missions: providing health care to eligible veterans; supporting research to benefit veterans and the larger society; providing education for health care trainees; and supporting emergency response.¹ In service of these goals, VA has academic affiliations with universities throughout the country, offering unique, extensive training and research opportunities. Both the VA and the affiliate benefit from these partnerships. For example, VA affiliations with University of California (UC) medical schools benefit veteran care while facilitating the UC academic mission. Through these affiliations, trainees who learn within the VHA’s highly effective integrated care model become health care professionals (HCPs) who are prepared to enter health care systems in California and meet the state’s demand for high-quality integrated care with an emphasis on primary care, mental health care, and care for aging populations.^2,3

This report explores the history of the VHA, current veteran demographics and needs, VA academic affiliations, and the integrated care model of training in all VHA facilities. The VA and UC academic affiliation is described further with regard to shared research and educational functions. Finally, we identify potential risks to academic affiliations associated with increased VA reliance on community-based care following the implementation of recent legislation. We provide suggestions for VA academic affiliates to help assess and guide the potential impact of increased VA-managed community care.

VHA Resources

The VHA serves more than 9 million veterans through 170 medical centers and 1,074 outpatient care sites.¹ In fiscal year 2017, the VA provided 109 million outpatient visits, and treated 615,000 inpatient medicine/surgical patients and 149,000 patients in inpatient mental health.⁴ The VHA focuses on the distinct concerns of veterans, which arise from military service as well as their broader health care needs. Veterans have higher rates of medical and mental health conditions than those of the general public; different cohorts in this population experience distinct medical and mental health concerns (Table 1).⁵

In addition, although veterans are disproportionately older men, the population is diversifying.⁶ For example, the number of female veterans is growing; furthermore, changes in the law now allow lesbian, gay, bisexual, and transgender (LGBT) individuals to serve openly, which has both reduced barriers for this population and allowed for LGBT veterans who were not eligible for VA care due to less than honorable discharges to have those discharges upgraded. As a result, care has been tailored to include the development of Women Veterans Program Managers and related services and LGBT and related identities resources such as LGBT Veteran Care Coordinators in every VA facility nationwide.^7,8 The VA continues to adapt to serve all veterans; part of this adaptation is training HCPs to provide veteran-centered care for a growing and diversifying population.

VHA Resources in California

California has the largest population of veterans in the United States (Table 2).^9,10 Of the 9,116,200 VA enrollees nationwide, 760,910 (8%) reside in California, and of those, 463,410 had at least 1 VA visit in the past year.^3,10 The VHA is organized into 21 Veterans Integrated Service Networks (VISNs) that include multiple health care systems in the region associated with each VISN. California is part of VISN 21 (Northern California, Nevada, and Pacific Islands) and VISN 22 (Southern California, Nevada, and New Mexico). Among veterans who served in the recent Iraq and Afghanistan conflicts, 5.5% accessed care in VISN 21 and 9.3% accessed care in VISN 22.¹¹ The VHA provides critical infrastructure for meeting complex veteran needs, as well as related specialized training, education, and research for HCPs. This specialization has been the basis for the broad system of affiliations between VA and academic systems.

The VA continues to be a high priority in the federal budget process.¹² In 2017, slightly more than 9% of the VA health care budget, $6.4 billion, was spent on medical care in California.¹⁰ Consequently, California has a noteworthy portion of VA infrastructure (Table 3).^13,14 California has 8 VA medical centers (VAMCs) with hospital service (Fresno, Loma Linda, Long Beach, Palo Alto, Sacramento, San Diego, San Francisco, West Los Angeles), 3 VAMCs without hospital service (2 locations in the Palo Alto system and Sepulveda), 1 stand-alone extended-care facility (Martinez Community Living Center), and 1 stand-alone residential care facility (San Diego Domiciliary).⁹ The vast VA infrastructure in California and large population of veterans creates a strong demand for HCPs in the state.

VA Education and Collaboration

VA has been training clinicians and scholars since 1946, when VA academic affiliations were established by Memorandum Number 2.^15,16 Today, the VA is the largest educator of HCPs in the United States.¹⁷ In 2015, an estimated $10.3 to $12.5 billion was spent on mandatory Medicare graduate medical education (GME).¹⁸ In 2017, the VA spent $1.78 billion of discretionary funding on GME to fund 11,000 full-time equivalent (FTE) slots, leading to > 43,000 physician residents (> 30% of all physician residents) spending part of their training in a VHA facility.^18,19

This training mission has multiple benefits. It provides the VA with access to new HCPs who have the necessary training in veteran-specific needs, while supporting the national need for HCPs. In 2018, 120,890 clinical trainees received some or all of their training in the VA system.²⁰ Of the 152 US medical schools that are accredited by the Liaison Committee on Medical Education, 95% collaborate with the VA for training while 100% of the 34 doctor of osteopathic medicine programs have VA training collaborations.²⁰ The VA currently has an additional 18 partnerships with nursing schools.²¹ Further, 1,800 college and universities, including Hispanic-serving institutions and historically black colleges and universities, have VHA affiliations that provide training for more than 40 clinical health profession education programs.¹⁷

This training model has been successful in supporting VA staffing, as health care providers who trained in the VA are more likely to work in the VA.²² Among current VA employees, > 80% of optometrists, > 70% of podiatrists and psychologists, and > 60% of physicians received some part of their training in the VA system.²³ In combination with recent increased funding for staffing, the ability of the VA to directly hire trainees in identified professions, and the expansion of loan forgiveness to high-demand specialties (eg, psychiatry), the training partnership between the VA and affiliates has been critical in maintaining the needed VA workforce.^22,24,25

The VA Office of Academic Affiliations is responsible for all graduate medical and dental education administration in the VA system, which makes up 85% of its total budget. For each trainee, the VA provides approximately $60,000 toward their stipend in exchange for training and patient care time at a VHA hospital (Kenneth R. Jones, PhD, email communication, August 27, 2018).

California Health Care Education

The UC public university system, founded in 1869, currently has 10 campuses with a combined student body of > 280,000 students, along with 227,000 faculty and staff members.²⁶ For every research dollar provided by California, the UC secures $7 in federal and private funding.²⁶ The UC has 6 medical centers (Davis, Irvine, Los Angeles, Riverside, San Diego, and San Francisco); each is affiliated with at least 1 local VAMC.^27,28

California trains a substantial share of health care trainees. In 2016, there were 10,429 physician residents in training in California.²⁹ In 2017/2018, the San Francisco VAMC trained 1,178 medical students/residents, 57 pharmacy students, 25 nurse practitioner students, 19 optometry interns/students/residents, 11 dental students/residents, and 3 physical therapy students.²⁰ In total, 6,223 UC health professions students were trained in VHA facilities during the 2017/2018 training year (Table 4).²⁰ As of 2016, there were 105,907 physicians in California, and of those, 57% completed their GME in California.²⁹ In California in 2015, 74 GME-sponsoring institutions graduated 3,568 residents and fellows, an increase of 10% since 1997.³⁰ Of these sponsoring institutions, 6 of the top 8 programs were UC schools that graduated 48.4% (1,727) of all California residents and fellows in 2015.³⁰

VA Research Collaborations

The VA Office of Research and Development has existed for more than 90 years with a mission to improve veteran health and well-being via research and attract, train, and retain high-caliber researchers. VA provides a rich environment to conduct observational and interventional research due to its large, diverse veteran population, institutional support, and integrated information system with extensive EHR data.⁴¹ The success of the VA in facilitating research is evidenced by the fact that 3 VA investigators have been awarded Nobel prizes, and 7 have received Lasker Foundation Awards.⁴² The size of the VA allows for innovative large-scale research, such as the Million Veteran Program (MVP). The MVP study developed a mega-biobank of VA health records, questionnaires, and blood samples from nearly 1 million veterans to study genetic influences on health and disease and integrate genetic testing into health care delivery.⁴³ In addition to producing high-quality, innovative research, more than 60% of VA investigators also provide direct patient care.⁴²

VA research areas of focus include homelessness, polytrauma, traumatic brain injury, hearing and vision loss, spinal cord injury, mental health, pain management, precision medicine, prosthetics and amputation care, women’s health, and chronic diseases, such as Parkinson and Alzheimer diseases.⁴⁴ The VA estimates that, in 2021, total VA research spending will include a request of $787 million in addition to $370 million from the National Institutes of Health, the Department of Defense, and the Centers for Disease Control and Prevention, and $170 million from other nonfederal sources, for a projected total of $1.3 billion. This budget will support 2,200 projects with direct research and reimbursable employment of 3,275 FTEs,which are key to supporting VA academic affiliations.⁴⁵ These funds translate into substantial benefits to the UC system, including shared research and training resources, grant-funding opportunities for UC faculty, and the ability to recruit top researchers, educators, and clinicians to its institutions.

VA Reliance on Community Care 

The current VHA model is an integrated health care system that provides comprehensive, wraparound services to enrolled veterans, which are cost-effective, high quality, and consistently found to have equal or superior quality of care compared with that in the community.^6,46-50 Despite public criticism about wait times and access to care in the VA system, one study showed that VA wait-time statistics were comparable with or faster than those for community HCPs.^51,52 However, VA care coordination has undergone several changes to address these public criticisms, namely, the Veterans Access, Choice and Accountability Act of 2014 (38 USC § 1703 VACAA) and the VA MISSION Act of 2018 (42 USC § 274). VACAA was designed to increase access to care for veterans who live ≥ 40 miles from VA health care facilities or who are unable to been seen within 30 days of their preferred or clinically appropriate date.⁵³ More than 2 million veterans (almost 25% of VHA-enrolled veterans) have received community care since the inception of VACAA in 2014.⁵⁴

Recently, the MISSION Act mandated developing additional VA-coordinated community-based care through the establishment of a Veterans Community Care Program, which was established using existing VA 2019 fiscal year funds and did not include additional appropriations despite expanded criteria for community care referrals.⁵⁵ Without additional future appropriations, VA funds would be shifted from VA care into community care. While increasing access to community care has in some cases led to care that is faster and closer and that was previously inaccessible in local VA specialty care, these efforts could reduce veteran engagement with the VA system.⁵⁶

The changes implemented in VACAA and the VA MISSION Act were driven by important and valid concerns, including evidence of VA staff and officials covering up service deficiencies.⁵¹ Veterans in rural areas often have limited access to VA resources, and long travel to VAMCs or clinics can be an impediment. Veterans who have chosen community care tended to be those who have poorer health status, who live further away from VA facilities, women, and those who identified as White or Hispanic.^56,57 While VA health care is on average equivalent to or better than community resources, there is significant variability in quality within the VA system. Advocates have argued that providing competition and choice for veterans places pressure on the VA to improve care where it is not meeting expectations. Therefore, access to community care is an important resource for veterans and needs to be implemented effectively and efficiently to help veterans receive the care they need. However, expansion of community care access, depending on how it is implemented, also can have effects on academic partnerships and the education and research missions that should be incorporated into planning.

Each VA health care system receives funding through the Veterans Equitable Reimbursement Allocation (VERA), which provides funds largely based on the number of enrolled veterans and the complexity of the care they receive.⁵⁸ One potential implication of the shift among veterans to community care is a reduction in patients enrolled in VA programs, thus decreasing funding given to the VA to allocate for training and research. By definition, increased VA-managed community care means less opportunity for integrated training that brings together primary, mental health, and substance use care to meet patient needs. The Center for Medicare and Medicaid Services has developed a national initiative to help states develop programs in integrated care, particularly for individuals who are eligible for both Medicare and Medicaid.⁵⁹ For states to develop integrated care, they need trainees who function well in this model. Integrated care training is particularly vulnerable to disruption because any portion of a veteran’s care being transferred to the community can impede integration. In effect, training in integrated care, likely the most efficient and cost-effective approach to health care for reasons discussed earlier, could be reduced as providers and trainees are required to manage and coordinate patient care between separate institutions.³⁵

Educational Impact

The shift in usage from VA to community care has potential implications for academic affiliates, particularly in education and research.⁶⁰ If more people are served in community settings, potentially some VAMCs could be reduced, realigned, or closed. If this restructuring happens, academic partnerships could be impacted negatively. The VA is instituting an Infrastructure Review Commission with the task of examining current VA utilization. If a VA site with an academic affiliate was considered for realignment or closure, the reduction would eliminate the ability of the academic affiliate to provide education and research collaborations at that site.

In a less drastic manner, increasing care in the community may change opportunities for academic affiliates to partner with the VA. As noted, the UC system and California veterans benefit immensely from the VHA as an integrated health care system with dedicated missions of education and research. This partnership is a model in which the VA is the primary source of care for eligible enrolled veterans and provides integrated comprehensive services. If the VA moves to serving primarily as a coordinator of community HCPs rather than a direct provider of health care, academic affiliates would need to make major adjustments to both the education and training models. This change could particularly affect specialty training programs that rely on having adequate volumes of patients to provide an extensive experience to meet training needs. If fewer veterans receive care directly from the VA and are instead dispersed in the community, that will reduce the ability of academic faculty to participate in the education of medical and affiliated trainees and to participate in research in VA settings. It is unclear what other model could replace such a system and be as beneficial to the VA and the academic partners with which it is currently affiliated.

Given the needs that led to the VA increasing access to care and the potential implications discussed for the VA and partnerships with academic affiliates, VA health care systems and academic affiliate partners should consider several steps. These steps involve assessment, coordination, and promotion.

Both the VA and academic affiliates would benefit if the VA shared assessment data on the use of community care, particularly identifying changes that relate to key training and/or research missions. Such data sharing can be critical to determine whether any risks (or potential opportunities) need to be addressed. In addition, increasing research on the outcomes related to both VA care and community-based care is of high value to determine whether the current changes are achieving intended goals. The VA recently funded such work through its research service, and such work is critical for guiding future policy for the VA and for the affiliates.

Coordination among the VA, academic affiliates, and community partners is vital for change. The issue of community care expansion should be a standing item on coordination meetings and shared governance councils between the institutions. It may make sense to establish specific workgroups or committees to coordinate tracking and assessment of the effect of community care expansion on the shared academic mission. One way to address the potential effect of increased community care on the research and education missions would be to include community partners into the partnerships. This strategy could potentially take a number of different forms, from providing education and training to community HCPs, having VA trainees rotate to community settings, or inviting community settings to be research sites for clinical trials. Such partnerships could potentially improve patient care and support the other academic missions. Coordination could be meaningfully improved by having community HCPs access the VA EHR, thus easing communications. Funding is available for EHR access in the VA MISSION Act and should be a high priority as community care expands. The more that community partners can access and connect with the VA EHR the better they will be able to coordinate care.

Third, the VA and its academic partners need to promote and educate veterans, their families, and their advocates on the benefits that are available through VA care and that are enhanced through academic partnerships. While the VA has been the target of justified criticism, many of its strengths addressed here are not broadly recognized. The VA could promote its sharing of staff and resources with the top academic health care institutions in an area and that veterans often have access to resources that otherwise would not be available without the academic affiliate. Making sure veterans are aware of the benefits available can potentially mitigate the need for community care.

Conclusions

Given changes from VACAA and the VA MISSION Act, VA and academic affiliates should be active partners in planning for future health care by providing input and feedback on VA structure to help shape federal and state systems moving forward. Institutions can take steps to steer their futures and meet growing clinical, training, and research needs. The VA and its academic partners in health care research are well positioned to develop projects to assess the effects of these changes. Evaluation of key variables including patient care, education, and research productivity are warranted to guide policymakers as they assess whether these changes in the VA are achieving the expressed goals of improving veteran care. Other opportunities to collaborate in the wake of the MISSION Act remain to be discovered within each academic affiliation. By strengthening working relationships between VA and academic teams, these deeply important partnerships can continue to produce clinical, research, and education outcomes that meet the needs of our veterans, our federal and state health care systems, and our country.

Acknowledgments

Dr. Sells was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Academic Affiliations VA Quality Scholars Advanced Fellowship Program.

The Veterans Health Administration (VHA), 1 of 3 administrative branches in the US Department of Veterans Affairs (VA), is the largest integrated health care system in the United States.¹ The VHA has 4 missions: providing health care to eligible veterans; supporting research to benefit veterans and the larger society; providing education for health care trainees; and supporting emergency response.¹ In service of these goals, VA has academic affiliations with universities throughout the country, offering unique, extensive training and research opportunities. Both the VA and the affiliate benefit from these partnerships. For example, VA affiliations with University of California (UC) medical schools benefit veteran care while facilitating the UC academic mission. Through these affiliations, trainees who learn within the VHA’s highly effective integrated care model become health care professionals (HCPs) who are prepared to enter health care systems in California and meet the state’s demand for high-quality integrated care with an emphasis on primary care, mental health care, and care for aging populations.^2,3

This report explores the history of the VHA, current veteran demographics and needs, VA academic affiliations, and the integrated care model of training in all VHA facilities. The VA and UC academic affiliation is described further with regard to shared research and educational functions. Finally, we identify potential risks to academic affiliations associated with increased VA reliance on community-based care following the implementation of recent legislation. We provide suggestions for VA academic affiliates to help assess and guide the potential impact of increased VA-managed community care.

VHA Resources

The VHA serves more than 9 million veterans through 170 medical centers and 1,074 outpatient care sites.¹ In fiscal year 2017, the VA provided 109 million outpatient visits, and treated 615,000 inpatient medicine/surgical patients and 149,000 patients in inpatient mental health.⁴ The VHA focuses on the distinct concerns of veterans, which arise from military service as well as their broader health care needs. Veterans have higher rates of medical and mental health conditions than those of the general public; different cohorts in this population experience distinct medical and mental health concerns (Table 1).⁵

In addition, although veterans are disproportionately older men, the population is diversifying.⁶ For example, the number of female veterans is growing; furthermore, changes in the law now allow lesbian, gay, bisexual, and transgender (LGBT) individuals to serve openly, which has both reduced barriers for this population and allowed for LGBT veterans who were not eligible for VA care due to less than honorable discharges to have those discharges upgraded. As a result, care has been tailored to include the development of Women Veterans Program Managers and related services and LGBT and related identities resources such as LGBT Veteran Care Coordinators in every VA facility nationwide.^7,8 The VA continues to adapt to serve all veterans; part of this adaptation is training HCPs to provide veteran-centered care for a growing and diversifying population.

VHA Resources in California

California has the largest population of veterans in the United States (Table 2).^9,10 Of the 9,116,200 VA enrollees nationwide, 760,910 (8%) reside in California, and of those, 463,410 had at least 1 VA visit in the past year.^3,10 The VHA is organized into 21 Veterans Integrated Service Networks (VISNs) that include multiple health care systems in the region associated with each VISN. California is part of VISN 21 (Northern California, Nevada, and Pacific Islands) and VISN 22 (Southern California, Nevada, and New Mexico). Among veterans who served in the recent Iraq and Afghanistan conflicts, 5.5% accessed care in VISN 21 and 9.3% accessed care in VISN 22.¹¹ The VHA provides critical infrastructure for meeting complex veteran needs, as well as related specialized training, education, and research for HCPs. This specialization has been the basis for the broad system of affiliations between VA and academic systems.

The VA continues to be a high priority in the federal budget process.¹² In 2017, slightly more than 9% of the VA health care budget, $6.4 billion, was spent on medical care in California.¹⁰ Consequently, California has a noteworthy portion of VA infrastructure (Table 3).^13,14 California has 8 VA medical centers (VAMCs) with hospital service (Fresno, Loma Linda, Long Beach, Palo Alto, Sacramento, San Diego, San Francisco, West Los Angeles), 3 VAMCs without hospital service (2 locations in the Palo Alto system and Sepulveda), 1 stand-alone extended-care facility (Martinez Community Living Center), and 1 stand-alone residential care facility (San Diego Domiciliary).⁹ The vast VA infrastructure in California and large population of veterans creates a strong demand for HCPs in the state.

VA Education and Collaboration

VA has been training clinicians and scholars since 1946, when VA academic affiliations were established by Memorandum Number 2.^15,16 Today, the VA is the largest educator of HCPs in the United States.¹⁷ In 2015, an estimated $10.3 to $12.5 billion was spent on mandatory Medicare graduate medical education (GME).¹⁸ In 2017, the VA spent $1.78 billion of discretionary funding on GME to fund 11,000 full-time equivalent (FTE) slots, leading to > 43,000 physician residents (> 30% of all physician residents) spending part of their training in a VHA facility.^18,19

This training mission has multiple benefits. It provides the VA with access to new HCPs who have the necessary training in veteran-specific needs, while supporting the national need for HCPs. In 2018, 120,890 clinical trainees received some or all of their training in the VA system.²⁰ Of the 152 US medical schools that are accredited by the Liaison Committee on Medical Education, 95% collaborate with the VA for training while 100% of the 34 doctor of osteopathic medicine programs have VA training collaborations.²⁰ The VA currently has an additional 18 partnerships with nursing schools.²¹ Further, 1,800 college and universities, including Hispanic-serving institutions and historically black colleges and universities, have VHA affiliations that provide training for more than 40 clinical health profession education programs.¹⁷

This training model has been successful in supporting VA staffing, as health care providers who trained in the VA are more likely to work in the VA.²² Among current VA employees, > 80% of optometrists, > 70% of podiatrists and psychologists, and > 60% of physicians received some part of their training in the VA system.²³ In combination with recent increased funding for staffing, the ability of the VA to directly hire trainees in identified professions, and the expansion of loan forgiveness to high-demand specialties (eg, psychiatry), the training partnership between the VA and affiliates has been critical in maintaining the needed VA workforce.^22,24,25

The VA Office of Academic Affiliations is responsible for all graduate medical and dental education administration in the VA system, which makes up 85% of its total budget. For each trainee, the VA provides approximately $60,000 toward their stipend in exchange for training and patient care time at a VHA hospital (Kenneth R. Jones, PhD, email communication, August 27, 2018).

California Health Care Education

The UC public university system, founded in 1869, currently has 10 campuses with a combined student body of > 280,000 students, along with 227,000 faculty and staff members.²⁶ For every research dollar provided by California, the UC secures $7 in federal and private funding.²⁶ The UC has 6 medical centers (Davis, Irvine, Los Angeles, Riverside, San Diego, and San Francisco); each is affiliated with at least 1 local VAMC.^27,28

California trains a substantial share of health care trainees. In 2016, there were 10,429 physician residents in training in California.²⁹ In 2017/2018, the San Francisco VAMC trained 1,178 medical students/residents, 57 pharmacy students, 25 nurse practitioner students, 19 optometry interns/students/residents, 11 dental students/residents, and 3 physical therapy students.²⁰ In total, 6,223 UC health professions students were trained in VHA facilities during the 2017/2018 training year (Table 4).²⁰ As of 2016, there were 105,907 physicians in California, and of those, 57% completed their GME in California.²⁹ In California in 2015, 74 GME-sponsoring institutions graduated 3,568 residents and fellows, an increase of 10% since 1997.³⁰ Of these sponsoring institutions, 6 of the top 8 programs were UC schools that graduated 48.4% (1,727) of all California residents and fellows in 2015.³⁰

VA Research Collaborations

The VA Office of Research and Development has existed for more than 90 years with a mission to improve veteran health and well-being via research and attract, train, and retain high-caliber researchers. VA provides a rich environment to conduct observational and interventional research due to its large, diverse veteran population, institutional support, and integrated information system with extensive EHR data.⁴¹ The success of the VA in facilitating research is evidenced by the fact that 3 VA investigators have been awarded Nobel prizes, and 7 have received Lasker Foundation Awards.⁴² The size of the VA allows for innovative large-scale research, such as the Million Veteran Program (MVP). The MVP study developed a mega-biobank of VA health records, questionnaires, and blood samples from nearly 1 million veterans to study genetic influences on health and disease and integrate genetic testing into health care delivery.⁴³ In addition to producing high-quality, innovative research, more than 60% of VA investigators also provide direct patient care.⁴²

VA research areas of focus include homelessness, polytrauma, traumatic brain injury, hearing and vision loss, spinal cord injury, mental health, pain management, precision medicine, prosthetics and amputation care, women’s health, and chronic diseases, such as Parkinson and Alzheimer diseases.⁴⁴ The VA estimates that, in 2021, total VA research spending will include a request of $787 million in addition to $370 million from the National Institutes of Health, the Department of Defense, and the Centers for Disease Control and Prevention, and $170 million from other nonfederal sources, for a projected total of $1.3 billion. This budget will support 2,200 projects with direct research and reimbursable employment of 3,275 FTEs,which are key to supporting VA academic affiliations.⁴⁵ These funds translate into substantial benefits to the UC system, including shared research and training resources, grant-funding opportunities for UC faculty, and the ability to recruit top researchers, educators, and clinicians to its institutions.

VA Reliance on Community Care 

The current VHA model is an integrated health care system that provides comprehensive, wraparound services to enrolled veterans, which are cost-effective, high quality, and consistently found to have equal or superior quality of care compared with that in the community.^6,46-50 Despite public criticism about wait times and access to care in the VA system, one study showed that VA wait-time statistics were comparable with or faster than those for community HCPs.^51,52 However, VA care coordination has undergone several changes to address these public criticisms, namely, the Veterans Access, Choice and Accountability Act of 2014 (38 USC § 1703 VACAA) and the VA MISSION Act of 2018 (42 USC § 274). VACAA was designed to increase access to care for veterans who live ≥ 40 miles from VA health care facilities or who are unable to been seen within 30 days of their preferred or clinically appropriate date.⁵³ More than 2 million veterans (almost 25% of VHA-enrolled veterans) have received community care since the inception of VACAA in 2014.⁵⁴

Recently, the MISSION Act mandated developing additional VA-coordinated community-based care through the establishment of a Veterans Community Care Program, which was established using existing VA 2019 fiscal year funds and did not include additional appropriations despite expanded criteria for community care referrals.⁵⁵ Without additional future appropriations, VA funds would be shifted from VA care into community care. While increasing access to community care has in some cases led to care that is faster and closer and that was previously inaccessible in local VA specialty care, these efforts could reduce veteran engagement with the VA system.⁵⁶

The changes implemented in VACAA and the VA MISSION Act were driven by important and valid concerns, including evidence of VA staff and officials covering up service deficiencies.⁵¹ Veterans in rural areas often have limited access to VA resources, and long travel to VAMCs or clinics can be an impediment. Veterans who have chosen community care tended to be those who have poorer health status, who live further away from VA facilities, women, and those who identified as White or Hispanic.^56,57 While VA health care is on average equivalent to or better than community resources, there is significant variability in quality within the VA system. Advocates have argued that providing competition and choice for veterans places pressure on the VA to improve care where it is not meeting expectations. Therefore, access to community care is an important resource for veterans and needs to be implemented effectively and efficiently to help veterans receive the care they need. However, expansion of community care access, depending on how it is implemented, also can have effects on academic partnerships and the education and research missions that should be incorporated into planning.

Each VA health care system receives funding through the Veterans Equitable Reimbursement Allocation (VERA), which provides funds largely based on the number of enrolled veterans and the complexity of the care they receive.⁵⁸ One potential implication of the shift among veterans to community care is a reduction in patients enrolled in VA programs, thus decreasing funding given to the VA to allocate for training and research. By definition, increased VA-managed community care means less opportunity for integrated training that brings together primary, mental health, and substance use care to meet patient needs. The Center for Medicare and Medicaid Services has developed a national initiative to help states develop programs in integrated care, particularly for individuals who are eligible for both Medicare and Medicaid.⁵⁹ For states to develop integrated care, they need trainees who function well in this model. Integrated care training is particularly vulnerable to disruption because any portion of a veteran’s care being transferred to the community can impede integration. In effect, training in integrated care, likely the most efficient and cost-effective approach to health care for reasons discussed earlier, could be reduced as providers and trainees are required to manage and coordinate patient care between separate institutions.³⁵

Educational Impact

The shift in usage from VA to community care has potential implications for academic affiliates, particularly in education and research.⁶⁰ If more people are served in community settings, potentially some VAMCs could be reduced, realigned, or closed. If this restructuring happens, academic partnerships could be impacted negatively. The VA is instituting an Infrastructure Review Commission with the task of examining current VA utilization. If a VA site with an academic affiliate was considered for realignment or closure, the reduction would eliminate the ability of the academic affiliate to provide education and research collaborations at that site.

In a less drastic manner, increasing care in the community may change opportunities for academic affiliates to partner with the VA. As noted, the UC system and California veterans benefit immensely from the VHA as an integrated health care system with dedicated missions of education and research. This partnership is a model in which the VA is the primary source of care for eligible enrolled veterans and provides integrated comprehensive services. If the VA moves to serving primarily as a coordinator of community HCPs rather than a direct provider of health care, academic affiliates would need to make major adjustments to both the education and training models. This change could particularly affect specialty training programs that rely on having adequate volumes of patients to provide an extensive experience to meet training needs. If fewer veterans receive care directly from the VA and are instead dispersed in the community, that will reduce the ability of academic faculty to participate in the education of medical and affiliated trainees and to participate in research in VA settings. It is unclear what other model could replace such a system and be as beneficial to the VA and the academic partners with which it is currently affiliated.

Given the needs that led to the VA increasing access to care and the potential implications discussed for the VA and partnerships with academic affiliates, VA health care systems and academic affiliate partners should consider several steps. These steps involve assessment, coordination, and promotion.

Both the VA and academic affiliates would benefit if the VA shared assessment data on the use of community care, particularly identifying changes that relate to key training and/or research missions. Such data sharing can be critical to determine whether any risks (or potential opportunities) need to be addressed. In addition, increasing research on the outcomes related to both VA care and community-based care is of high value to determine whether the current changes are achieving intended goals. The VA recently funded such work through its research service, and such work is critical for guiding future policy for the VA and for the affiliates.

Coordination among the VA, academic affiliates, and community partners is vital for change. The issue of community care expansion should be a standing item on coordination meetings and shared governance councils between the institutions. It may make sense to establish specific workgroups or committees to coordinate tracking and assessment of the effect of community care expansion on the shared academic mission. One way to address the potential effect of increased community care on the research and education missions would be to include community partners into the partnerships. This strategy could potentially take a number of different forms, from providing education and training to community HCPs, having VA trainees rotate to community settings, or inviting community settings to be research sites for clinical trials. Such partnerships could potentially improve patient care and support the other academic missions. Coordination could be meaningfully improved by having community HCPs access the VA EHR, thus easing communications. Funding is available for EHR access in the VA MISSION Act and should be a high priority as community care expands. The more that community partners can access and connect with the VA EHR the better they will be able to coordinate care.

Third, the VA and its academic partners need to promote and educate veterans, their families, and their advocates on the benefits that are available through VA care and that are enhanced through academic partnerships. While the VA has been the target of justified criticism, many of its strengths addressed here are not broadly recognized. The VA could promote its sharing of staff and resources with the top academic health care institutions in an area and that veterans often have access to resources that otherwise would not be available without the academic affiliate. Making sure veterans are aware of the benefits available can potentially mitigate the need for community care.

Conclusions

Given changes from VACAA and the VA MISSION Act, VA and academic affiliates should be active partners in planning for future health care by providing input and feedback on VA structure to help shape federal and state systems moving forward. Institutions can take steps to steer their futures and meet growing clinical, training, and research needs. The VA and its academic partners in health care research are well positioned to develop projects to assess the effects of these changes. Evaluation of key variables including patient care, education, and research productivity are warranted to guide policymakers as they assess whether these changes in the VA are achieving the expressed goals of improving veteran care. Other opportunities to collaborate in the wake of the MISSION Act remain to be discovered within each academic affiliation. By strengthening working relationships between VA and academic teams, these deeply important partnerships can continue to produce clinical, research, and education outcomes that meet the needs of our veterans, our federal and state health care systems, and our country.

Acknowledgments

Dr. Sells was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Academic Affiliations VA Quality Scholars Advanced Fellowship Program.

The Veterans Health Administration (VHA), 1 of 3 administrative branches in the US Department of Veterans Affairs (VA), is the largest integrated health care system in the United States.¹ The VHA has 4 missions: providing health care to eligible veterans; supporting research to benefit veterans and the larger society; providing education for health care trainees; and supporting emergency response.¹ In service of these goals, VA has academic affiliations with universities throughout the country, offering unique, extensive training and research opportunities. Both the VA and the affiliate benefit from these partnerships. For example, VA affiliations with University of California (UC) medical schools benefit veteran care while facilitating the UC academic mission. Through these affiliations, trainees who learn within the VHA’s highly effective integrated care model become health care professionals (HCPs) who are prepared to enter health care systems in California and meet the state’s demand for high-quality integrated care with an emphasis on primary care, mental health care, and care for aging populations.^2,3

This report explores the history of the VHA, current veteran demographics and needs, VA academic affiliations, and the integrated care model of training in all VHA facilities. The VA and UC academic affiliation is described further with regard to shared research and educational functions. Finally, we identify potential risks to academic affiliations associated with increased VA reliance on community-based care following the implementation of recent legislation. We provide suggestions for VA academic affiliates to help assess and guide the potential impact of increased VA-managed community care.

VHA Resources

The VHA serves more than 9 million veterans through 170 medical centers and 1,074 outpatient care sites.¹ In fiscal year 2017, the VA provided 109 million outpatient visits, and treated 615,000 inpatient medicine/surgical patients and 149,000 patients in inpatient mental health.⁴ The VHA focuses on the distinct concerns of veterans, which arise from military service as well as their broader health care needs. Veterans have higher rates of medical and mental health conditions than those of the general public; different cohorts in this population experience distinct medical and mental health concerns (Table 1).⁵

In addition, although veterans are disproportionately older men, the population is diversifying.⁶ For example, the number of female veterans is growing; furthermore, changes in the law now allow lesbian, gay, bisexual, and transgender (LGBT) individuals to serve openly, which has both reduced barriers for this population and allowed for LGBT veterans who were not eligible for VA care due to less than honorable discharges to have those discharges upgraded. As a result, care has been tailored to include the development of Women Veterans Program Managers and related services and LGBT and related identities resources such as LGBT Veteran Care Coordinators in every VA facility nationwide.^7,8 The VA continues to adapt to serve all veterans; part of this adaptation is training HCPs to provide veteran-centered care for a growing and diversifying population.

VHA Resources in California

California has the largest population of veterans in the United States (Table 2).^9,10 Of the 9,116,200 VA enrollees nationwide, 760,910 (8%) reside in California, and of those, 463,410 had at least 1 VA visit in the past year.^3,10 The VHA is organized into 21 Veterans Integrated Service Networks (VISNs) that include multiple health care systems in the region associated with each VISN. California is part of VISN 21 (Northern California, Nevada, and Pacific Islands) and VISN 22 (Southern California, Nevada, and New Mexico). Among veterans who served in the recent Iraq and Afghanistan conflicts, 5.5% accessed care in VISN 21 and 9.3% accessed care in VISN 22.¹¹ The VHA provides critical infrastructure for meeting complex veteran needs, as well as related specialized training, education, and research for HCPs. This specialization has been the basis for the broad system of affiliations between VA and academic systems.

The VA continues to be a high priority in the federal budget process.¹² In 2017, slightly more than 9% of the VA health care budget, $6.4 billion, was spent on medical care in California.¹⁰ Consequently, California has a noteworthy portion of VA infrastructure (Table 3).^13,14 California has 8 VA medical centers (VAMCs) with hospital service (Fresno, Loma Linda, Long Beach, Palo Alto, Sacramento, San Diego, San Francisco, West Los Angeles), 3 VAMCs without hospital service (2 locations in the Palo Alto system and Sepulveda), 1 stand-alone extended-care facility (Martinez Community Living Center), and 1 stand-alone residential care facility (San Diego Domiciliary).⁹ The vast VA infrastructure in California and large population of veterans creates a strong demand for HCPs in the state.

VA Education and Collaboration

VA has been training clinicians and scholars since 1946, when VA academic affiliations were established by Memorandum Number 2.^15,16 Today, the VA is the largest educator of HCPs in the United States.¹⁷ In 2015, an estimated $10.3 to $12.5 billion was spent on mandatory Medicare graduate medical education (GME).¹⁸ In 2017, the VA spent $1.78 billion of discretionary funding on GME to fund 11,000 full-time equivalent (FTE) slots, leading to > 43,000 physician residents (> 30% of all physician residents) spending part of their training in a VHA facility.^18,19

This training mission has multiple benefits. It provides the VA with access to new HCPs who have the necessary training in veteran-specific needs, while supporting the national need for HCPs. In 2018, 120,890 clinical trainees received some or all of their training in the VA system.²⁰ Of the 152 US medical schools that are accredited by the Liaison Committee on Medical Education, 95% collaborate with the VA for training while 100% of the 34 doctor of osteopathic medicine programs have VA training collaborations.²⁰ The VA currently has an additional 18 partnerships with nursing schools.²¹ Further, 1,800 college and universities, including Hispanic-serving institutions and historically black colleges and universities, have VHA affiliations that provide training for more than 40 clinical health profession education programs.¹⁷

This training model has been successful in supporting VA staffing, as health care providers who trained in the VA are more likely to work in the VA.²² Among current VA employees, > 80% of optometrists, > 70% of podiatrists and psychologists, and > 60% of physicians received some part of their training in the VA system.²³ In combination with recent increased funding for staffing, the ability of the VA to directly hire trainees in identified professions, and the expansion of loan forgiveness to high-demand specialties (eg, psychiatry), the training partnership between the VA and affiliates has been critical in maintaining the needed VA workforce.^22,24,25

The VA Office of Academic Affiliations is responsible for all graduate medical and dental education administration in the VA system, which makes up 85% of its total budget. For each trainee, the VA provides approximately $60,000 toward their stipend in exchange for training and patient care time at a VHA hospital (Kenneth R. Jones, PhD, email communication, August 27, 2018).

California Health Care Education

The UC public university system, founded in 1869, currently has 10 campuses with a combined student body of > 280,000 students, along with 227,000 faculty and staff members.²⁶ For every research dollar provided by California, the UC secures $7 in federal and private funding.²⁶ The UC has 6 medical centers (Davis, Irvine, Los Angeles, Riverside, San Diego, and San Francisco); each is affiliated with at least 1 local VAMC.^27,28

California trains a substantial share of health care trainees. In 2016, there were 10,429 physician residents in training in California.²⁹ In 2017/2018, the San Francisco VAMC trained 1,178 medical students/residents, 57 pharmacy students, 25 nurse practitioner students, 19 optometry interns/students/residents, 11 dental students/residents, and 3 physical therapy students.²⁰ In total, 6,223 UC health professions students were trained in VHA facilities during the 2017/2018 training year (Table 4).²⁰ As of 2016, there were 105,907 physicians in California, and of those, 57% completed their GME in California.²⁹ In California in 2015, 74 GME-sponsoring institutions graduated 3,568 residents and fellows, an increase of 10% since 1997.³⁰ Of these sponsoring institutions, 6 of the top 8 programs were UC schools that graduated 48.4% (1,727) of all California residents and fellows in 2015.³⁰

VA Research Collaborations

The VA Office of Research and Development has existed for more than 90 years with a mission to improve veteran health and well-being via research and attract, train, and retain high-caliber researchers. VA provides a rich environment to conduct observational and interventional research due to its large, diverse veteran population, institutional support, and integrated information system with extensive EHR data.⁴¹ The success of the VA in facilitating research is evidenced by the fact that 3 VA investigators have been awarded Nobel prizes, and 7 have received Lasker Foundation Awards.⁴² The size of the VA allows for innovative large-scale research, such as the Million Veteran Program (MVP). The MVP study developed a mega-biobank of VA health records, questionnaires, and blood samples from nearly 1 million veterans to study genetic influences on health and disease and integrate genetic testing into health care delivery.⁴³ In addition to producing high-quality, innovative research, more than 60% of VA investigators also provide direct patient care.⁴²

VA research areas of focus include homelessness, polytrauma, traumatic brain injury, hearing and vision loss, spinal cord injury, mental health, pain management, precision medicine, prosthetics and amputation care, women’s health, and chronic diseases, such as Parkinson and Alzheimer diseases.⁴⁴ The VA estimates that, in 2021, total VA research spending will include a request of $787 million in addition to $370 million from the National Institutes of Health, the Department of Defense, and the Centers for Disease Control and Prevention, and $170 million from other nonfederal sources, for a projected total of $1.3 billion. This budget will support 2,200 projects with direct research and reimbursable employment of 3,275 FTEs,which are key to supporting VA academic affiliations.⁴⁵ These funds translate into substantial benefits to the UC system, including shared research and training resources, grant-funding opportunities for UC faculty, and the ability to recruit top researchers, educators, and clinicians to its institutions.

VA Reliance on Community Care 

The current VHA model is an integrated health care system that provides comprehensive, wraparound services to enrolled veterans, which are cost-effective, high quality, and consistently found to have equal or superior quality of care compared with that in the community.^6,46-50 Despite public criticism about wait times and access to care in the VA system, one study showed that VA wait-time statistics were comparable with or faster than those for community HCPs.^51,52 However, VA care coordination has undergone several changes to address these public criticisms, namely, the Veterans Access, Choice and Accountability Act of 2014 (38 USC § 1703 VACAA) and the VA MISSION Act of 2018 (42 USC § 274). VACAA was designed to increase access to care for veterans who live ≥ 40 miles from VA health care facilities or who are unable to been seen within 30 days of their preferred or clinically appropriate date.⁵³ More than 2 million veterans (almost 25% of VHA-enrolled veterans) have received community care since the inception of VACAA in 2014.⁵⁴

Recently, the MISSION Act mandated developing additional VA-coordinated community-based care through the establishment of a Veterans Community Care Program, which was established using existing VA 2019 fiscal year funds and did not include additional appropriations despite expanded criteria for community care referrals.⁵⁵ Without additional future appropriations, VA funds would be shifted from VA care into community care. While increasing access to community care has in some cases led to care that is faster and closer and that was previously inaccessible in local VA specialty care, these efforts could reduce veteran engagement with the VA system.⁵⁶

The changes implemented in VACAA and the VA MISSION Act were driven by important and valid concerns, including evidence of VA staff and officials covering up service deficiencies.⁵¹ Veterans in rural areas often have limited access to VA resources, and long travel to VAMCs or clinics can be an impediment. Veterans who have chosen community care tended to be those who have poorer health status, who live further away from VA facilities, women, and those who identified as White or Hispanic.^56,57 While VA health care is on average equivalent to or better than community resources, there is significant variability in quality within the VA system. Advocates have argued that providing competition and choice for veterans places pressure on the VA to improve care where it is not meeting expectations. Therefore, access to community care is an important resource for veterans and needs to be implemented effectively and efficiently to help veterans receive the care they need. However, expansion of community care access, depending on how it is implemented, also can have effects on academic partnerships and the education and research missions that should be incorporated into planning.

Each VA health care system receives funding through the Veterans Equitable Reimbursement Allocation (VERA), which provides funds largely based on the number of enrolled veterans and the complexity of the care they receive.⁵⁸ One potential implication of the shift among veterans to community care is a reduction in patients enrolled in VA programs, thus decreasing funding given to the VA to allocate for training and research. By definition, increased VA-managed community care means less opportunity for integrated training that brings together primary, mental health, and substance use care to meet patient needs. The Center for Medicare and Medicaid Services has developed a national initiative to help states develop programs in integrated care, particularly for individuals who are eligible for both Medicare and Medicaid.⁵⁹ For states to develop integrated care, they need trainees who function well in this model. Integrated care training is particularly vulnerable to disruption because any portion of a veteran’s care being transferred to the community can impede integration. In effect, training in integrated care, likely the most efficient and cost-effective approach to health care for reasons discussed earlier, could be reduced as providers and trainees are required to manage and coordinate patient care between separate institutions.³⁵

Educational Impact

The shift in usage from VA to community care has potential implications for academic affiliates, particularly in education and research.⁶⁰ If more people are served in community settings, potentially some VAMCs could be reduced, realigned, or closed. If this restructuring happens, academic partnerships could be impacted negatively. The VA is instituting an Infrastructure Review Commission with the task of examining current VA utilization. If a VA site with an academic affiliate was considered for realignment or closure, the reduction would eliminate the ability of the academic affiliate to provide education and research collaborations at that site.

In a less drastic manner, increasing care in the community may change opportunities for academic affiliates to partner with the VA. As noted, the UC system and California veterans benefit immensely from the VHA as an integrated health care system with dedicated missions of education and research. This partnership is a model in which the VA is the primary source of care for eligible enrolled veterans and provides integrated comprehensive services. If the VA moves to serving primarily as a coordinator of community HCPs rather than a direct provider of health care, academic affiliates would need to make major adjustments to both the education and training models. This change could particularly affect specialty training programs that rely on having adequate volumes of patients to provide an extensive experience to meet training needs. If fewer veterans receive care directly from the VA and are instead dispersed in the community, that will reduce the ability of academic faculty to participate in the education of medical and affiliated trainees and to participate in research in VA settings. It is unclear what other model could replace such a system and be as beneficial to the VA and the academic partners with which it is currently affiliated.

Given the needs that led to the VA increasing access to care and the potential implications discussed for the VA and partnerships with academic affiliates, VA health care systems and academic affiliate partners should consider several steps. These steps involve assessment, coordination, and promotion.

Both the VA and academic affiliates would benefit if the VA shared assessment data on the use of community care, particularly identifying changes that relate to key training and/or research missions. Such data sharing can be critical to determine whether any risks (or potential opportunities) need to be addressed. In addition, increasing research on the outcomes related to both VA care and community-based care is of high value to determine whether the current changes are achieving intended goals. The VA recently funded such work through its research service, and such work is critical for guiding future policy for the VA and for the affiliates.

Coordination among the VA, academic affiliates, and community partners is vital for change. The issue of community care expansion should be a standing item on coordination meetings and shared governance councils between the institutions. It may make sense to establish specific workgroups or committees to coordinate tracking and assessment of the effect of community care expansion on the shared academic mission. One way to address the potential effect of increased community care on the research and education missions would be to include community partners into the partnerships. This strategy could potentially take a number of different forms, from providing education and training to community HCPs, having VA trainees rotate to community settings, or inviting community settings to be research sites for clinical trials. Such partnerships could potentially improve patient care and support the other academic missions. Coordination could be meaningfully improved by having community HCPs access the VA EHR, thus easing communications. Funding is available for EHR access in the VA MISSION Act and should be a high priority as community care expands. The more that community partners can access and connect with the VA EHR the better they will be able to coordinate care.

Third, the VA and its academic partners need to promote and educate veterans, their families, and their advocates on the benefits that are available through VA care and that are enhanced through academic partnerships. While the VA has been the target of justified criticism, many of its strengths addressed here are not broadly recognized. The VA could promote its sharing of staff and resources with the top academic health care institutions in an area and that veterans often have access to resources that otherwise would not be available without the academic affiliate. Making sure veterans are aware of the benefits available can potentially mitigate the need for community care.

Conclusions

Given changes from VACAA and the VA MISSION Act, VA and academic affiliates should be active partners in planning for future health care by providing input and feedback on VA structure to help shape federal and state systems moving forward. Institutions can take steps to steer their futures and meet growing clinical, training, and research needs. The VA and its academic partners in health care research are well positioned to develop projects to assess the effects of these changes. Evaluation of key variables including patient care, education, and research productivity are warranted to guide policymakers as they assess whether these changes in the VA are achieving the expressed goals of improving veteran care. Other opportunities to collaborate in the wake of the MISSION Act remain to be discovered within each academic affiliation. By strengthening working relationships between VA and academic teams, these deeply important partnerships can continue to produce clinical, research, and education outcomes that meet the needs of our veterans, our federal and state health care systems, and our country.

Acknowledgments

Dr. Sells was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Academic Affiliations VA Quality Scholars Advanced Fellowship Program.

The early phase of the COVID-19 pandemic was an extraordinarily uncertain, yet innovative, time.¹ Few data describe site-level effects of the many adaptations made to deal with surging case numbers, but studies of larger hospital referral regions (HRR) provide important clues.

In this issue of the Journal of Hospital Medicine, Janke et al² describe how availability of hospital resources in a region relate to COVID-19 mortality between March and June 2020.The authors’ findings suggest that, at least for early periods of the pandemic, having more intensive care unit (ICU), hospital bed, or nursing capacity per COVID-19 case was associated with lower mortality, while physician availability was not. Moreover, months later there were no associations between service or physician availability and HRR COVID-19 mortality. The authors observed variations in mortality rates in places commonly thought to have been overwhelmed early in the pandemic (April 2020), as well as in cities (Boston, Philadelphia, Hartford, Detroit, and Camden, New Jersey) that had a less prominent place in the news at that time.

Larger hospitals tend to have the resources necessary to make wholesale changes when preparing for a pandemic wave. Thus, Janke et al’s results may not have fully captured the pandemic’s potential impact in settings with fewer resources or in smaller hospitals, which are currently being overwhelmed.³

The number of cases and hospitalizations in this third wave of COVID-19 continues to rise, and the strain on healthcare resources has been felt across entire regions, making the results of this study even more salient. Hospital outcomes for COVID-19 are sensitive to limitations in physical locations (number of beds, ICU capacity) and nursing capacity. Nurses more often are assigned specifically to a bed or unit, and the number of patients per nurse is limited by state or local statute. Innovations such as COVID-19 field hospitals or redeploying existing beds (eg, converting postanesthesia care units to ICUs) offset physically constrained resources.⁴ On the other hand, lower acuity in this phase of the pandemic (eg, fewer ICU admissions) and shorter lengths of stay may produce higher turnover, producing more workforce stress, regardless of bed availability.

Early work of our COVID-19 collaborative⁵ suggests that the focus on localizing patients to geographic units or teams has given way to strategies that utilize more flexible team and bed-finding approaches. Clinical care has evolved to focus on more aggressive discharge strategies, with remote monitoring and hospital-at-home capabilities. Overall, the pandemic is providing fodder for future studies examining interaction between case volumes, physician and nurse availability, and evolution in clinical care practices. Most critically, it provides an opportunity to study health system flexibility and robustness with a lens that incorporates a view of the hospital and its surroundings as tightly related parts of care delivery. Because if there is one thing the pandemic is teaching us, it is that, more than ever, no hospital can be an island unto itself, and each hospital is part of a larger ecosystem where rising tides are felt throughout.

The early phase of the COVID-19 pandemic was an extraordinarily uncertain, yet innovative, time.¹ Few data describe site-level effects of the many adaptations made to deal with surging case numbers, but studies of larger hospital referral regions (HRR) provide important clues.

In this issue of the Journal of Hospital Medicine, Janke et al² describe how availability of hospital resources in a region relate to COVID-19 mortality between March and June 2020.The authors’ findings suggest that, at least for early periods of the pandemic, having more intensive care unit (ICU), hospital bed, or nursing capacity per COVID-19 case was associated with lower mortality, while physician availability was not. Moreover, months later there were no associations between service or physician availability and HRR COVID-19 mortality. The authors observed variations in mortality rates in places commonly thought to have been overwhelmed early in the pandemic (April 2020), as well as in cities (Boston, Philadelphia, Hartford, Detroit, and Camden, New Jersey) that had a less prominent place in the news at that time.

Larger hospitals tend to have the resources necessary to make wholesale changes when preparing for a pandemic wave. Thus, Janke et al’s results may not have fully captured the pandemic’s potential impact in settings with fewer resources or in smaller hospitals, which are currently being overwhelmed.³

The number of cases and hospitalizations in this third wave of COVID-19 continues to rise, and the strain on healthcare resources has been felt across entire regions, making the results of this study even more salient. Hospital outcomes for COVID-19 are sensitive to limitations in physical locations (number of beds, ICU capacity) and nursing capacity. Nurses more often are assigned specifically to a bed or unit, and the number of patients per nurse is limited by state or local statute. Innovations such as COVID-19 field hospitals or redeploying existing beds (eg, converting postanesthesia care units to ICUs) offset physically constrained resources.⁴ On the other hand, lower acuity in this phase of the pandemic (eg, fewer ICU admissions) and shorter lengths of stay may produce higher turnover, producing more workforce stress, regardless of bed availability.

Early work of our COVID-19 collaborative⁵ suggests that the focus on localizing patients to geographic units or teams has given way to strategies that utilize more flexible team and bed-finding approaches. Clinical care has evolved to focus on more aggressive discharge strategies, with remote monitoring and hospital-at-home capabilities. Overall, the pandemic is providing fodder for future studies examining interaction between case volumes, physician and nurse availability, and evolution in clinical care practices. Most critically, it provides an opportunity to study health system flexibility and robustness with a lens that incorporates a view of the hospital and its surroundings as tightly related parts of care delivery. Because if there is one thing the pandemic is teaching us, it is that, more than ever, no hospital can be an island unto itself, and each hospital is part of a larger ecosystem where rising tides are felt throughout.

The early phase of the COVID-19 pandemic was an extraordinarily uncertain, yet innovative, time.¹ Few data describe site-level effects of the many adaptations made to deal with surging case numbers, but studies of larger hospital referral regions (HRR) provide important clues.

In this issue of the Journal of Hospital Medicine, Janke et al² describe how availability of hospital resources in a region relate to COVID-19 mortality between March and June 2020.The authors’ findings suggest that, at least for early periods of the pandemic, having more intensive care unit (ICU), hospital bed, or nursing capacity per COVID-19 case was associated with lower mortality, while physician availability was not. Moreover, months later there were no associations between service or physician availability and HRR COVID-19 mortality. The authors observed variations in mortality rates in places commonly thought to have been overwhelmed early in the pandemic (April 2020), as well as in cities (Boston, Philadelphia, Hartford, Detroit, and Camden, New Jersey) that had a less prominent place in the news at that time.

Larger hospitals tend to have the resources necessary to make wholesale changes when preparing for a pandemic wave. Thus, Janke et al’s results may not have fully captured the pandemic’s potential impact in settings with fewer resources or in smaller hospitals, which are currently being overwhelmed.³

The number of cases and hospitalizations in this third wave of COVID-19 continues to rise, and the strain on healthcare resources has been felt across entire regions, making the results of this study even more salient. Hospital outcomes for COVID-19 are sensitive to limitations in physical locations (number of beds, ICU capacity) and nursing capacity. Nurses more often are assigned specifically to a bed or unit, and the number of patients per nurse is limited by state or local statute. Innovations such as COVID-19 field hospitals or redeploying existing beds (eg, converting postanesthesia care units to ICUs) offset physically constrained resources.⁴ On the other hand, lower acuity in this phase of the pandemic (eg, fewer ICU admissions) and shorter lengths of stay may produce higher turnover, producing more workforce stress, regardless of bed availability.

Early work of our COVID-19 collaborative⁵ suggests that the focus on localizing patients to geographic units or teams has given way to strategies that utilize more flexible team and bed-finding approaches. Clinical care has evolved to focus on more aggressive discharge strategies, with remote monitoring and hospital-at-home capabilities. Overall, the pandemic is providing fodder for future studies examining interaction between case volumes, physician and nurse availability, and evolution in clinical care practices. Most critically, it provides an opportunity to study health system flexibility and robustness with a lens that incorporates a view of the hospital and its surroundings as tightly related parts of care delivery. Because if there is one thing the pandemic is teaching us, it is that, more than ever, no hospital can be an island unto itself, and each hospital is part of a larger ecosystem where rising tides are felt throughout.

Several studies have examined variation in outcomes of patients with COVID-19, with emphasis on hospital-level factors such as geographic location, workforce and resource availability, and COVID-19 community prevalence.^1,2 Block et al¹ examine variation in COVID-19 mortality across 117 US hospitals, exploring whether COVID-19 admission volume was associated with mortality. While their results suggest that patients admitted to hospitals in the highest quintiles of COVID-19 caseload had higher odds of in-hospital death, the authors were not able to fully adjust for severity of illness, tempering our ability to draw conclusions. However, their finding is consistent with work showing that emergency department crowding and high hospital utilization are associated with excess mortality.

Block et al¹ also found variation within quintiles of COVID-19 burden, suggesting that other hospital-level factors are influencing their performance. In response to the initial surge of COVID-19 in the United States, hospitals and healthcare systems made rapid, often major, adjustments to provide care. Four interdependent components contribute to an effective surge response: system, space, staff, and supplies. Although all four components are important, effective systems are critical. Systems domains include command, or the creation of leadership teams throughout the organization; control, or management, of infrastructure; communication of rapid, comprehensible messages internally and externally; coordination of resources across departments and professions; and continuity of operations.³ Little is known about how well hospitals have implemented these systems components throughout the pandemic, and while Janke et al² examined the association of resources with outcomes, neither their study nor Block et al’s was able to account for other organizational or systems-based aspects of surge response.

Studies that help us understand the organizational factors and care-delivery adaptations associated with better outcomes for patients with COVID-19 are sorely needed, and could provide important insights for organizational adaptation and change more generally. Janke et al² and, in their accompanying editorial, Auerbach and Greysen,⁴ call for “innovative protocols” and “flexibility” to meet the needs of high-demand, novel situations. However, organizations’ ability to innovate and adapt relies on their relationships and teamwork capability.

The relational infrastructure within an organization provides the basis for effective teamwork, facilitating other aspects of an organization’s surge response and ability to adapt. Relationships characterized by trust and mindfulness create a context of psychological safety that encourages sharing new ideas, and enable teams to rapidly make sense of new situations and create shared understandings that facilitate effective action: improvising, adapting, and learning. Trust and psychological safety are especially important during crises, as decision-making tends to evolve toward top-down processes in times of crisis.

Hospitals currently collect few data that speak to relationships and teamwork, limiting our ability to study these vital organizational characteristics and their role in the larger COVID-19 response. Surveys related to patient safety culture or provider wellness and burnout are likely the only data regularly collected by hospitals. Expanding these data to include measures of relational infrastructure will create more robust data not only to conduct research regarding organizational factors that are associated with patient outcomes, but also to allow health systems to improve relationships and teaming as a means of improving outcomes. Examples include relational coordination,⁵ relationships,⁶and learning scales.⁷

The hospitals to which patients are admitted make a difference in patient survival. The study by Block et al¹ highlights the importance of assessing the factors that enable health systems to adapt and innovate so that we can better understand hospital-level variation in outcomes.

Several studies have examined variation in outcomes of patients with COVID-19, with emphasis on hospital-level factors such as geographic location, workforce and resource availability, and COVID-19 community prevalence.^1,2 Block et al¹ examine variation in COVID-19 mortality across 117 US hospitals, exploring whether COVID-19 admission volume was associated with mortality. While their results suggest that patients admitted to hospitals in the highest quintiles of COVID-19 caseload had higher odds of in-hospital death, the authors were not able to fully adjust for severity of illness, tempering our ability to draw conclusions. However, their finding is consistent with work showing that emergency department crowding and high hospital utilization are associated with excess mortality.

Block et al¹ also found variation within quintiles of COVID-19 burden, suggesting that other hospital-level factors are influencing their performance. In response to the initial surge of COVID-19 in the United States, hospitals and healthcare systems made rapid, often major, adjustments to provide care. Four interdependent components contribute to an effective surge response: system, space, staff, and supplies. Although all four components are important, effective systems are critical. Systems domains include command, or the creation of leadership teams throughout the organization; control, or management, of infrastructure; communication of rapid, comprehensible messages internally and externally; coordination of resources across departments and professions; and continuity of operations.³ Little is known about how well hospitals have implemented these systems components throughout the pandemic, and while Janke et al² examined the association of resources with outcomes, neither their study nor Block et al’s was able to account for other organizational or systems-based aspects of surge response.

Studies that help us understand the organizational factors and care-delivery adaptations associated with better outcomes for patients with COVID-19 are sorely needed, and could provide important insights for organizational adaptation and change more generally. Janke et al² and, in their accompanying editorial, Auerbach and Greysen,⁴ call for “innovative protocols” and “flexibility” to meet the needs of high-demand, novel situations. However, organizations’ ability to innovate and adapt relies on their relationships and teamwork capability.

The relational infrastructure within an organization provides the basis for effective teamwork, facilitating other aspects of an organization’s surge response and ability to adapt. Relationships characterized by trust and mindfulness create a context of psychological safety that encourages sharing new ideas, and enable teams to rapidly make sense of new situations and create shared understandings that facilitate effective action: improvising, adapting, and learning. Trust and psychological safety are especially important during crises, as decision-making tends to evolve toward top-down processes in times of crisis.

Hospitals currently collect few data that speak to relationships and teamwork, limiting our ability to study these vital organizational characteristics and their role in the larger COVID-19 response. Surveys related to patient safety culture or provider wellness and burnout are likely the only data regularly collected by hospitals. Expanding these data to include measures of relational infrastructure will create more robust data not only to conduct research regarding organizational factors that are associated with patient outcomes, but also to allow health systems to improve relationships and teaming as a means of improving outcomes. Examples include relational coordination,⁵ relationships,⁶and learning scales.⁷

The hospitals to which patients are admitted make a difference in patient survival. The study by Block et al¹ highlights the importance of assessing the factors that enable health systems to adapt and innovate so that we can better understand hospital-level variation in outcomes.

Several studies have examined variation in outcomes of patients with COVID-19, with emphasis on hospital-level factors such as geographic location, workforce and resource availability, and COVID-19 community prevalence.^1,2 Block et al¹ examine variation in COVID-19 mortality across 117 US hospitals, exploring whether COVID-19 admission volume was associated with mortality. While their results suggest that patients admitted to hospitals in the highest quintiles of COVID-19 caseload had higher odds of in-hospital death, the authors were not able to fully adjust for severity of illness, tempering our ability to draw conclusions. However, their finding is consistent with work showing that emergency department crowding and high hospital utilization are associated with excess mortality.

Block et al¹ also found variation within quintiles of COVID-19 burden, suggesting that other hospital-level factors are influencing their performance. In response to the initial surge of COVID-19 in the United States, hospitals and healthcare systems made rapid, often major, adjustments to provide care. Four interdependent components contribute to an effective surge response: system, space, staff, and supplies. Although all four components are important, effective systems are critical. Systems domains include command, or the creation of leadership teams throughout the organization; control, or management, of infrastructure; communication of rapid, comprehensible messages internally and externally; coordination of resources across departments and professions; and continuity of operations.³ Little is known about how well hospitals have implemented these systems components throughout the pandemic, and while Janke et al² examined the association of resources with outcomes, neither their study nor Block et al’s was able to account for other organizational or systems-based aspects of surge response.

Studies that help us understand the organizational factors and care-delivery adaptations associated with better outcomes for patients with COVID-19 are sorely needed, and could provide important insights for organizational adaptation and change more generally. Janke et al² and, in their accompanying editorial, Auerbach and Greysen,⁴ call for “innovative protocols” and “flexibility” to meet the needs of high-demand, novel situations. However, organizations’ ability to innovate and adapt relies on their relationships and teamwork capability.

The relational infrastructure within an organization provides the basis for effective teamwork, facilitating other aspects of an organization’s surge response and ability to adapt. Relationships characterized by trust and mindfulness create a context of psychological safety that encourages sharing new ideas, and enable teams to rapidly make sense of new situations and create shared understandings that facilitate effective action: improvising, adapting, and learning. Trust and psychological safety are especially important during crises, as decision-making tends to evolve toward top-down processes in times of crisis.

Hospitals currently collect few data that speak to relationships and teamwork, limiting our ability to study these vital organizational characteristics and their role in the larger COVID-19 response. Surveys related to patient safety culture or provider wellness and burnout are likely the only data regularly collected by hospitals. Expanding these data to include measures of relational infrastructure will create more robust data not only to conduct research regarding organizational factors that are associated with patient outcomes, but also to allow health systems to improve relationships and teaming as a means of improving outcomes. Examples include relational coordination,⁵ relationships,⁶and learning scales.⁷

The hospitals to which patients are admitted make a difference in patient survival. The study by Block et al¹ highlights the importance of assessing the factors that enable health systems to adapt and innovate so that we can better understand hospital-level variation in outcomes.

Since the onset of the COVID-19 pandemic, the proclivity of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) for adults and its relative sparing of pediatric populations has been well characterized. Accordingly, policymakers have devoted significant attention to SARS-CoV-2’s impact on adult hospitals. Less consideration, however, has been given to children’s hospitals, which responded to the pandemic by suspending noncritical care encounters, conserving personal protective equipment, and implementing alternative care models.¹ While important, these strategic decisions may threaten the financial health of children’s hospitals.

In this issue of the Journal of Hospital Medicine, Synhorst et al¹ describe the impact of COVID-19 on US children’s hospitals.The authors utilized the Children’s Hospital Association’s PROSPECT database to compare year-over-year trends in healthcare encounters and hospital charges before and during the COVID-19 pandemic at 26 tertiary hospitals. The analysis focused on the first wave of COVID-19 in the United States from February through June 2020.

The results are staggering. Compared with 2019, the authors found significant decreases in healthcare encounters for all children’s hospitals beginning in March 2020, with a nadir in mid-April (corresponding to the first peak in adult hospitalizations). Inpatient bed days, emergency department (ED) visits, and surgeries decreased by a median of 36%, 65%, and 77%, respectively, per hospital during the nadir. Charges from February 1 to June 30, 2020, decreased by a median 24% per children’s hospital as compared to 2019—corresponding to a median $276 million decrease in charges per hospital. A quarter of hospitals faced more than $400 million in lost charges.¹

Why do these trends matter? Large decreases in utilization and associated charges likely represent significant unmet demand for child healthcare for both acute and chronic disease management. For example, with limited in-person evaluation available at the onset of illness, caregivers are presenting to EDs with sicker children.² With a shift to virtual care, clinicians may miss signs of child abuse from violence in the home—which can escalate during isolation.³ Children with chronic conditions may also be left without surveillance mechanisms, which may partly explain the autumn 2020 surge in acute mental health-related ED presentations.⁴ Furthermore, telemedicine may exacerbate care inequities for vulnerable populations lacking resources and/or English proficiency.

There is also a larger policy perspective to consider in evaluating these data: Because children’s hospitals largely operate in a fee-for-service reimbursement model, they often rely on marginal revenues to support mission-driven programming. In other words, revenue streams from profitable care segments (eg, elective surgeries) often help sustain institutional platforms operating at a loss, such as community safety net programs. Consequently, threats to marginal revenues can place mission-driven programming in jeopardy of being reduced or terminated.

The Synhorst et al¹ study was limited to hospital charges, which likely overestimate revenue losses based on actual reimbursements. Yet, this is the first study to quantify COVID-19’s financial toll on children’s hospitals, and charges offer a reasonable proxy for balance sheet trends. Thus, it is safe to assume that most hospitals incurred substantial losses during the 2020 fiscal year. Unfortunately, as the authors highlight, these losses differentially impacted hospitals based on existing resources¹—so some hospitals were likely forced to cut programs or reduce staff in an effort to return to profitability. In this way, COVID-19 has exposed the fragility of the fee-for-service model that children’s hospitals rely on for both patients and staff.

Children’s hospitals and the services they provide are essential to the health and well-being of children. The critical losses sustained by children’s hospitals due to COVID-19 threaten their ability to promote child health in the near and long term, with the greatest risk to vulnerable populations. Policymakers must act now to preserve these essential services for children.

Since the onset of the COVID-19 pandemic, the proclivity of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) for adults and its relative sparing of pediatric populations has been well characterized. Accordingly, policymakers have devoted significant attention to SARS-CoV-2’s impact on adult hospitals. Less consideration, however, has been given to children’s hospitals, which responded to the pandemic by suspending noncritical care encounters, conserving personal protective equipment, and implementing alternative care models.¹ While important, these strategic decisions may threaten the financial health of children’s hospitals.

In this issue of the Journal of Hospital Medicine, Synhorst et al¹ describe the impact of COVID-19 on US children’s hospitals.The authors utilized the Children’s Hospital Association’s PROSPECT database to compare year-over-year trends in healthcare encounters and hospital charges before and during the COVID-19 pandemic at 26 tertiary hospitals. The analysis focused on the first wave of COVID-19 in the United States from February through June 2020.

The results are staggering. Compared with 2019, the authors found significant decreases in healthcare encounters for all children’s hospitals beginning in March 2020, with a nadir in mid-April (corresponding to the first peak in adult hospitalizations). Inpatient bed days, emergency department (ED) visits, and surgeries decreased by a median of 36%, 65%, and 77%, respectively, per hospital during the nadir. Charges from February 1 to June 30, 2020, decreased by a median 24% per children’s hospital as compared to 2019—corresponding to a median $276 million decrease in charges per hospital. A quarter of hospitals faced more than $400 million in lost charges.¹

Why do these trends matter? Large decreases in utilization and associated charges likely represent significant unmet demand for child healthcare for both acute and chronic disease management. For example, with limited in-person evaluation available at the onset of illness, caregivers are presenting to EDs with sicker children.² With a shift to virtual care, clinicians may miss signs of child abuse from violence in the home—which can escalate during isolation.³ Children with chronic conditions may also be left without surveillance mechanisms, which may partly explain the autumn 2020 surge in acute mental health-related ED presentations.⁴ Furthermore, telemedicine may exacerbate care inequities for vulnerable populations lacking resources and/or English proficiency.

There is also a larger policy perspective to consider in evaluating these data: Because children’s hospitals largely operate in a fee-for-service reimbursement model, they often rely on marginal revenues to support mission-driven programming. In other words, revenue streams from profitable care segments (eg, elective surgeries) often help sustain institutional platforms operating at a loss, such as community safety net programs. Consequently, threats to marginal revenues can place mission-driven programming in jeopardy of being reduced or terminated.

The Synhorst et al¹ study was limited to hospital charges, which likely overestimate revenue losses based on actual reimbursements. Yet, this is the first study to quantify COVID-19’s financial toll on children’s hospitals, and charges offer a reasonable proxy for balance sheet trends. Thus, it is safe to assume that most hospitals incurred substantial losses during the 2020 fiscal year. Unfortunately, as the authors highlight, these losses differentially impacted hospitals based on existing resources¹—so some hospitals were likely forced to cut programs or reduce staff in an effort to return to profitability. In this way, COVID-19 has exposed the fragility of the fee-for-service model that children’s hospitals rely on for both patients and staff.

Children’s hospitals and the services they provide are essential to the health and well-being of children. The critical losses sustained by children’s hospitals due to COVID-19 threaten their ability to promote child health in the near and long term, with the greatest risk to vulnerable populations. Policymakers must act now to preserve these essential services for children.

Since the onset of the COVID-19 pandemic, the proclivity of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) for adults and its relative sparing of pediatric populations has been well characterized. Accordingly, policymakers have devoted significant attention to SARS-CoV-2’s impact on adult hospitals. Less consideration, however, has been given to children’s hospitals, which responded to the pandemic by suspending noncritical care encounters, conserving personal protective equipment, and implementing alternative care models.¹ While important, these strategic decisions may threaten the financial health of children’s hospitals.

In this issue of the Journal of Hospital Medicine, Synhorst et al¹ describe the impact of COVID-19 on US children’s hospitals.The authors utilized the Children’s Hospital Association’s PROSPECT database to compare year-over-year trends in healthcare encounters and hospital charges before and during the COVID-19 pandemic at 26 tertiary hospitals. The analysis focused on the first wave of COVID-19 in the United States from February through June 2020.

The results are staggering. Compared with 2019, the authors found significant decreases in healthcare encounters for all children’s hospitals beginning in March 2020, with a nadir in mid-April (corresponding to the first peak in adult hospitalizations). Inpatient bed days, emergency department (ED) visits, and surgeries decreased by a median of 36%, 65%, and 77%, respectively, per hospital during the nadir. Charges from February 1 to June 30, 2020, decreased by a median 24% per children’s hospital as compared to 2019—corresponding to a median $276 million decrease in charges per hospital. A quarter of hospitals faced more than $400 million in lost charges.¹

Why do these trends matter? Large decreases in utilization and associated charges likely represent significant unmet demand for child healthcare for both acute and chronic disease management. For example, with limited in-person evaluation available at the onset of illness, caregivers are presenting to EDs with sicker children.² With a shift to virtual care, clinicians may miss signs of child abuse from violence in the home—which can escalate during isolation.³ Children with chronic conditions may also be left without surveillance mechanisms, which may partly explain the autumn 2020 surge in acute mental health-related ED presentations.⁴ Furthermore, telemedicine may exacerbate care inequities for vulnerable populations lacking resources and/or English proficiency.

There is also a larger policy perspective to consider in evaluating these data: Because children’s hospitals largely operate in a fee-for-service reimbursement model, they often rely on marginal revenues to support mission-driven programming. In other words, revenue streams from profitable care segments (eg, elective surgeries) often help sustain institutional platforms operating at a loss, such as community safety net programs. Consequently, threats to marginal revenues can place mission-driven programming in jeopardy of being reduced or terminated.

The Synhorst et al¹ study was limited to hospital charges, which likely overestimate revenue losses based on actual reimbursements. Yet, this is the first study to quantify COVID-19’s financial toll on children’s hospitals, and charges offer a reasonable proxy for balance sheet trends. Thus, it is safe to assume that most hospitals incurred substantial losses during the 2020 fiscal year. Unfortunately, as the authors highlight, these losses differentially impacted hospitals based on existing resources¹—so some hospitals were likely forced to cut programs or reduce staff in an effort to return to profitability. In this way, COVID-19 has exposed the fragility of the fee-for-service model that children’s hospitals rely on for both patients and staff.

Children’s hospitals and the services they provide are essential to the health and well-being of children. The critical losses sustained by children’s hospitals due to COVID-19 threaten their ability to promote child health in the near and long term, with the greatest risk to vulnerable populations. Policymakers must act now to preserve these essential services for children.

Study Overview

Objective. To evaluate the efficacy and safety of lenvatinib in combination with everolimus or pembrolizumab compared with sunitinib alone for the treatment of newly diagnosed advanced clear cell renal cell carcinoma (ccRCC).

Design. Global, multicenter, randomized, open-label, phase 3 trial.

Intervention. Patients were randomized in a 1:1:1 ratio to receive treatment with 1 of 3 regimens: lenvatinib 20 mg daily plus pembrolizumab 200 mg on day 1 of each 21-day cycle; lenvatinib 18 mg daily plus everolimus 5 mg once daily for each 21-day cycle; or sunitinib 50 mg daily for 4 weeks followed by 2 weeks off. Patients were stratified according to geographic region and Memorial Sloan Kettering Cancer Center (MSKCC) prognostic risk group.

Setting and participants. A total of 1417 patients were screened, and 1069 patients underwent randomization between October 2016 and July 2019: 355 patients were randomized to the lenvatinib plus pembrolizumab group, 357 were randomized to the lenvatinib plus everolimus group, and 357 were randomized to the sunitinib alone group. The patients must have had a diagnosis of previously untreated advanced renal cell carcinoma with a clear-cell component. All the patients need to have a Karnofsky performance status of at least 70, adequate renal function, and controlled blood pressure with or without antihypertensive medications.

Main outcome measures. The primary endpoint assessed the progression-free survival (PFS) as evaluated by independent review committee using RECIST, version 1.1. Imaging was performed at the time of screening and every 8 weeks thereafter. Secondary endpoints were safety, overall survival (OS), and objective response rate as well as investigator-assessed PFS. Also, they assessed the duration of response. During the treatment period, the safety and adverse events were assessed up to 30 days from the last dose of the trial drug.

Main results. The baseline characteristics were well balanced between the treatment groups. More than 70% of enrolled participants were male. Approximately 60% of participants were MSKCC intermediate risk, 27% were favorable risk, and 9% were poor risk. Patients with a PD-L1 combined positive score of 1% or more represented 30% of the population. The remainder had a PD-L1 combined positive score of <1% (30%) or such data were not available (38%). Liver metastases were present in 17% of patients at baseline in each group, and 70% of patients had a prior nephrectomy. The data cutoff occurred in August 2020 for PFS and the median follow-up for OS was 26.6 months. Around 40% of the participants in the lenvatinib plus pembrolizumab group, 18.8% in the sunitinib group, and 31% in the lenvatinib plus everolimus group were still receiving trial treatment at data cutoff. The leading cause for discontinuing therapy was disease progression. Approximately 50% of patients in the lenvatinib/everolimus group and sunitinib group received subsequent checkpoint inhibitor therapy after progression.

The median PFS in the lenvatinib plus pembrolizumab group was significantly longer than in the sunitinib group, 23.9 months vs 9.2 months (hazard ratio [HR], 0.39; 95% CI, 0.32-0.49; P < 0.001). The median PFS was also significantly longer in the lenvatinib plus everolimus group compared with sunitinib, 14.7 vs 9.2 months (HR 0.65; 95% CI 0.53-0.80; P < 0.001). The PFS benefit favored the lenvatinib combination groups over sunitinib in all subgroups, including the MSKCC prognostic risk groups. The median OS was not reached with any treatment, with 79% of patients in the lenvatinib plus pembrolizumab group, 66% of patients in the lenvatinib plus everolimus group, and 70% in the sunitinib group still alive at 24 months. Survival was significantly longer in the lenvatinib plus pembrolizumab group compared with sunitinib (HR, 0.66; 95% CI, 0.49-0.88; P = 0.005). The OS favored lenvatinib/pembrolizumab over sunitinib in the PD-L1 positive or negative groups. The median duration of response in the lenvatinib plus pembrolizumab group was 25.8 months compared to 16.6 months and 14.6 months in the lenvatinib plus everolimus and sunitinib groups, respectively. Complete response rates were higher in the lenvatinib plus pembrolizumab group (16%) compared with lenvatinib/everolimus (9.8%) or sunitinib (4.2%). The median time to response was around 1.9 months in all 3 groups.

The most frequent adverse events seen in all groups were diarrhea, hypertension, fatigue, and nausea. Hypothyroidism was seen more frequently in the lenvatinib plus pembrolizumab group (47%). Grade 3 adverse events were seen in approximately 80% of patients in all groups. The most common grade 3 or higher adverse event was hypertension in all 3 groups. The median time for discontinuing treatment due to side effects was 8.97 months in the lenvatinib plus pembrolizumab arm, 5.49 months in the lenvatinib plus everolimus group, and 4.57 months in the sunitinib group. In the lenvatinib plus pembrolizumab group, 15 patients had grade 5 adverse events; 11 participants had fatal events not related to disease progression. In the lenvatinib plus everolimus group, there were 22 patients with grade 5 events, with 10 fatal events not related to disease progression. In the sunitinib group, 11 patients had grade 5 events, and only 2 fatal events were not linked to disease progression.

Conclusion. The combination of lenvatinib plus pembrolizumab significantly prolongs PFS and OS compared with sunitinib in patients with previously untreated and advanced ccRCC. The median OS has not yet been reached.

Commentary

The results of the current phase 3 CLEAR trial highlight the efficacy and safety of lenvatinib plus pembrolizumab as a first-line treatment in advanced ccRCC. This trial adds to the rapidly growing body of literature supporting the notion that the combination of anti-PD-1 based therapy with either CTLA-4 antibodies or VEGF receptor tyrosine kinase inhibitors (TKI) improves outcomes in previously untreated patients with advanced ccRCC. Previously presented data from Keynote-426 (pembrolizumab plus axitinib), Checkmate-214 (nivolumab plus ipilimumab), and Javelin Renal 101 (Avelumab plus axitinib) have also shown improved outcomes with combination therapy in the frontline setting.^1-4 While the landscape of therapeutic options in the frontline setting continues to grow, there remains lack of clarity as to how to tailor our therapeutic decisions for specific patient populations. The exception would be nivolumab and ipilimumab, which are currently indicated for IMDC intermediate- or poor-risk patients.

The combination of VEGFR TKI therapy and PD-1 antibodies provides rapid disease control, with a median time to response in the current study of 1.9 months, and, generally speaking, a low risk of progression in the first 6 months of therapy. While cross-trial comparisons are always problematic, the PFS reported in this study and others with VEGFR TKI and PD-1 antibody combinations is quite impressive and surpasses that noted in Checkmate 214.³ While the median OS survival has not yet been reached, the long duration of PFS and complete response rate of 16% in this study certainly make this an attractive frontline option for newly diagnosed patients with advanced ccRCC. Longer follow-up is needed to confirm the survival benefit noted.

Applications for Clinical Practice

The current data support the use VEGFR TKI and anti-PD1 therapy in the frontline setting. How to choose between such combination regimens or combination immunotherapy remains unclear, and further biomarker-based assessments are needed to help guide therapeutic decisions for our patients.

Study Overview

Objective. To evaluate the efficacy and safety of lenvatinib in combination with everolimus or pembrolizumab compared with sunitinib alone for the treatment of newly diagnosed advanced clear cell renal cell carcinoma (ccRCC).

Design. Global, multicenter, randomized, open-label, phase 3 trial.

Intervention. Patients were randomized in a 1:1:1 ratio to receive treatment with 1 of 3 regimens: lenvatinib 20 mg daily plus pembrolizumab 200 mg on day 1 of each 21-day cycle; lenvatinib 18 mg daily plus everolimus 5 mg once daily for each 21-day cycle; or sunitinib 50 mg daily for 4 weeks followed by 2 weeks off. Patients were stratified according to geographic region and Memorial Sloan Kettering Cancer Center (MSKCC) prognostic risk group.

Setting and participants. A total of 1417 patients were screened, and 1069 patients underwent randomization between October 2016 and July 2019: 355 patients were randomized to the lenvatinib plus pembrolizumab group, 357 were randomized to the lenvatinib plus everolimus group, and 357 were randomized to the sunitinib alone group. The patients must have had a diagnosis of previously untreated advanced renal cell carcinoma with a clear-cell component. All the patients need to have a Karnofsky performance status of at least 70, adequate renal function, and controlled blood pressure with or without antihypertensive medications.

Main outcome measures. The primary endpoint assessed the progression-free survival (PFS) as evaluated by independent review committee using RECIST, version 1.1. Imaging was performed at the time of screening and every 8 weeks thereafter. Secondary endpoints were safety, overall survival (OS), and objective response rate as well as investigator-assessed PFS. Also, they assessed the duration of response. During the treatment period, the safety and adverse events were assessed up to 30 days from the last dose of the trial drug.

Main results. The baseline characteristics were well balanced between the treatment groups. More than 70% of enrolled participants were male. Approximately 60% of participants were MSKCC intermediate risk, 27% were favorable risk, and 9% were poor risk. Patients with a PD-L1 combined positive score of 1% or more represented 30% of the population. The remainder had a PD-L1 combined positive score of <1% (30%) or such data were not available (38%). Liver metastases were present in 17% of patients at baseline in each group, and 70% of patients had a prior nephrectomy. The data cutoff occurred in August 2020 for PFS and the median follow-up for OS was 26.6 months. Around 40% of the participants in the lenvatinib plus pembrolizumab group, 18.8% in the sunitinib group, and 31% in the lenvatinib plus everolimus group were still receiving trial treatment at data cutoff. The leading cause for discontinuing therapy was disease progression. Approximately 50% of patients in the lenvatinib/everolimus group and sunitinib group received subsequent checkpoint inhibitor therapy after progression.

The median PFS in the lenvatinib plus pembrolizumab group was significantly longer than in the sunitinib group, 23.9 months vs 9.2 months (hazard ratio [HR], 0.39; 95% CI, 0.32-0.49; P < 0.001). The median PFS was also significantly longer in the lenvatinib plus everolimus group compared with sunitinib, 14.7 vs 9.2 months (HR 0.65; 95% CI 0.53-0.80; P < 0.001). The PFS benefit favored the lenvatinib combination groups over sunitinib in all subgroups, including the MSKCC prognostic risk groups. The median OS was not reached with any treatment, with 79% of patients in the lenvatinib plus pembrolizumab group, 66% of patients in the lenvatinib plus everolimus group, and 70% in the sunitinib group still alive at 24 months. Survival was significantly longer in the lenvatinib plus pembrolizumab group compared with sunitinib (HR, 0.66; 95% CI, 0.49-0.88; P = 0.005). The OS favored lenvatinib/pembrolizumab over sunitinib in the PD-L1 positive or negative groups. The median duration of response in the lenvatinib plus pembrolizumab group was 25.8 months compared to 16.6 months and 14.6 months in the lenvatinib plus everolimus and sunitinib groups, respectively. Complete response rates were higher in the lenvatinib plus pembrolizumab group (16%) compared with lenvatinib/everolimus (9.8%) or sunitinib (4.2%). The median time to response was around 1.9 months in all 3 groups.

The most frequent adverse events seen in all groups were diarrhea, hypertension, fatigue, and nausea. Hypothyroidism was seen more frequently in the lenvatinib plus pembrolizumab group (47%). Grade 3 adverse events were seen in approximately 80% of patients in all groups. The most common grade 3 or higher adverse event was hypertension in all 3 groups. The median time for discontinuing treatment due to side effects was 8.97 months in the lenvatinib plus pembrolizumab arm, 5.49 months in the lenvatinib plus everolimus group, and 4.57 months in the sunitinib group. In the lenvatinib plus pembrolizumab group, 15 patients had grade 5 adverse events; 11 participants had fatal events not related to disease progression. In the lenvatinib plus everolimus group, there were 22 patients with grade 5 events, with 10 fatal events not related to disease progression. In the sunitinib group, 11 patients had grade 5 events, and only 2 fatal events were not linked to disease progression.

Conclusion. The combination of lenvatinib plus pembrolizumab significantly prolongs PFS and OS compared with sunitinib in patients with previously untreated and advanced ccRCC. The median OS has not yet been reached.

Commentary

The results of the current phase 3 CLEAR trial highlight the efficacy and safety of lenvatinib plus pembrolizumab as a first-line treatment in advanced ccRCC. This trial adds to the rapidly growing body of literature supporting the notion that the combination of anti-PD-1 based therapy with either CTLA-4 antibodies or VEGF receptor tyrosine kinase inhibitors (TKI) improves outcomes in previously untreated patients with advanced ccRCC. Previously presented data from Keynote-426 (pembrolizumab plus axitinib), Checkmate-214 (nivolumab plus ipilimumab), and Javelin Renal 101 (Avelumab plus axitinib) have also shown improved outcomes with combination therapy in the frontline setting.^1-4 While the landscape of therapeutic options in the frontline setting continues to grow, there remains lack of clarity as to how to tailor our therapeutic decisions for specific patient populations. The exception would be nivolumab and ipilimumab, which are currently indicated for IMDC intermediate- or poor-risk patients.

The combination of VEGFR TKI therapy and PD-1 antibodies provides rapid disease control, with a median time to response in the current study of 1.9 months, and, generally speaking, a low risk of progression in the first 6 months of therapy. While cross-trial comparisons are always problematic, the PFS reported in this study and others with VEGFR TKI and PD-1 antibody combinations is quite impressive and surpasses that noted in Checkmate 214.³ While the median OS survival has not yet been reached, the long duration of PFS and complete response rate of 16% in this study certainly make this an attractive frontline option for newly diagnosed patients with advanced ccRCC. Longer follow-up is needed to confirm the survival benefit noted.

Applications for Clinical Practice

The current data support the use VEGFR TKI and anti-PD1 therapy in the frontline setting. How to choose between such combination regimens or combination immunotherapy remains unclear, and further biomarker-based assessments are needed to help guide therapeutic decisions for our patients.

Study Overview

Objective. To evaluate the efficacy and safety of lenvatinib in combination with everolimus or pembrolizumab compared with sunitinib alone for the treatment of newly diagnosed advanced clear cell renal cell carcinoma (ccRCC).

Design. Global, multicenter, randomized, open-label, phase 3 trial.

Intervention. Patients were randomized in a 1:1:1 ratio to receive treatment with 1 of 3 regimens: lenvatinib 20 mg daily plus pembrolizumab 200 mg on day 1 of each 21-day cycle; lenvatinib 18 mg daily plus everolimus 5 mg once daily for each 21-day cycle; or sunitinib 50 mg daily for 4 weeks followed by 2 weeks off. Patients were stratified according to geographic region and Memorial Sloan Kettering Cancer Center (MSKCC) prognostic risk group.

Setting and participants. A total of 1417 patients were screened, and 1069 patients underwent randomization between October 2016 and July 2019: 355 patients were randomized to the lenvatinib plus pembrolizumab group, 357 were randomized to the lenvatinib plus everolimus group, and 357 were randomized to the sunitinib alone group. The patients must have had a diagnosis of previously untreated advanced renal cell carcinoma with a clear-cell component. All the patients need to have a Karnofsky performance status of at least 70, adequate renal function, and controlled blood pressure with or without antihypertensive medications.

Main outcome measures. The primary endpoint assessed the progression-free survival (PFS) as evaluated by independent review committee using RECIST, version 1.1. Imaging was performed at the time of screening and every 8 weeks thereafter. Secondary endpoints were safety, overall survival (OS), and objective response rate as well as investigator-assessed PFS. Also, they assessed the duration of response. During the treatment period, the safety and adverse events were assessed up to 30 days from the last dose of the trial drug.

Main results. The baseline characteristics were well balanced between the treatment groups. More than 70% of enrolled participants were male. Approximately 60% of participants were MSKCC intermediate risk, 27% were favorable risk, and 9% were poor risk. Patients with a PD-L1 combined positive score of 1% or more represented 30% of the population. The remainder had a PD-L1 combined positive score of <1% (30%) or such data were not available (38%). Liver metastases were present in 17% of patients at baseline in each group, and 70% of patients had a prior nephrectomy. The data cutoff occurred in August 2020 for PFS and the median follow-up for OS was 26.6 months. Around 40% of the participants in the lenvatinib plus pembrolizumab group, 18.8% in the sunitinib group, and 31% in the lenvatinib plus everolimus group were still receiving trial treatment at data cutoff. The leading cause for discontinuing therapy was disease progression. Approximately 50% of patients in the lenvatinib/everolimus group and sunitinib group received subsequent checkpoint inhibitor therapy after progression.

The median PFS in the lenvatinib plus pembrolizumab group was significantly longer than in the sunitinib group, 23.9 months vs 9.2 months (hazard ratio [HR], 0.39; 95% CI, 0.32-0.49; P < 0.001). The median PFS was also significantly longer in the lenvatinib plus everolimus group compared with sunitinib, 14.7 vs 9.2 months (HR 0.65; 95% CI 0.53-0.80; P < 0.001). The PFS benefit favored the lenvatinib combination groups over sunitinib in all subgroups, including the MSKCC prognostic risk groups. The median OS was not reached with any treatment, with 79% of patients in the lenvatinib plus pembrolizumab group, 66% of patients in the lenvatinib plus everolimus group, and 70% in the sunitinib group still alive at 24 months. Survival was significantly longer in the lenvatinib plus pembrolizumab group compared with sunitinib (HR, 0.66; 95% CI, 0.49-0.88; P = 0.005). The OS favored lenvatinib/pembrolizumab over sunitinib in the PD-L1 positive or negative groups. The median duration of response in the lenvatinib plus pembrolizumab group was 25.8 months compared to 16.6 months and 14.6 months in the lenvatinib plus everolimus and sunitinib groups, respectively. Complete response rates were higher in the lenvatinib plus pembrolizumab group (16%) compared with lenvatinib/everolimus (9.8%) or sunitinib (4.2%). The median time to response was around 1.9 months in all 3 groups.

The most frequent adverse events seen in all groups were diarrhea, hypertension, fatigue, and nausea. Hypothyroidism was seen more frequently in the lenvatinib plus pembrolizumab group (47%). Grade 3 adverse events were seen in approximately 80% of patients in all groups. The most common grade 3 or higher adverse event was hypertension in all 3 groups. The median time for discontinuing treatment due to side effects was 8.97 months in the lenvatinib plus pembrolizumab arm, 5.49 months in the lenvatinib plus everolimus group, and 4.57 months in the sunitinib group. In the lenvatinib plus pembrolizumab group, 15 patients had grade 5 adverse events; 11 participants had fatal events not related to disease progression. In the lenvatinib plus everolimus group, there were 22 patients with grade 5 events, with 10 fatal events not related to disease progression. In the sunitinib group, 11 patients had grade 5 events, and only 2 fatal events were not linked to disease progression.

Conclusion. The combination of lenvatinib plus pembrolizumab significantly prolongs PFS and OS compared with sunitinib in patients with previously untreated and advanced ccRCC. The median OS has not yet been reached.

Commentary

The results of the current phase 3 CLEAR trial highlight the efficacy and safety of lenvatinib plus pembrolizumab as a first-line treatment in advanced ccRCC. This trial adds to the rapidly growing body of literature supporting the notion that the combination of anti-PD-1 based therapy with either CTLA-4 antibodies or VEGF receptor tyrosine kinase inhibitors (TKI) improves outcomes in previously untreated patients with advanced ccRCC. Previously presented data from Keynote-426 (pembrolizumab plus axitinib), Checkmate-214 (nivolumab plus ipilimumab), and Javelin Renal 101 (Avelumab plus axitinib) have also shown improved outcomes with combination therapy in the frontline setting.^1-4 While the landscape of therapeutic options in the frontline setting continues to grow, there remains lack of clarity as to how to tailor our therapeutic decisions for specific patient populations. The exception would be nivolumab and ipilimumab, which are currently indicated for IMDC intermediate- or poor-risk patients.

The combination of VEGFR TKI therapy and PD-1 antibodies provides rapid disease control, with a median time to response in the current study of 1.9 months, and, generally speaking, a low risk of progression in the first 6 months of therapy. While cross-trial comparisons are always problematic, the PFS reported in this study and others with VEGFR TKI and PD-1 antibody combinations is quite impressive and surpasses that noted in Checkmate 214.³ While the median OS survival has not yet been reached, the long duration of PFS and complete response rate of 16% in this study certainly make this an attractive frontline option for newly diagnosed patients with advanced ccRCC. Longer follow-up is needed to confirm the survival benefit noted.

Applications for Clinical Practice

The current data support the use VEGFR TKI and anti-PD1 therapy in the frontline setting. How to choose between such combination regimens or combination immunotherapy remains unclear, and further biomarker-based assessments are needed to help guide therapeutic decisions for our patients.