Venous thromboembolism (VTE), comprising deep venous thrombosis and pulmonary embolism (PE),¹ is a serious medical condition that results in preventable morbidity and mortality.^1-5 VTE affects all age groups, all races/ethnicities, and both genders, but there are known factors that increase the risk of developing VTE (eg, advanced age, undergoing surgery, hospitalization, and immobility).^1-3,5-7 Prevention of VTE among hospitalized patients is of paramount importance to avoid preventable death, chronic illness/long-term complications,⁸ longer hospital stays, and increased hospital costs.⁹ Fortunately, there is clear evidence that provision of appropriate prophylaxis can decrease the risk of a VTE event occurring, and broadly accepted best-practice guidelines reflect this evidence.^3,5

Given the inadequacy of current VTE-related quality measures to identify actionable failures in the provision of VTE prophylaxis, our group created a VTE process-of-care measure to assess adherence to the components of VTE prophylaxis: (1) early ambulation, (2) mechanical prophylaxis (sequential compression devices [SCDs]), and (3) chemoprophylaxis administered at the correct dose and frequency for the duration of the patient’s hospital stay.^3,10,11 This quality measure was conceived, created, and iteratively revised to measure whether optimal care is provided to patients throughout their hospitalization and identify actionable areas in which failures of care occur, in order to decrease the risk of a VTE event. Data from our institution provided evidence that while ambulation and SCD component measure adherence is high, chemoprophylaxis adherence required significant improvement.¹⁰ When chemoprophylaxis process measure adherence data were analyzed further, a major failure mode was patient refusal of one or more doses. However, the drivers of patient refusal are not well defined in the literature, and previous studies have called for a greater focus on developing interventions to improve VTE chemoprophylaxis administration.¹²

Previous research has shown that nurses can influence patient compliance with VTE prophylaxis.^13-15 A mixed-methods study by Elder et al. found that nurses in units with high rates of failure to provide optimal chemoprophylaxis offered the medication as optional, leading researchers to conclude that nurses perceived chemoprophylaxis as discretionary.¹³ Another study by Lee et al., conducted a survey of bedside registered nurses and identified nurses’ lack of education on VTE prevention as a significant barrier to providing care.¹⁴ These studies show that multiple levels of influence impact how nurses provide VTE chemoprophylaxis, particularly when they encounter patients who refuse chemoprophylaxis.

To explore the nuance and interplay of multiple influences, we used the Theoretical Domains Framework (TDF), an integrative framework that applies theoretical approaches to interventions aimed at behavior change.^15-18 The framework contains 14 interrelated domains that characterize the behavior being studied, in this case, administration of VTE chemoprophylaxis. Consequently, we designed a nurse-focused, qualitative evaluation with the objective to identify nursing-related barriers to administration of VTE chemoprophylaxis.

Inpatient Unit Selection

The study team accessed data from the hospital’s Enterprise Data Warehouse to review patient refusal rates of VTE chemoprophylaxis for each inpatient unit in the hospital. Patient refusal was utilized as a proxy measure for the behavior of nurses attempting to administer VTE chemoprophylaxis. Of the 14 medical and surgical units in the hospital, two medical and two surgical units were selected to participate in the qualitative evaluation based on having the highest patient refusal rates. One unit (surgical) was also selected to serve as a benchmark because it had the lowest patient refusal rate. Table 1 includes the refusal rates for the five units. Given the low refusal rate for the best performing unit, we suspected that it would be possible to decrease the patient refusal rate for other units with similar patient populations and interprofessional teams at the institution.

Observations

We observed chemoprophylaxis administration on the five units to understand the process for ordering and administering chemoprophylaxis. An observation protocol was utilized to document the date, time, and location of the observation as well as descriptive notes including accounts of particular events.^19,20 Observations occurred in May 2016 and informed the creation of a process map outlining the procedure for ordering and administering VTE chemoprophylaxis. The process map was utilized to create the focus group interview guide and ensure the interview guide included pertinent questions for each step of the process (Appendix A).

Focus Group Interviews

We conducted focus group interviews with day and night shift nurses on the five units to assess nurses’ understanding of VTE chemoprophylaxis and nurses’ perceptions of barriers to administration of VTE chemoprophylaxis. The study team chose to conduct focus group interviews in an effort to maximize participation and to speak with multiple nurses within a shorter period of time. The focus group structure allowed the study team to speak with nurses during their shifts, as one could briefly step out, if required, for patient care and return to rejoin the discussion.

We developed a semistructured interview guide²¹ with questions focused on identifying nurses’ perceptions of guideline-recommended care for VTE chemoprophylaxis, where they learned these guidelines, how nurses discuss chemoprophylaxis with patients, how they handle the conversation with patients who refuse, and if there are times when chemoprophylaxis is not necessary. The interview guide was vetted by a multidisciplinary team consisting of clinical nursing coordinators and nurse managers from medical and surgical units, hospital quality leaders, surgeons and general internists, and qualitative research experts. The interview guide is included as Appendix B.

The unit clinical coordinators and nurse managers identified dates and times for the focus groups that would be minimally disruptive to the unit. For each of the four units with a high patient refusal rate, two focus groups were conducted during the lunch hour and one was conducted at the end of the night shift to ensure that both day and night shift nurses were included in the study. Two focus groups were conducted with the best-practice unit during the lunch hour. For each focus group, the clinical coordinator identified two to eight nurses who could step away from patient care to participate or who had completed their shifts. In total, approximately 67 nurses participated in the focus groups.

The focus groups (n = 14) lasted approximately 40 minutes during May and June 2016. Two members of the study team cofacilitated interviews, which were recorded and transcribed verbatim.

Coding and Data Analysis

To develop the code book, the study team, consisting of three qualitative researchers, independently read one focus group transcript and applied the TDF domains to the nurses’ perceptions of barriers to administration of VTE chemoprophylaxis.^21-24 In addition to coding by domain, the study team also coded nursing perceptions as barriers or facilitators. The study team reviewed the coded transcript and reconciled any differences in coding. This process was repeated for a second transcript, and then all remaining transcripts were assigned to two out of three study team members for coding, with the entire study team meeting to reconcile any differences. If necessary, the team member who did not code a transcript acted as the tie-breaker if there were discrepancies in codes that could not be reconciled.

Once coding was completed, we identified the TDF domains that were most relevant to the administration of VTE chemoprophylaxis.¹⁶ Member checking (testing the analysis, interpretations, and conclusions with members of those groups from whom the data were originally obtained) was performed with the four clinical nursing coordinators and four nurse managers from the participating units to establish face validity of the themes identified from the focus group interviews.²⁵

The study team used MaxQDA, V12 (Berlin, Germany) to support data coding and analysis.²⁶ The Northwestern University institutional review board office deemed this project research on nonhuman-subjects because it focused on the process of providing VTE chemoprophylaxis and not about the patients themselves. The purpose of the study was explained at the beginning of each focus group, and nurses gave verbal consent to have the focus group recorded.

We conducted 14 focus groups with day and night shift nurses from five units (two medical and three surgical) at a single institution. All nurses invited to participate in a focus group agreed to participate. The data were coded and grouped by domain and identified as barriers or facilitators. The findings included below are for the domains most relevant to the provision of VTE prophylaxis. Table 2 provides illustrative verbatim quotes for each domain that was represented in the focus groups.

Knowledge

All interviewees recognized that providing some form of prophylaxis to mitigate the risk of a VTE event is essential. Some nurses stated that seeing a patient ambulating meant they would consider not administering prescribed chemoprophylaxis, while others would try to negotiate with patients by asking the patient to allow one dose of chemoprophylaxis prescribed two to three times daily because it was better than receiving no doses.

Environmental Context and Resources

Multiple barriers to providing optimal care were associated with the environmental context and a lack of resources. There was a lack of accessible, comprehensive, patient-centered education materials on VTE chemoprophylaxis to supplement a nurse’s explanation about the importance of chemoprophylaxis. Furthermore, many nurses cited the perceived patient pain of chemoprophylaxis injections as the main deterrent to patient compliance, especially subcutaneous heparin injections, which occur up to three times in 24 hours and often cause more pain at the site of injection than low-molecular-weight heparin. Nurses felt that transitioning patients from receiving subcutaneous heparin injections to receiving low-molecular-weight heparin could be a main driver to reduce patient refusals.

Skills

Nurses felt inadequately equipped to handle patient refusals. Many said that patient refusal of treatments was never discussed in nursing school. As a result, when patients refused treatments, the nurses did not know how to handle the situation. They felt that they lacked the tools and techniques to persuade the patient to comply.

Beliefs about Capabilities

Nurses did not know their own patient refusal rate or benchmarks of an acceptable refusal rate in contrast to one that is too high. Without this feedback, they were unable to assess their own behavior or performance related to providing VTE chemoprophylaxis.

Nurses play a critical role in providing VTE chemoprophylaxis to patients throughout their hospitalization. This study provided a unique opportunity to perform an in-depth, qualitative analysis of the barriers nurses face in providing patients with VTE chemoprophylaxis as part of their daily work caring for patients. We discovered several nursing-related barriers to the provision of VTE chemoprophylaxis, including lack of knowledge, resources, skill, and misconceptions of their capability to provide VTE chemoprophylaxis. We used a bottom-up approach by incorporating the voices of unit nurses, clinical coordinators, and nurse managers to understand potential barriers. Our findings brought to light the challenge of delivering standardized care in an area of care that is generally agreed upon, yet not fully followed. Some nurses display greater proficiency than others at communicating with patients who do not understand their risk for VTE and need for chemoprophylaxis. Furthermore, there is a pronounced misconception around the delivery of VTE chemoprophylaxis. Nurses have the inaccurate belief that even if ordered, chemoprophylaxis is not required. This misconception was widespread among nurses taking care of both medical and surgical patients. These factors appear to be modifiable targets for quality improvement and highlight the need for a skills-based education during the new hire onboarding process, as well as ongoing reeducation to ensure nursing staff have the skills to appropriately provide best-practice care for VTE chemoprophylaxis. Nurses felt ownership of the results of the qualitative evaluation because they were included in every aspect from the beginning.²⁷ This sense of ownership will support future quality improvement efforts to develop a skills-based intervention to improve the provision of VTE chemoprophylaxis.^18,27

This study has certain limitations. First, it was a qualitative study assessing nursing-related barriers to providing VTE chemoprophylaxis at a single institution, and the results cannot be generalized broadly. However, the techniques and results are transferable to other hospital settings and other clinical care situations. Thus, we believe that other institutions can utilize our methods and that similar lessons can be learned and applied. Furthermore, the validity of our study is bolstered by concordance between the results of this study and those of other studies conducted on the topic of provision of VTE prophylaxis by nurses.^13-15,21 Other studies utilized observations and surveys to determine potential nurse-related barriers to the provision of VTE prophylaxis, such as lack of knowledge and the belief that the need for prophylaxis can be determined based on whether or not the patient is ambulating;^13,14 however, by utilizing focus group interviews, we allowed nurses to speak in their own voices about their experiences with VTE prophylaxis, and we were able to delve deeper and identify additional barriers that emerged from discussions with nurses, such as the lack of skill and misconceptions of capability.^28,29 Second, the study focused solely on nurses. Additional initiatives are underway to assess the roles of resident physicians, attending physicians, and patients in the provision of VTE prophylaxis.

Nursing-related barriers to the provision of VTE chemoprophylaxis include a lack of knowledge, resources, skills, and misconceptions of the consequences of missed elements of VTE prophylaxis. Future initiatives will focus on equipping nurses to have meaningful conversations with patients and engaging patients in their care through development of a multifaceted bundle of interventions. Furthermore, similar methods of qualitative inquiry will be used to identify the role of resident and attending physicians and patients in the provision of VTE chemoprophylaxis.

Acknowledgments

The authors thank Sonali Oberoi, Joanne Prinz, Nancy Tomaska, and Kate Paredes, as well as all the nurses who participated in focus group interviews for this study and the nurse managers and clinical coordinators who helped to schedule the focus group interviews.

Disclosures

The authors declare that they have no competing interests.

Funding

This study was funded by the Surgical Outcomes and Quality Improvement Center at Northwestern University.

Venous thromboembolism (VTE), comprising deep venous thrombosis and pulmonary embolism (PE),¹ is a serious medical condition that results in preventable morbidity and mortality.^1-5 VTE affects all age groups, all races/ethnicities, and both genders, but there are known factors that increase the risk of developing VTE (eg, advanced age, undergoing surgery, hospitalization, and immobility).^1-3,5-7 Prevention of VTE among hospitalized patients is of paramount importance to avoid preventable death, chronic illness/long-term complications,⁸ longer hospital stays, and increased hospital costs.⁹ Fortunately, there is clear evidence that provision of appropriate prophylaxis can decrease the risk of a VTE event occurring, and broadly accepted best-practice guidelines reflect this evidence.^3,5

Given the inadequacy of current VTE-related quality measures to identify actionable failures in the provision of VTE prophylaxis, our group created a VTE process-of-care measure to assess adherence to the components of VTE prophylaxis: (1) early ambulation, (2) mechanical prophylaxis (sequential compression devices [SCDs]), and (3) chemoprophylaxis administered at the correct dose and frequency for the duration of the patient’s hospital stay.^3,10,11 This quality measure was conceived, created, and iteratively revised to measure whether optimal care is provided to patients throughout their hospitalization and identify actionable areas in which failures of care occur, in order to decrease the risk of a VTE event. Data from our institution provided evidence that while ambulation and SCD component measure adherence is high, chemoprophylaxis adherence required significant improvement.¹⁰ When chemoprophylaxis process measure adherence data were analyzed further, a major failure mode was patient refusal of one or more doses. However, the drivers of patient refusal are not well defined in the literature, and previous studies have called for a greater focus on developing interventions to improve VTE chemoprophylaxis administration.¹²

Previous research has shown that nurses can influence patient compliance with VTE prophylaxis.^13-15 A mixed-methods study by Elder et al. found that nurses in units with high rates of failure to provide optimal chemoprophylaxis offered the medication as optional, leading researchers to conclude that nurses perceived chemoprophylaxis as discretionary.¹³ Another study by Lee et al., conducted a survey of bedside registered nurses and identified nurses’ lack of education on VTE prevention as a significant barrier to providing care.¹⁴ These studies show that multiple levels of influence impact how nurses provide VTE chemoprophylaxis, particularly when they encounter patients who refuse chemoprophylaxis.

To explore the nuance and interplay of multiple influences, we used the Theoretical Domains Framework (TDF), an integrative framework that applies theoretical approaches to interventions aimed at behavior change.^15-18 The framework contains 14 interrelated domains that characterize the behavior being studied, in this case, administration of VTE chemoprophylaxis. Consequently, we designed a nurse-focused, qualitative evaluation with the objective to identify nursing-related barriers to administration of VTE chemoprophylaxis.

Inpatient Unit Selection

The study team accessed data from the hospital’s Enterprise Data Warehouse to review patient refusal rates of VTE chemoprophylaxis for each inpatient unit in the hospital. Patient refusal was utilized as a proxy measure for the behavior of nurses attempting to administer VTE chemoprophylaxis. Of the 14 medical and surgical units in the hospital, two medical and two surgical units were selected to participate in the qualitative evaluation based on having the highest patient refusal rates. One unit (surgical) was also selected to serve as a benchmark because it had the lowest patient refusal rate. Table 1 includes the refusal rates for the five units. Given the low refusal rate for the best performing unit, we suspected that it would be possible to decrease the patient refusal rate for other units with similar patient populations and interprofessional teams at the institution.

Observations

We observed chemoprophylaxis administration on the five units to understand the process for ordering and administering chemoprophylaxis. An observation protocol was utilized to document the date, time, and location of the observation as well as descriptive notes including accounts of particular events.^19,20 Observations occurred in May 2016 and informed the creation of a process map outlining the procedure for ordering and administering VTE chemoprophylaxis. The process map was utilized to create the focus group interview guide and ensure the interview guide included pertinent questions for each step of the process (Appendix A).

Focus Group Interviews

We conducted focus group interviews with day and night shift nurses on the five units to assess nurses’ understanding of VTE chemoprophylaxis and nurses’ perceptions of barriers to administration of VTE chemoprophylaxis. The study team chose to conduct focus group interviews in an effort to maximize participation and to speak with multiple nurses within a shorter period of time. The focus group structure allowed the study team to speak with nurses during their shifts, as one could briefly step out, if required, for patient care and return to rejoin the discussion.

We developed a semistructured interview guide²¹ with questions focused on identifying nurses’ perceptions of guideline-recommended care for VTE chemoprophylaxis, where they learned these guidelines, how nurses discuss chemoprophylaxis with patients, how they handle the conversation with patients who refuse, and if there are times when chemoprophylaxis is not necessary. The interview guide was vetted by a multidisciplinary team consisting of clinical nursing coordinators and nurse managers from medical and surgical units, hospital quality leaders, surgeons and general internists, and qualitative research experts. The interview guide is included as Appendix B.

The unit clinical coordinators and nurse managers identified dates and times for the focus groups that would be minimally disruptive to the unit. For each of the four units with a high patient refusal rate, two focus groups were conducted during the lunch hour and one was conducted at the end of the night shift to ensure that both day and night shift nurses were included in the study. Two focus groups were conducted with the best-practice unit during the lunch hour. For each focus group, the clinical coordinator identified two to eight nurses who could step away from patient care to participate or who had completed their shifts. In total, approximately 67 nurses participated in the focus groups.

The focus groups (n = 14) lasted approximately 40 minutes during May and June 2016. Two members of the study team cofacilitated interviews, which were recorded and transcribed verbatim.

Coding and Data Analysis

To develop the code book, the study team, consisting of three qualitative researchers, independently read one focus group transcript and applied the TDF domains to the nurses’ perceptions of barriers to administration of VTE chemoprophylaxis.^21-24 In addition to coding by domain, the study team also coded nursing perceptions as barriers or facilitators. The study team reviewed the coded transcript and reconciled any differences in coding. This process was repeated for a second transcript, and then all remaining transcripts were assigned to two out of three study team members for coding, with the entire study team meeting to reconcile any differences. If necessary, the team member who did not code a transcript acted as the tie-breaker if there were discrepancies in codes that could not be reconciled.

Once coding was completed, we identified the TDF domains that were most relevant to the administration of VTE chemoprophylaxis.¹⁶ Member checking (testing the analysis, interpretations, and conclusions with members of those groups from whom the data were originally obtained) was performed with the four clinical nursing coordinators and four nurse managers from the participating units to establish face validity of the themes identified from the focus group interviews.²⁵

The study team used MaxQDA, V12 (Berlin, Germany) to support data coding and analysis.²⁶ The Northwestern University institutional review board office deemed this project research on nonhuman-subjects because it focused on the process of providing VTE chemoprophylaxis and not about the patients themselves. The purpose of the study was explained at the beginning of each focus group, and nurses gave verbal consent to have the focus group recorded.

We conducted 14 focus groups with day and night shift nurses from five units (two medical and three surgical) at a single institution. All nurses invited to participate in a focus group agreed to participate. The data were coded and grouped by domain and identified as barriers or facilitators. The findings included below are for the domains most relevant to the provision of VTE prophylaxis. Table 2 provides illustrative verbatim quotes for each domain that was represented in the focus groups.

Knowledge

All interviewees recognized that providing some form of prophylaxis to mitigate the risk of a VTE event is essential. Some nurses stated that seeing a patient ambulating meant they would consider not administering prescribed chemoprophylaxis, while others would try to negotiate with patients by asking the patient to allow one dose of chemoprophylaxis prescribed two to three times daily because it was better than receiving no doses.

Environmental Context and Resources

Multiple barriers to providing optimal care were associated with the environmental context and a lack of resources. There was a lack of accessible, comprehensive, patient-centered education materials on VTE chemoprophylaxis to supplement a nurse’s explanation about the importance of chemoprophylaxis. Furthermore, many nurses cited the perceived patient pain of chemoprophylaxis injections as the main deterrent to patient compliance, especially subcutaneous heparin injections, which occur up to three times in 24 hours and often cause more pain at the site of injection than low-molecular-weight heparin. Nurses felt that transitioning patients from receiving subcutaneous heparin injections to receiving low-molecular-weight heparin could be a main driver to reduce patient refusals.

Skills

Nurses felt inadequately equipped to handle patient refusals. Many said that patient refusal of treatments was never discussed in nursing school. As a result, when patients refused treatments, the nurses did not know how to handle the situation. They felt that they lacked the tools and techniques to persuade the patient to comply.

Beliefs about Capabilities

Nurses did not know their own patient refusal rate or benchmarks of an acceptable refusal rate in contrast to one that is too high. Without this feedback, they were unable to assess their own behavior or performance related to providing VTE chemoprophylaxis.

Nurses play a critical role in providing VTE chemoprophylaxis to patients throughout their hospitalization. This study provided a unique opportunity to perform an in-depth, qualitative analysis of the barriers nurses face in providing patients with VTE chemoprophylaxis as part of their daily work caring for patients. We discovered several nursing-related barriers to the provision of VTE chemoprophylaxis, including lack of knowledge, resources, skill, and misconceptions of their capability to provide VTE chemoprophylaxis. We used a bottom-up approach by incorporating the voices of unit nurses, clinical coordinators, and nurse managers to understand potential barriers. Our findings brought to light the challenge of delivering standardized care in an area of care that is generally agreed upon, yet not fully followed. Some nurses display greater proficiency than others at communicating with patients who do not understand their risk for VTE and need for chemoprophylaxis. Furthermore, there is a pronounced misconception around the delivery of VTE chemoprophylaxis. Nurses have the inaccurate belief that even if ordered, chemoprophylaxis is not required. This misconception was widespread among nurses taking care of both medical and surgical patients. These factors appear to be modifiable targets for quality improvement and highlight the need for a skills-based education during the new hire onboarding process, as well as ongoing reeducation to ensure nursing staff have the skills to appropriately provide best-practice care for VTE chemoprophylaxis. Nurses felt ownership of the results of the qualitative evaluation because they were included in every aspect from the beginning.²⁷ This sense of ownership will support future quality improvement efforts to develop a skills-based intervention to improve the provision of VTE chemoprophylaxis.^18,27

This study has certain limitations. First, it was a qualitative study assessing nursing-related barriers to providing VTE chemoprophylaxis at a single institution, and the results cannot be generalized broadly. However, the techniques and results are transferable to other hospital settings and other clinical care situations. Thus, we believe that other institutions can utilize our methods and that similar lessons can be learned and applied. Furthermore, the validity of our study is bolstered by concordance between the results of this study and those of other studies conducted on the topic of provision of VTE prophylaxis by nurses.^13-15,21 Other studies utilized observations and surveys to determine potential nurse-related barriers to the provision of VTE prophylaxis, such as lack of knowledge and the belief that the need for prophylaxis can be determined based on whether or not the patient is ambulating;^13,14 however, by utilizing focus group interviews, we allowed nurses to speak in their own voices about their experiences with VTE prophylaxis, and we were able to delve deeper and identify additional barriers that emerged from discussions with nurses, such as the lack of skill and misconceptions of capability.^28,29 Second, the study focused solely on nurses. Additional initiatives are underway to assess the roles of resident physicians, attending physicians, and patients in the provision of VTE prophylaxis.

Nursing-related barriers to the provision of VTE chemoprophylaxis include a lack of knowledge, resources, skills, and misconceptions of the consequences of missed elements of VTE prophylaxis. Future initiatives will focus on equipping nurses to have meaningful conversations with patients and engaging patients in their care through development of a multifaceted bundle of interventions. Furthermore, similar methods of qualitative inquiry will be used to identify the role of resident and attending physicians and patients in the provision of VTE chemoprophylaxis.

Acknowledgments

The authors thank Sonali Oberoi, Joanne Prinz, Nancy Tomaska, and Kate Paredes, as well as all the nurses who participated in focus group interviews for this study and the nurse managers and clinical coordinators who helped to schedule the focus group interviews.

Disclosures

The authors declare that they have no competing interests.

Funding

This study was funded by the Surgical Outcomes and Quality Improvement Center at Northwestern University.

Venous thromboembolism (VTE), comprising deep venous thrombosis and pulmonary embolism (PE),¹ is a serious medical condition that results in preventable morbidity and mortality.^1-5 VTE affects all age groups, all races/ethnicities, and both genders, but there are known factors that increase the risk of developing VTE (eg, advanced age, undergoing surgery, hospitalization, and immobility).^1-3,5-7 Prevention of VTE among hospitalized patients is of paramount importance to avoid preventable death, chronic illness/long-term complications,⁸ longer hospital stays, and increased hospital costs.⁹ Fortunately, there is clear evidence that provision of appropriate prophylaxis can decrease the risk of a VTE event occurring, and broadly accepted best-practice guidelines reflect this evidence.^3,5

Given the inadequacy of current VTE-related quality measures to identify actionable failures in the provision of VTE prophylaxis, our group created a VTE process-of-care measure to assess adherence to the components of VTE prophylaxis: (1) early ambulation, (2) mechanical prophylaxis (sequential compression devices [SCDs]), and (3) chemoprophylaxis administered at the correct dose and frequency for the duration of the patient’s hospital stay.^3,10,11 This quality measure was conceived, created, and iteratively revised to measure whether optimal care is provided to patients throughout their hospitalization and identify actionable areas in which failures of care occur, in order to decrease the risk of a VTE event. Data from our institution provided evidence that while ambulation and SCD component measure adherence is high, chemoprophylaxis adherence required significant improvement.¹⁰ When chemoprophylaxis process measure adherence data were analyzed further, a major failure mode was patient refusal of one or more doses. However, the drivers of patient refusal are not well defined in the literature, and previous studies have called for a greater focus on developing interventions to improve VTE chemoprophylaxis administration.¹²

Previous research has shown that nurses can influence patient compliance with VTE prophylaxis.^13-15 A mixed-methods study by Elder et al. found that nurses in units with high rates of failure to provide optimal chemoprophylaxis offered the medication as optional, leading researchers to conclude that nurses perceived chemoprophylaxis as discretionary.¹³ Another study by Lee et al., conducted a survey of bedside registered nurses and identified nurses’ lack of education on VTE prevention as a significant barrier to providing care.¹⁴ These studies show that multiple levels of influence impact how nurses provide VTE chemoprophylaxis, particularly when they encounter patients who refuse chemoprophylaxis.

To explore the nuance and interplay of multiple influences, we used the Theoretical Domains Framework (TDF), an integrative framework that applies theoretical approaches to interventions aimed at behavior change.^15-18 The framework contains 14 interrelated domains that characterize the behavior being studied, in this case, administration of VTE chemoprophylaxis. Consequently, we designed a nurse-focused, qualitative evaluation with the objective to identify nursing-related barriers to administration of VTE chemoprophylaxis.

Inpatient Unit Selection

The study team accessed data from the hospital’s Enterprise Data Warehouse to review patient refusal rates of VTE chemoprophylaxis for each inpatient unit in the hospital. Patient refusal was utilized as a proxy measure for the behavior of nurses attempting to administer VTE chemoprophylaxis. Of the 14 medical and surgical units in the hospital, two medical and two surgical units were selected to participate in the qualitative evaluation based on having the highest patient refusal rates. One unit (surgical) was also selected to serve as a benchmark because it had the lowest patient refusal rate. Table 1 includes the refusal rates for the five units. Given the low refusal rate for the best performing unit, we suspected that it would be possible to decrease the patient refusal rate for other units with similar patient populations and interprofessional teams at the institution.

Observations

We observed chemoprophylaxis administration on the five units to understand the process for ordering and administering chemoprophylaxis. An observation protocol was utilized to document the date, time, and location of the observation as well as descriptive notes including accounts of particular events.^19,20 Observations occurred in May 2016 and informed the creation of a process map outlining the procedure for ordering and administering VTE chemoprophylaxis. The process map was utilized to create the focus group interview guide and ensure the interview guide included pertinent questions for each step of the process (Appendix A).

Focus Group Interviews

We conducted focus group interviews with day and night shift nurses on the five units to assess nurses’ understanding of VTE chemoprophylaxis and nurses’ perceptions of barriers to administration of VTE chemoprophylaxis. The study team chose to conduct focus group interviews in an effort to maximize participation and to speak with multiple nurses within a shorter period of time. The focus group structure allowed the study team to speak with nurses during their shifts, as one could briefly step out, if required, for patient care and return to rejoin the discussion.

We developed a semistructured interview guide²¹ with questions focused on identifying nurses’ perceptions of guideline-recommended care for VTE chemoprophylaxis, where they learned these guidelines, how nurses discuss chemoprophylaxis with patients, how they handle the conversation with patients who refuse, and if there are times when chemoprophylaxis is not necessary. The interview guide was vetted by a multidisciplinary team consisting of clinical nursing coordinators and nurse managers from medical and surgical units, hospital quality leaders, surgeons and general internists, and qualitative research experts. The interview guide is included as Appendix B.

The unit clinical coordinators and nurse managers identified dates and times for the focus groups that would be minimally disruptive to the unit. For each of the four units with a high patient refusal rate, two focus groups were conducted during the lunch hour and one was conducted at the end of the night shift to ensure that both day and night shift nurses were included in the study. Two focus groups were conducted with the best-practice unit during the lunch hour. For each focus group, the clinical coordinator identified two to eight nurses who could step away from patient care to participate or who had completed their shifts. In total, approximately 67 nurses participated in the focus groups.

The focus groups (n = 14) lasted approximately 40 minutes during May and June 2016. Two members of the study team cofacilitated interviews, which were recorded and transcribed verbatim.

Coding and Data Analysis

To develop the code book, the study team, consisting of three qualitative researchers, independently read one focus group transcript and applied the TDF domains to the nurses’ perceptions of barriers to administration of VTE chemoprophylaxis.^21-24 In addition to coding by domain, the study team also coded nursing perceptions as barriers or facilitators. The study team reviewed the coded transcript and reconciled any differences in coding. This process was repeated for a second transcript, and then all remaining transcripts were assigned to two out of three study team members for coding, with the entire study team meeting to reconcile any differences. If necessary, the team member who did not code a transcript acted as the tie-breaker if there were discrepancies in codes that could not be reconciled.

Once coding was completed, we identified the TDF domains that were most relevant to the administration of VTE chemoprophylaxis.¹⁶ Member checking (testing the analysis, interpretations, and conclusions with members of those groups from whom the data were originally obtained) was performed with the four clinical nursing coordinators and four nurse managers from the participating units to establish face validity of the themes identified from the focus group interviews.²⁵

The study team used MaxQDA, V12 (Berlin, Germany) to support data coding and analysis.²⁶ The Northwestern University institutional review board office deemed this project research on nonhuman-subjects because it focused on the process of providing VTE chemoprophylaxis and not about the patients themselves. The purpose of the study was explained at the beginning of each focus group, and nurses gave verbal consent to have the focus group recorded.

We conducted 14 focus groups with day and night shift nurses from five units (two medical and three surgical) at a single institution. All nurses invited to participate in a focus group agreed to participate. The data were coded and grouped by domain and identified as barriers or facilitators. The findings included below are for the domains most relevant to the provision of VTE prophylaxis. Table 2 provides illustrative verbatim quotes for each domain that was represented in the focus groups.

Knowledge

All interviewees recognized that providing some form of prophylaxis to mitigate the risk of a VTE event is essential. Some nurses stated that seeing a patient ambulating meant they would consider not administering prescribed chemoprophylaxis, while others would try to negotiate with patients by asking the patient to allow one dose of chemoprophylaxis prescribed two to three times daily because it was better than receiving no doses.

Environmental Context and Resources

Multiple barriers to providing optimal care were associated with the environmental context and a lack of resources. There was a lack of accessible, comprehensive, patient-centered education materials on VTE chemoprophylaxis to supplement a nurse’s explanation about the importance of chemoprophylaxis. Furthermore, many nurses cited the perceived patient pain of chemoprophylaxis injections as the main deterrent to patient compliance, especially subcutaneous heparin injections, which occur up to three times in 24 hours and often cause more pain at the site of injection than low-molecular-weight heparin. Nurses felt that transitioning patients from receiving subcutaneous heparin injections to receiving low-molecular-weight heparin could be a main driver to reduce patient refusals.

Skills

Nurses felt inadequately equipped to handle patient refusals. Many said that patient refusal of treatments was never discussed in nursing school. As a result, when patients refused treatments, the nurses did not know how to handle the situation. They felt that they lacked the tools and techniques to persuade the patient to comply.

Beliefs about Capabilities

Nurses did not know their own patient refusal rate or benchmarks of an acceptable refusal rate in contrast to one that is too high. Without this feedback, they were unable to assess their own behavior or performance related to providing VTE chemoprophylaxis.

Nurses play a critical role in providing VTE chemoprophylaxis to patients throughout their hospitalization. This study provided a unique opportunity to perform an in-depth, qualitative analysis of the barriers nurses face in providing patients with VTE chemoprophylaxis as part of their daily work caring for patients. We discovered several nursing-related barriers to the provision of VTE chemoprophylaxis, including lack of knowledge, resources, skill, and misconceptions of their capability to provide VTE chemoprophylaxis. We used a bottom-up approach by incorporating the voices of unit nurses, clinical coordinators, and nurse managers to understand potential barriers. Our findings brought to light the challenge of delivering standardized care in an area of care that is generally agreed upon, yet not fully followed. Some nurses display greater proficiency than others at communicating with patients who do not understand their risk for VTE and need for chemoprophylaxis. Furthermore, there is a pronounced misconception around the delivery of VTE chemoprophylaxis. Nurses have the inaccurate belief that even if ordered, chemoprophylaxis is not required. This misconception was widespread among nurses taking care of both medical and surgical patients. These factors appear to be modifiable targets for quality improvement and highlight the need for a skills-based education during the new hire onboarding process, as well as ongoing reeducation to ensure nursing staff have the skills to appropriately provide best-practice care for VTE chemoprophylaxis. Nurses felt ownership of the results of the qualitative evaluation because they were included in every aspect from the beginning.²⁷ This sense of ownership will support future quality improvement efforts to develop a skills-based intervention to improve the provision of VTE chemoprophylaxis.^18,27

This study has certain limitations. First, it was a qualitative study assessing nursing-related barriers to providing VTE chemoprophylaxis at a single institution, and the results cannot be generalized broadly. However, the techniques and results are transferable to other hospital settings and other clinical care situations. Thus, we believe that other institutions can utilize our methods and that similar lessons can be learned and applied. Furthermore, the validity of our study is bolstered by concordance between the results of this study and those of other studies conducted on the topic of provision of VTE prophylaxis by nurses.^13-15,21 Other studies utilized observations and surveys to determine potential nurse-related barriers to the provision of VTE prophylaxis, such as lack of knowledge and the belief that the need for prophylaxis can be determined based on whether or not the patient is ambulating;^13,14 however, by utilizing focus group interviews, we allowed nurses to speak in their own voices about their experiences with VTE prophylaxis, and we were able to delve deeper and identify additional barriers that emerged from discussions with nurses, such as the lack of skill and misconceptions of capability.^28,29 Second, the study focused solely on nurses. Additional initiatives are underway to assess the roles of resident physicians, attending physicians, and patients in the provision of VTE prophylaxis.

Nursing-related barriers to the provision of VTE chemoprophylaxis include a lack of knowledge, resources, skills, and misconceptions of the consequences of missed elements of VTE prophylaxis. Future initiatives will focus on equipping nurses to have meaningful conversations with patients and engaging patients in their care through development of a multifaceted bundle of interventions. Furthermore, similar methods of qualitative inquiry will be used to identify the role of resident and attending physicians and patients in the provision of VTE chemoprophylaxis.

Acknowledgments

The authors thank Sonali Oberoi, Joanne Prinz, Nancy Tomaska, and Kate Paredes, as well as all the nurses who participated in focus group interviews for this study and the nurse managers and clinical coordinators who helped to schedule the focus group interviews.

Disclosures

The authors declare that they have no competing interests.

Funding

This study was funded by the Surgical Outcomes and Quality Improvement Center at Northwestern University.

Asthma results in approximately 125,000 hospitalizations for children annually in the United States.^1,2 The National Heart, Lung, and Blood Institute guidelines recommend that children with persistent asthma be treated with a daily controller medication, ie, an inhaled corticosteroid (ICS).³ Hospitalization for an asthma exacerbation provides an opportunity to optimize daily controller medications and improve disease self-management by providing access to medications and teaching appropriate use of complicated inhalation devices.

To reduce readmission⁴ by mitigating low rates of postdischarge filling of ICS prescriptions,^5,6 a strategy of “meds-in-hand” was implemented at discharge. “Meds-in-hand” mitigates medication access as a barrier to adherence by ensuring that patients are discharged from the hospital with all required medications in hand, removing any barriers to filling their initial prescriptions.⁷ The Asthma Improvement Collaborative at Cincinnati Children’s Hospital Medical Center (CCHMC) previously applied quality improvement methodology to implement “meds-in-hand” as a key intervention in a broad strategy that successfully reduced asthma-specific utilization for the 30-day period following an asthma-related hospitalization of publicly insured children from 12% to 7%.^8,9

At the onset of the work described in this manuscript, children hospitalized with an acute exacerbation of persistent asthma were most often treated with an ICS while inpatients in addition to a standard short course of oral systemic corticosteroids. Conceptually, inpatient administration of ICS provided the opportunity to teach effective device usage with each inpatient administration and to reinforce daily use of the ICS as part of the patient’s daily home medication regimen. However, a proportion of patients admitted for an asthma exacerbation were noted to receive more than one ICS inhaler during their admission, most commonly due to a change in dose or type of ICS. When this occurred, the initially dispensed inhaler was discarded despite weeks of potential doses remaining. While some hospitals preferentially dispense ICS devices marketed to institutions with fewer doses per device, our pharmacy primarily dispensed ICS devices identical to retail locations containing at least a one-month supply of medication. In addition to the wasted medication, this practice resulted in additional work by healthcare staff, unnecessary patient charges, and potentially contributed to confusion about the discharge medication regimen.

Our specific aim for this quality improvement study was to reduce the monthly percentage of admissions for an acute asthma exacerbation treated with >1 ICS from 7% to 4% over a six-month period.

Context

CCHMC is a quaternary care pediatric health system with more than 600 inpatient beds and 800-900 inpatient admissions per year for acute asthma exacerbation. The Hospital Medicine service cares for patients with asthma on five clinical teams across two different campuses. Care teams are supervised by an attending physician and may include residents, fellows, or nurse practitioners. Patients hospitalized for an acute asthma exacerbation may receive a consult from the Asthma Center consult team, staffed by faculty from either the Pediatric Pulmonology or Allergy/Immunology divisions. Respiratory therapists (RTs) administer inhaled medications and provide asthma education.

Planning the Intervention

Our improvement team included physicians from Hospital Medicine and Pulmonary Medicine, an Asthma Education Coordinator, a Clinical Pharmacist, a Pediatric Chief Resident, and a clinical research coordinator. Initial interventions targeted a single resident team at the main campus before spreading improvement activities to all resident teams at the main campus and then the satellite campus by February 2017.

Development of our process map (Figure 1) revealed that the decision for ordering inpatient ICS treatment frequently occurred at admission. Subsequently, the care team or consulting team might make a change in the ICS to fine-tune the outpatient medication regimen given that admission for asthma often results from suboptimal chronic symptom control. Baseline analysis of changes in ICS orders revealed that 81% of ICS changes were associated with a step-up in therapy, defined as an increase in the daily dose of the ICS or the addition of a long-acting beta-agonist. The other common ICS adjustment, accounting for 17%, was a change in corticosteroid without a step-up in therapy, (ie, beclomethasone to fluticasone) that typically occurred near the end of the hospitalization to accommodate outpatient insurance formularies, independent of patient factors related to illness severity.

Improvement Activities

We conceived key drivers or domains that would be necessary to address to effect change. Key drivers included a standardized process for delayed initiation of ICS and confirmation of outpatient insurance prescription drug coverage, prescriber education, and real-time failure notification.

PDSA Interventions

PDSA 1 & 2: Standardized Process for Initiation of ICS

Our initial tests of change targeted the timing of when an ICS was ordered during hospitalization for an asthma exacerbation. Providers were instructed to delay ordering an ICS until the patient’s albuterol treatments were spaced to every three hours and to include a standardized communication prompt within the albuterol order. The prompt instructed the RT to contact the provider once the patient’s albuterol treatments were spaced to every three hours and ask for an ICS order, if appropriate. This intervention was abandoned because it did not reliably occur.

The subsequent intervention delayed the start of ICS treatment by using a PRN indication advising that the ICS was to be administered once the patient’s albuterol treatments were spaced to every three hours. However, after an error resulted in the PRN indication being included on a discharge prescription for an ICS, the PRN indication was abandoned. Subsequent work to develop a standardized process for delayed initiation of ICS occurred as part of the workflow to address the confirmation of outpatient formulary coverage as described next.

PDSA 3: Prioritize the Use of the Institution’s Outpatient Pharmacy

Medication changes that occurred because of outpatient insurance formulary denials were a unique challenge; they required a medication change after the discharge treatment plan had been finalized, and a prescription was already submitted to the outpatient pharmacy. In addition, neither our inpatient electronic medical record nor our inpatient hospital pharmacy has access to decision support tools that incorporate outpatient prescription formulary coverage. Alternatively, outpatient pharmacies have a standard workflow that routinely confirms insurance coverage before dispensing medication. The institutional policy was modified to allow for the inpatient administration of patient-supplied medications, pursuant to an inpatient order. Patient-supplied medications include those brought from home or those supplied by the outpatient pharmacy.

Subsequently, we developed a standardized process to confirm outpatient prescription drug coverage by using our hospital-based outpatient pharmacy to dispense ICS for inpatient treatment and asthma education. This new workflow included placing an order for an ICS at admission as a patient-supplied medication with an administration comment to “please administer once available from the outpatient pharmacy” (Figure 1). Then, once the discharge medication plan is finalized, the prescription is submitted to the outpatient pharmacy. Following verification of insurance coverage, the outpatient pharmacy dispenses the ICS, allowing it to be used for patient education and inpatient administration. If the patient is ineligible to have their prescription filled by the outpatient pharmacy for reasons other than formulary coverage, the ICS is dispensed from the hospital inpatient pharmacy as per the previous standard workflow. Inpatient ICS inhalers are then relabeled for home use per the existing practice to support medications-in-hand.

Further workflow improvements occurred following the development of an algorithm to help the outpatient pharmacy contact the correct inpatient team, and augmentation of the medication delivery process included notification of the RT when the ICS was delivered to the inpatient unit.

PDSA 4: Prescriber Education

Prescribers received education regarding PDSA interventions before testing and throughout the improvement cycle. Education sessions included informal coaching by the Asthma Education Coordinator, e-mail reminders containing screenshots of the ordering process, and formal didactic sessions for ordering providers. The Asthma Education Coordinator also provided education to the nursing and respiratory therapy staff regarding the implemented process and workflow changes.

PDSA 5: Real-Time Failure Notification

To supplement education for the complicated process change, the improvement team utilized a decision support tool (Vigilanz Corp., Chicago, IL) linked to EMR data to provide notification of real-time process failures. When a patient with an admission diagnosis of asthma had a prescription for an ICS verified and dispensed by the inpatient pharmacy, an automated message with relevant patient information would be sent to a member of the improvement team. Following a brief chart review, directed feedback could be offered to the ordering provider, allowing the prescription to be redirected to the outpatient pharmacy.

Study of the Improvement

Patients of all ages, with the International Classification of Diseases, Ninth Revision, and Tenth Revision codes for asthma (493.xx or J45.xx) were included in data collection and analysis if they were treated by the Hospital Medicine service, as the first inpatient service or after transfer from the ICU, and prescribed an ICS with or without a long-acting beta-agonist. Data were collected retrospectively and aggregated monthly. The baseline period was from January 2015 through October 2016. The intervention period was from November 2016 through March 2018. The prolonged baseline and study periods were utilized to understand the seasonal nature of asthma exacerbations.

Measures

Our primary outcome measure was defined as the monthly number of patients admitted to Hospital Medicine for an acute asthma exacerbation administered more than one ICS divided by the total number of asthma patients administered at least one dose of an ICS (patient-supplied or dispensed from the inpatient pharmacy). A full list of ICS is included in the appendix Table.

A secondary process measure approximated our adherence to obtaining ICS from the outpatient pharmacy for inpatient use. All medications administered during hospitalization are documented in the medication administration report. However, only medications dispensed from the inpatient pharmacy are associated with a patient charge. Patient-supplied medications, including those dispensed from the hospital outpatient pharmacy, are not associated with an inpatient charge. Therefore, the secondary process measure was defined as the monthly number of asthma patients administered an ICS not associated with an inpatient charge divided by the total number of asthma patients administered an ICS.

A cost outcome measure was developed to track changes in the average cost of an ICS included on inpatient bills during hospitalization for an asthma exacerbation. This outcome measure was defined as the total monthly cost, using the average wholesale price, of the ICS included on the inpatient bill for an asthma exacerbation, divided by the total number of asthma patients administered at least one dose of an ICS (patient supplied or dispensed from the inpatient pharmacy). The costs of patient-supplied medications (including those dispensed from the outpatient pharmacy) are not included on inpatient hospital bills or this secondary outcome measure.

Our a priori intent was to reduce ICS medication waste while maintaining a highly reliable system that included inpatient administration and education with ICS devices and maintain our medications-in-hand practice. A balancing measure was developed to monitor the reliability of inpatient administration of ICS. It was defined as the monthly number of patients who received a discharge prescription for an ICS and were administered an ICS while an inpatient divided by the total number of asthma patients with a discharge prescription for an ICS.

Analysis

Measures were evaluated using statistical process control charts and special cause variation was determined by previously established rules. Our primary, secondary, and balancing measures were all evaluated using a p-chart with variable subgroup size. The cost outcome measure was evaluated using an X-bar S control chart.^11-13

Primary Outcome Measure

During the baseline period, 7.4% of patients admitted to Hospital Medicine for an acute asthma exacerbation were administered more than one ICS, ranging from 0%-20% of patients per month (Figure 2). Following the start of our interventions, we met criteria for special cause allowing adjustment of the centerline.¹³ The mean percentage of patients receiving more than one ICS decreased from 7.4% to 0.7%. Figure 2 includes the n-value displayed each month and represents all patients admitted to the Hospital Medicine service with an asthma exacerbation who were administered at least one ICS.

Secondary Process Measure

During the baseline period, there were only rare occurrences (less than 1%) of a patient-supplied ICS being administered during an asthma admission. Following the start of our intervention period, the frequency of inpatient administration of patient-supplied ICS showed a rapid increase and met rules for special cause with an increase in the mean percent from 0.7% to 50% (Figure 3). The n-value displayed each month represents all patients admitted to the Hospital Medicine service for an asthma exacerbation administered at least one ICS.

Cost Outcome Measure

The average cost of an ICS billed during hospitalization for an acute asthma exacerbation was $236.57 per ICS during the baseline period. After the intervention period, the average inpatient cost for ICS decreased by 62% to $90.25 per ICS (Figure 4).

Balancing Measure

Our team reduced the monthly percent of children hospitalized with an acute asthma exacerbation administered more than one ICS from 7.4% to 0.7% after implementation of a new workflow process for ordering ICS utilizing the hospital-based outpatient pharmacy. The new workflow delayed ordering and administration of the initial inpatient ICS treatment, allowing time to consider a step-up in therapy. The brief delay in initiating ICS is not expected to have clinical consequence given the concomitant treatment with systemic corticosteroids. In addition, the outpatient pharmacy was utilized to verify insurance coverage reliably prior to dispensing ICS, reducing medication waste, and discharge delays due to outpatient medication formulary conflicts.

Our hospital’s previous approach to inpatient asthma care resulted in a highly reliable process to ensure patients were discharged with medications-in-hand as part of a broader system that effectively decreased reutilization. However, the previous process inadvertently resulted in medication waste. This waste included nearly full inhalers being discarded, additional work by the healthcare team (ordering providers, pharmacists, and RTs), and unnecessary patient charges.

While the primary driver of our decision to use the outpatient pharmacy was to adjudicate insurance prescription coverage reliably to prevent waste, this change likely resulted in a financial benefit to patients. The average cost per asthma admission of an inpatient billed for ICS using the average wholesale price, decreased by 62% following our interventions. The decrease in cost was primarily driven by using patient-supplied medications, including prescriptions newly filled by the on-site outpatient pharmacy, whose costs were not captured in this measure. While our secondary measure may underestimate the total expense incurred by families for an ICS, families likely receive their medications at a lower cost from the outpatient pharmacy than if the ICS was provided by an inpatient pharmacy. The average wholesale price is not what families are charged or pay for medications, partly due to differences in overhead costs that result in inpatient pharmacies having significantly higher charges than outpatient pharmacies. In addition, the 6.7% absolute reduction of our primary measure resulted in direct savings by reducing inpatient medication waste. Our process results in 67 fewer wasted ICS devices ($15,960) per 1,000 admissions for asthma exacerbation, extrapolated using the average cost ($238.20, average wholesale price) of each ICS during the baseline period.

Our quality improvement study had several limitations. (1) The interventions occurred at a single center with an established culture that embraces quality improvement, which may limit the generalizability of the work. (2) Our process verified insurance coverage with a hospital-based outpatient pharmacy. Some ICS prescriptions continued to be dispensed from the inpatient pharmacy, limiting our ability to verify insurance coverage. Local factors, including regulatory restrictions and delivery requirements, may limit the generalizability of using an outpatient pharmacy in this manner. (3) We achieved our goal of decreasing medication waste, but our a priori goal was to maintain our commitment to our established practice of interactive patient education with an ICS device as well as medications-in-hand at time of discharge. Our balancing measure showed a decrease in the percent of patients with a discharge prescription for an ICS who also received an inpatient dose of that ICS. This implies a decreased fidelity in our previously established education protocols. We had postulated that this occurred when the patient-supplied medication arrived on the day of discharge, but not close to when the medication was scheduled on the medication administration report, preventing administration. However, this is not a direct measure of patients receiving medications-in-hand or interactive medication education. Both may have occurred without administration of the ICS. (4) Despite a hospital culture that embraces quality improvement, this project required a significant change in the workflow that required considerable education at the time of implementation to integrate the new process reliably. However, once the process was in place, we have been able to sustain our improvement with limited educational investment.

Implementation of a new process for ordering ICS that emphasized delaying treatment until all necessary information was available and using an outpatient pharmacy to confirm insurance formulary coverage reduced the waste associated with more than one ICS being prescribed during a single admission.

Acknowledgments

The authors thank Sally Pope, MPH and Dr. Michael Carlisle, MD for their contribution to the quality improvement project. Thank you to Drs. Karen McDowell, MD and Carolyn Kercsmar, MD for advisement of our quality improvement project.

The authors appreciate the following individuals for their invaluable contributions. Dr. Hoefgen conceptualized and designed the study, was a member of the primary improvement team, carried out initial analysis, drafted the initial manuscript, and reviewed and revised the manuscript. Drs. Jones and Torres Garcia, and Mr. Hare were members of the primary improvement team who contributed to the design of the quality improvement study and interventions, ongoing data interpretation, and critically reviewed the manuscript. Dr. Courter contributed to the conceptualization and designed the study, was a member of the primary improvement team, designed data collection instruments, and critically reviewed and revised the manuscript. Dr. Simmons conceptualized and designed the study, critically reviewed the manuscript for important intellectual content, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclaimer

The information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by the BHPR, HRSA, DHHS, or the U.S. Government.

Asthma results in approximately 125,000 hospitalizations for children annually in the United States.^1,2 The National Heart, Lung, and Blood Institute guidelines recommend that children with persistent asthma be treated with a daily controller medication, ie, an inhaled corticosteroid (ICS).³ Hospitalization for an asthma exacerbation provides an opportunity to optimize daily controller medications and improve disease self-management by providing access to medications and teaching appropriate use of complicated inhalation devices.

To reduce readmission⁴ by mitigating low rates of postdischarge filling of ICS prescriptions,^5,6 a strategy of “meds-in-hand” was implemented at discharge. “Meds-in-hand” mitigates medication access as a barrier to adherence by ensuring that patients are discharged from the hospital with all required medications in hand, removing any barriers to filling their initial prescriptions.⁷ The Asthma Improvement Collaborative at Cincinnati Children’s Hospital Medical Center (CCHMC) previously applied quality improvement methodology to implement “meds-in-hand” as a key intervention in a broad strategy that successfully reduced asthma-specific utilization for the 30-day period following an asthma-related hospitalization of publicly insured children from 12% to 7%.^8,9

At the onset of the work described in this manuscript, children hospitalized with an acute exacerbation of persistent asthma were most often treated with an ICS while inpatients in addition to a standard short course of oral systemic corticosteroids. Conceptually, inpatient administration of ICS provided the opportunity to teach effective device usage with each inpatient administration and to reinforce daily use of the ICS as part of the patient’s daily home medication regimen. However, a proportion of patients admitted for an asthma exacerbation were noted to receive more than one ICS inhaler during their admission, most commonly due to a change in dose or type of ICS. When this occurred, the initially dispensed inhaler was discarded despite weeks of potential doses remaining. While some hospitals preferentially dispense ICS devices marketed to institutions with fewer doses per device, our pharmacy primarily dispensed ICS devices identical to retail locations containing at least a one-month supply of medication. In addition to the wasted medication, this practice resulted in additional work by healthcare staff, unnecessary patient charges, and potentially contributed to confusion about the discharge medication regimen.

Our specific aim for this quality improvement study was to reduce the monthly percentage of admissions for an acute asthma exacerbation treated with >1 ICS from 7% to 4% over a six-month period.

Context

CCHMC is a quaternary care pediatric health system with more than 600 inpatient beds and 800-900 inpatient admissions per year for acute asthma exacerbation. The Hospital Medicine service cares for patients with asthma on five clinical teams across two different campuses. Care teams are supervised by an attending physician and may include residents, fellows, or nurse practitioners. Patients hospitalized for an acute asthma exacerbation may receive a consult from the Asthma Center consult team, staffed by faculty from either the Pediatric Pulmonology or Allergy/Immunology divisions. Respiratory therapists (RTs) administer inhaled medications and provide asthma education.

Planning the Intervention

Our improvement team included physicians from Hospital Medicine and Pulmonary Medicine, an Asthma Education Coordinator, a Clinical Pharmacist, a Pediatric Chief Resident, and a clinical research coordinator. Initial interventions targeted a single resident team at the main campus before spreading improvement activities to all resident teams at the main campus and then the satellite campus by February 2017.

Development of our process map (Figure 1) revealed that the decision for ordering inpatient ICS treatment frequently occurred at admission. Subsequently, the care team or consulting team might make a change in the ICS to fine-tune the outpatient medication regimen given that admission for asthma often results from suboptimal chronic symptom control. Baseline analysis of changes in ICS orders revealed that 81% of ICS changes were associated with a step-up in therapy, defined as an increase in the daily dose of the ICS or the addition of a long-acting beta-agonist. The other common ICS adjustment, accounting for 17%, was a change in corticosteroid without a step-up in therapy, (ie, beclomethasone to fluticasone) that typically occurred near the end of the hospitalization to accommodate outpatient insurance formularies, independent of patient factors related to illness severity.

Improvement Activities

We conceived key drivers or domains that would be necessary to address to effect change. Key drivers included a standardized process for delayed initiation of ICS and confirmation of outpatient insurance prescription drug coverage, prescriber education, and real-time failure notification.

PDSA Interventions

PDSA 1 & 2: Standardized Process for Initiation of ICS

Our initial tests of change targeted the timing of when an ICS was ordered during hospitalization for an asthma exacerbation. Providers were instructed to delay ordering an ICS until the patient’s albuterol treatments were spaced to every three hours and to include a standardized communication prompt within the albuterol order. The prompt instructed the RT to contact the provider once the patient’s albuterol treatments were spaced to every three hours and ask for an ICS order, if appropriate. This intervention was abandoned because it did not reliably occur.

The subsequent intervention delayed the start of ICS treatment by using a PRN indication advising that the ICS was to be administered once the patient’s albuterol treatments were spaced to every three hours. However, after an error resulted in the PRN indication being included on a discharge prescription for an ICS, the PRN indication was abandoned. Subsequent work to develop a standardized process for delayed initiation of ICS occurred as part of the workflow to address the confirmation of outpatient formulary coverage as described next.

PDSA 3: Prioritize the Use of the Institution’s Outpatient Pharmacy

Medication changes that occurred because of outpatient insurance formulary denials were a unique challenge; they required a medication change after the discharge treatment plan had been finalized, and a prescription was already submitted to the outpatient pharmacy. In addition, neither our inpatient electronic medical record nor our inpatient hospital pharmacy has access to decision support tools that incorporate outpatient prescription formulary coverage. Alternatively, outpatient pharmacies have a standard workflow that routinely confirms insurance coverage before dispensing medication. The institutional policy was modified to allow for the inpatient administration of patient-supplied medications, pursuant to an inpatient order. Patient-supplied medications include those brought from home or those supplied by the outpatient pharmacy.

Subsequently, we developed a standardized process to confirm outpatient prescription drug coverage by using our hospital-based outpatient pharmacy to dispense ICS for inpatient treatment and asthma education. This new workflow included placing an order for an ICS at admission as a patient-supplied medication with an administration comment to “please administer once available from the outpatient pharmacy” (Figure 1). Then, once the discharge medication plan is finalized, the prescription is submitted to the outpatient pharmacy. Following verification of insurance coverage, the outpatient pharmacy dispenses the ICS, allowing it to be used for patient education and inpatient administration. If the patient is ineligible to have their prescription filled by the outpatient pharmacy for reasons other than formulary coverage, the ICS is dispensed from the hospital inpatient pharmacy as per the previous standard workflow. Inpatient ICS inhalers are then relabeled for home use per the existing practice to support medications-in-hand.

Further workflow improvements occurred following the development of an algorithm to help the outpatient pharmacy contact the correct inpatient team, and augmentation of the medication delivery process included notification of the RT when the ICS was delivered to the inpatient unit.

PDSA 4: Prescriber Education

Prescribers received education regarding PDSA interventions before testing and throughout the improvement cycle. Education sessions included informal coaching by the Asthma Education Coordinator, e-mail reminders containing screenshots of the ordering process, and formal didactic sessions for ordering providers. The Asthma Education Coordinator also provided education to the nursing and respiratory therapy staff regarding the implemented process and workflow changes.

PDSA 5: Real-Time Failure Notification

To supplement education for the complicated process change, the improvement team utilized a decision support tool (Vigilanz Corp., Chicago, IL) linked to EMR data to provide notification of real-time process failures. When a patient with an admission diagnosis of asthma had a prescription for an ICS verified and dispensed by the inpatient pharmacy, an automated message with relevant patient information would be sent to a member of the improvement team. Following a brief chart review, directed feedback could be offered to the ordering provider, allowing the prescription to be redirected to the outpatient pharmacy.

Study of the Improvement

Patients of all ages, with the International Classification of Diseases, Ninth Revision, and Tenth Revision codes for asthma (493.xx or J45.xx) were included in data collection and analysis if they were treated by the Hospital Medicine service, as the first inpatient service or after transfer from the ICU, and prescribed an ICS with or without a long-acting beta-agonist. Data were collected retrospectively and aggregated monthly. The baseline period was from January 2015 through October 2016. The intervention period was from November 2016 through March 2018. The prolonged baseline and study periods were utilized to understand the seasonal nature of asthma exacerbations.

Measures

Our primary outcome measure was defined as the monthly number of patients admitted to Hospital Medicine for an acute asthma exacerbation administered more than one ICS divided by the total number of asthma patients administered at least one dose of an ICS (patient-supplied or dispensed from the inpatient pharmacy). A full list of ICS is included in the appendix Table.

A secondary process measure approximated our adherence to obtaining ICS from the outpatient pharmacy for inpatient use. All medications administered during hospitalization are documented in the medication administration report. However, only medications dispensed from the inpatient pharmacy are associated with a patient charge. Patient-supplied medications, including those dispensed from the hospital outpatient pharmacy, are not associated with an inpatient charge. Therefore, the secondary process measure was defined as the monthly number of asthma patients administered an ICS not associated with an inpatient charge divided by the total number of asthma patients administered an ICS.

A cost outcome measure was developed to track changes in the average cost of an ICS included on inpatient bills during hospitalization for an asthma exacerbation. This outcome measure was defined as the total monthly cost, using the average wholesale price, of the ICS included on the inpatient bill for an asthma exacerbation, divided by the total number of asthma patients administered at least one dose of an ICS (patient supplied or dispensed from the inpatient pharmacy). The costs of patient-supplied medications (including those dispensed from the outpatient pharmacy) are not included on inpatient hospital bills or this secondary outcome measure.

Our a priori intent was to reduce ICS medication waste while maintaining a highly reliable system that included inpatient administration and education with ICS devices and maintain our medications-in-hand practice. A balancing measure was developed to monitor the reliability of inpatient administration of ICS. It was defined as the monthly number of patients who received a discharge prescription for an ICS and were administered an ICS while an inpatient divided by the total number of asthma patients with a discharge prescription for an ICS.

Analysis

Measures were evaluated using statistical process control charts and special cause variation was determined by previously established rules. Our primary, secondary, and balancing measures were all evaluated using a p-chart with variable subgroup size. The cost outcome measure was evaluated using an X-bar S control chart.^11-13

Primary Outcome Measure

During the baseline period, 7.4% of patients admitted to Hospital Medicine for an acute asthma exacerbation were administered more than one ICS, ranging from 0%-20% of patients per month (Figure 2). Following the start of our interventions, we met criteria for special cause allowing adjustment of the centerline.¹³ The mean percentage of patients receiving more than one ICS decreased from 7.4% to 0.7%. Figure 2 includes the n-value displayed each month and represents all patients admitted to the Hospital Medicine service with an asthma exacerbation who were administered at least one ICS.

Secondary Process Measure

During the baseline period, there were only rare occurrences (less than 1%) of a patient-supplied ICS being administered during an asthma admission. Following the start of our intervention period, the frequency of inpatient administration of patient-supplied ICS showed a rapid increase and met rules for special cause with an increase in the mean percent from 0.7% to 50% (Figure 3). The n-value displayed each month represents all patients admitted to the Hospital Medicine service for an asthma exacerbation administered at least one ICS.

Cost Outcome Measure

The average cost of an ICS billed during hospitalization for an acute asthma exacerbation was $236.57 per ICS during the baseline period. After the intervention period, the average inpatient cost for ICS decreased by 62% to $90.25 per ICS (Figure 4).

Balancing Measure

Our team reduced the monthly percent of children hospitalized with an acute asthma exacerbation administered more than one ICS from 7.4% to 0.7% after implementation of a new workflow process for ordering ICS utilizing the hospital-based outpatient pharmacy. The new workflow delayed ordering and administration of the initial inpatient ICS treatment, allowing time to consider a step-up in therapy. The brief delay in initiating ICS is not expected to have clinical consequence given the concomitant treatment with systemic corticosteroids. In addition, the outpatient pharmacy was utilized to verify insurance coverage reliably prior to dispensing ICS, reducing medication waste, and discharge delays due to outpatient medication formulary conflicts.

Our hospital’s previous approach to inpatient asthma care resulted in a highly reliable process to ensure patients were discharged with medications-in-hand as part of a broader system that effectively decreased reutilization. However, the previous process inadvertently resulted in medication waste. This waste included nearly full inhalers being discarded, additional work by the healthcare team (ordering providers, pharmacists, and RTs), and unnecessary patient charges.

While the primary driver of our decision to use the outpatient pharmacy was to adjudicate insurance prescription coverage reliably to prevent waste, this change likely resulted in a financial benefit to patients. The average cost per asthma admission of an inpatient billed for ICS using the average wholesale price, decreased by 62% following our interventions. The decrease in cost was primarily driven by using patient-supplied medications, including prescriptions newly filled by the on-site outpatient pharmacy, whose costs were not captured in this measure. While our secondary measure may underestimate the total expense incurred by families for an ICS, families likely receive their medications at a lower cost from the outpatient pharmacy than if the ICS was provided by an inpatient pharmacy. The average wholesale price is not what families are charged or pay for medications, partly due to differences in overhead costs that result in inpatient pharmacies having significantly higher charges than outpatient pharmacies. In addition, the 6.7% absolute reduction of our primary measure resulted in direct savings by reducing inpatient medication waste. Our process results in 67 fewer wasted ICS devices ($15,960) per 1,000 admissions for asthma exacerbation, extrapolated using the average cost ($238.20, average wholesale price) of each ICS during the baseline period.

Our quality improvement study had several limitations. (1) The interventions occurred at a single center with an established culture that embraces quality improvement, which may limit the generalizability of the work. (2) Our process verified insurance coverage with a hospital-based outpatient pharmacy. Some ICS prescriptions continued to be dispensed from the inpatient pharmacy, limiting our ability to verify insurance coverage. Local factors, including regulatory restrictions and delivery requirements, may limit the generalizability of using an outpatient pharmacy in this manner. (3) We achieved our goal of decreasing medication waste, but our a priori goal was to maintain our commitment to our established practice of interactive patient education with an ICS device as well as medications-in-hand at time of discharge. Our balancing measure showed a decrease in the percent of patients with a discharge prescription for an ICS who also received an inpatient dose of that ICS. This implies a decreased fidelity in our previously established education protocols. We had postulated that this occurred when the patient-supplied medication arrived on the day of discharge, but not close to when the medication was scheduled on the medication administration report, preventing administration. However, this is not a direct measure of patients receiving medications-in-hand or interactive medication education. Both may have occurred without administration of the ICS. (4) Despite a hospital culture that embraces quality improvement, this project required a significant change in the workflow that required considerable education at the time of implementation to integrate the new process reliably. However, once the process was in place, we have been able to sustain our improvement with limited educational investment.

Implementation of a new process for ordering ICS that emphasized delaying treatment until all necessary information was available and using an outpatient pharmacy to confirm insurance formulary coverage reduced the waste associated with more than one ICS being prescribed during a single admission.

Acknowledgments

The authors thank Sally Pope, MPH and Dr. Michael Carlisle, MD for their contribution to the quality improvement project. Thank you to Drs. Karen McDowell, MD and Carolyn Kercsmar, MD for advisement of our quality improvement project.

The authors appreciate the following individuals for their invaluable contributions. Dr. Hoefgen conceptualized and designed the study, was a member of the primary improvement team, carried out initial analysis, drafted the initial manuscript, and reviewed and revised the manuscript. Drs. Jones and Torres Garcia, and Mr. Hare were members of the primary improvement team who contributed to the design of the quality improvement study and interventions, ongoing data interpretation, and critically reviewed the manuscript. Dr. Courter contributed to the conceptualization and designed the study, was a member of the primary improvement team, designed data collection instruments, and critically reviewed and revised the manuscript. Dr. Simmons conceptualized and designed the study, critically reviewed the manuscript for important intellectual content, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclaimer

The information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by the BHPR, HRSA, DHHS, or the U.S. Government.

Asthma results in approximately 125,000 hospitalizations for children annually in the United States.^1,2 The National Heart, Lung, and Blood Institute guidelines recommend that children with persistent asthma be treated with a daily controller medication, ie, an inhaled corticosteroid (ICS).³ Hospitalization for an asthma exacerbation provides an opportunity to optimize daily controller medications and improve disease self-management by providing access to medications and teaching appropriate use of complicated inhalation devices.

To reduce readmission⁴ by mitigating low rates of postdischarge filling of ICS prescriptions,^5,6 a strategy of “meds-in-hand” was implemented at discharge. “Meds-in-hand” mitigates medication access as a barrier to adherence by ensuring that patients are discharged from the hospital with all required medications in hand, removing any barriers to filling their initial prescriptions.⁷ The Asthma Improvement Collaborative at Cincinnati Children’s Hospital Medical Center (CCHMC) previously applied quality improvement methodology to implement “meds-in-hand” as a key intervention in a broad strategy that successfully reduced asthma-specific utilization for the 30-day period following an asthma-related hospitalization of publicly insured children from 12% to 7%.^8,9

At the onset of the work described in this manuscript, children hospitalized with an acute exacerbation of persistent asthma were most often treated with an ICS while inpatients in addition to a standard short course of oral systemic corticosteroids. Conceptually, inpatient administration of ICS provided the opportunity to teach effective device usage with each inpatient administration and to reinforce daily use of the ICS as part of the patient’s daily home medication regimen. However, a proportion of patients admitted for an asthma exacerbation were noted to receive more than one ICS inhaler during their admission, most commonly due to a change in dose or type of ICS. When this occurred, the initially dispensed inhaler was discarded despite weeks of potential doses remaining. While some hospitals preferentially dispense ICS devices marketed to institutions with fewer doses per device, our pharmacy primarily dispensed ICS devices identical to retail locations containing at least a one-month supply of medication. In addition to the wasted medication, this practice resulted in additional work by healthcare staff, unnecessary patient charges, and potentially contributed to confusion about the discharge medication regimen.

Our specific aim for this quality improvement study was to reduce the monthly percentage of admissions for an acute asthma exacerbation treated with >1 ICS from 7% to 4% over a six-month period.

Context

CCHMC is a quaternary care pediatric health system with more than 600 inpatient beds and 800-900 inpatient admissions per year for acute asthma exacerbation. The Hospital Medicine service cares for patients with asthma on five clinical teams across two different campuses. Care teams are supervised by an attending physician and may include residents, fellows, or nurse practitioners. Patients hospitalized for an acute asthma exacerbation may receive a consult from the Asthma Center consult team, staffed by faculty from either the Pediatric Pulmonology or Allergy/Immunology divisions. Respiratory therapists (RTs) administer inhaled medications and provide asthma education.

Planning the Intervention

Our improvement team included physicians from Hospital Medicine and Pulmonary Medicine, an Asthma Education Coordinator, a Clinical Pharmacist, a Pediatric Chief Resident, and a clinical research coordinator. Initial interventions targeted a single resident team at the main campus before spreading improvement activities to all resident teams at the main campus and then the satellite campus by February 2017.

Development of our process map (Figure 1) revealed that the decision for ordering inpatient ICS treatment frequently occurred at admission. Subsequently, the care team or consulting team might make a change in the ICS to fine-tune the outpatient medication regimen given that admission for asthma often results from suboptimal chronic symptom control. Baseline analysis of changes in ICS orders revealed that 81% of ICS changes were associated with a step-up in therapy, defined as an increase in the daily dose of the ICS or the addition of a long-acting beta-agonist. The other common ICS adjustment, accounting for 17%, was a change in corticosteroid without a step-up in therapy, (ie, beclomethasone to fluticasone) that typically occurred near the end of the hospitalization to accommodate outpatient insurance formularies, independent of patient factors related to illness severity.

Improvement Activities

We conceived key drivers or domains that would be necessary to address to effect change. Key drivers included a standardized process for delayed initiation of ICS and confirmation of outpatient insurance prescription drug coverage, prescriber education, and real-time failure notification.

PDSA Interventions

PDSA 1 & 2: Standardized Process for Initiation of ICS

Our initial tests of change targeted the timing of when an ICS was ordered during hospitalization for an asthma exacerbation. Providers were instructed to delay ordering an ICS until the patient’s albuterol treatments were spaced to every three hours and to include a standardized communication prompt within the albuterol order. The prompt instructed the RT to contact the provider once the patient’s albuterol treatments were spaced to every three hours and ask for an ICS order, if appropriate. This intervention was abandoned because it did not reliably occur.

The subsequent intervention delayed the start of ICS treatment by using a PRN indication advising that the ICS was to be administered once the patient’s albuterol treatments were spaced to every three hours. However, after an error resulted in the PRN indication being included on a discharge prescription for an ICS, the PRN indication was abandoned. Subsequent work to develop a standardized process for delayed initiation of ICS occurred as part of the workflow to address the confirmation of outpatient formulary coverage as described next.

PDSA 3: Prioritize the Use of the Institution’s Outpatient Pharmacy

Medication changes that occurred because of outpatient insurance formulary denials were a unique challenge; they required a medication change after the discharge treatment plan had been finalized, and a prescription was already submitted to the outpatient pharmacy. In addition, neither our inpatient electronic medical record nor our inpatient hospital pharmacy has access to decision support tools that incorporate outpatient prescription formulary coverage. Alternatively, outpatient pharmacies have a standard workflow that routinely confirms insurance coverage before dispensing medication. The institutional policy was modified to allow for the inpatient administration of patient-supplied medications, pursuant to an inpatient order. Patient-supplied medications include those brought from home or those supplied by the outpatient pharmacy.

Subsequently, we developed a standardized process to confirm outpatient prescription drug coverage by using our hospital-based outpatient pharmacy to dispense ICS for inpatient treatment and asthma education. This new workflow included placing an order for an ICS at admission as a patient-supplied medication with an administration comment to “please administer once available from the outpatient pharmacy” (Figure 1). Then, once the discharge medication plan is finalized, the prescription is submitted to the outpatient pharmacy. Following verification of insurance coverage, the outpatient pharmacy dispenses the ICS, allowing it to be used for patient education and inpatient administration. If the patient is ineligible to have their prescription filled by the outpatient pharmacy for reasons other than formulary coverage, the ICS is dispensed from the hospital inpatient pharmacy as per the previous standard workflow. Inpatient ICS inhalers are then relabeled for home use per the existing practice to support medications-in-hand.

Further workflow improvements occurred following the development of an algorithm to help the outpatient pharmacy contact the correct inpatient team, and augmentation of the medication delivery process included notification of the RT when the ICS was delivered to the inpatient unit.

PDSA 4: Prescriber Education

Prescribers received education regarding PDSA interventions before testing and throughout the improvement cycle. Education sessions included informal coaching by the Asthma Education Coordinator, e-mail reminders containing screenshots of the ordering process, and formal didactic sessions for ordering providers. The Asthma Education Coordinator also provided education to the nursing and respiratory therapy staff regarding the implemented process and workflow changes.

PDSA 5: Real-Time Failure Notification

To supplement education for the complicated process change, the improvement team utilized a decision support tool (Vigilanz Corp., Chicago, IL) linked to EMR data to provide notification of real-time process failures. When a patient with an admission diagnosis of asthma had a prescription for an ICS verified and dispensed by the inpatient pharmacy, an automated message with relevant patient information would be sent to a member of the improvement team. Following a brief chart review, directed feedback could be offered to the ordering provider, allowing the prescription to be redirected to the outpatient pharmacy.

Study of the Improvement

Patients of all ages, with the International Classification of Diseases, Ninth Revision, and Tenth Revision codes for asthma (493.xx or J45.xx) were included in data collection and analysis if they were treated by the Hospital Medicine service, as the first inpatient service or after transfer from the ICU, and prescribed an ICS with or without a long-acting beta-agonist. Data were collected retrospectively and aggregated monthly. The baseline period was from January 2015 through October 2016. The intervention period was from November 2016 through March 2018. The prolonged baseline and study periods were utilized to understand the seasonal nature of asthma exacerbations.

Measures

Our primary outcome measure was defined as the monthly number of patients admitted to Hospital Medicine for an acute asthma exacerbation administered more than one ICS divided by the total number of asthma patients administered at least one dose of an ICS (patient-supplied or dispensed from the inpatient pharmacy). A full list of ICS is included in the appendix Table.

A secondary process measure approximated our adherence to obtaining ICS from the outpatient pharmacy for inpatient use. All medications administered during hospitalization are documented in the medication administration report. However, only medications dispensed from the inpatient pharmacy are associated with a patient charge. Patient-supplied medications, including those dispensed from the hospital outpatient pharmacy, are not associated with an inpatient charge. Therefore, the secondary process measure was defined as the monthly number of asthma patients administered an ICS not associated with an inpatient charge divided by the total number of asthma patients administered an ICS.

A cost outcome measure was developed to track changes in the average cost of an ICS included on inpatient bills during hospitalization for an asthma exacerbation. This outcome measure was defined as the total monthly cost, using the average wholesale price, of the ICS included on the inpatient bill for an asthma exacerbation, divided by the total number of asthma patients administered at least one dose of an ICS (patient supplied or dispensed from the inpatient pharmacy). The costs of patient-supplied medications (including those dispensed from the outpatient pharmacy) are not included on inpatient hospital bills or this secondary outcome measure.

Our a priori intent was to reduce ICS medication waste while maintaining a highly reliable system that included inpatient administration and education with ICS devices and maintain our medications-in-hand practice. A balancing measure was developed to monitor the reliability of inpatient administration of ICS. It was defined as the monthly number of patients who received a discharge prescription for an ICS and were administered an ICS while an inpatient divided by the total number of asthma patients with a discharge prescription for an ICS.

Analysis

Measures were evaluated using statistical process control charts and special cause variation was determined by previously established rules. Our primary, secondary, and balancing measures were all evaluated using a p-chart with variable subgroup size. The cost outcome measure was evaluated using an X-bar S control chart.^11-13

Primary Outcome Measure

During the baseline period, 7.4% of patients admitted to Hospital Medicine for an acute asthma exacerbation were administered more than one ICS, ranging from 0%-20% of patients per month (Figure 2). Following the start of our interventions, we met criteria for special cause allowing adjustment of the centerline.¹³ The mean percentage of patients receiving more than one ICS decreased from 7.4% to 0.7%. Figure 2 includes the n-value displayed each month and represents all patients admitted to the Hospital Medicine service with an asthma exacerbation who were administered at least one ICS.

Secondary Process Measure

During the baseline period, there were only rare occurrences (less than 1%) of a patient-supplied ICS being administered during an asthma admission. Following the start of our intervention period, the frequency of inpatient administration of patient-supplied ICS showed a rapid increase and met rules for special cause with an increase in the mean percent from 0.7% to 50% (Figure 3). The n-value displayed each month represents all patients admitted to the Hospital Medicine service for an asthma exacerbation administered at least one ICS.

Cost Outcome Measure

The average cost of an ICS billed during hospitalization for an acute asthma exacerbation was $236.57 per ICS during the baseline period. After the intervention period, the average inpatient cost for ICS decreased by 62% to $90.25 per ICS (Figure 4).

Balancing Measure

Our team reduced the monthly percent of children hospitalized with an acute asthma exacerbation administered more than one ICS from 7.4% to 0.7% after implementation of a new workflow process for ordering ICS utilizing the hospital-based outpatient pharmacy. The new workflow delayed ordering and administration of the initial inpatient ICS treatment, allowing time to consider a step-up in therapy. The brief delay in initiating ICS is not expected to have clinical consequence given the concomitant treatment with systemic corticosteroids. In addition, the outpatient pharmacy was utilized to verify insurance coverage reliably prior to dispensing ICS, reducing medication waste, and discharge delays due to outpatient medication formulary conflicts.

Our hospital’s previous approach to inpatient asthma care resulted in a highly reliable process to ensure patients were discharged with medications-in-hand as part of a broader system that effectively decreased reutilization. However, the previous process inadvertently resulted in medication waste. This waste included nearly full inhalers being discarded, additional work by the healthcare team (ordering providers, pharmacists, and RTs), and unnecessary patient charges.

While the primary driver of our decision to use the outpatient pharmacy was to adjudicate insurance prescription coverage reliably to prevent waste, this change likely resulted in a financial benefit to patients. The average cost per asthma admission of an inpatient billed for ICS using the average wholesale price, decreased by 62% following our interventions. The decrease in cost was primarily driven by using patient-supplied medications, including prescriptions newly filled by the on-site outpatient pharmacy, whose costs were not captured in this measure. While our secondary measure may underestimate the total expense incurred by families for an ICS, families likely receive their medications at a lower cost from the outpatient pharmacy than if the ICS was provided by an inpatient pharmacy. The average wholesale price is not what families are charged or pay for medications, partly due to differences in overhead costs that result in inpatient pharmacies having significantly higher charges than outpatient pharmacies. In addition, the 6.7% absolute reduction of our primary measure resulted in direct savings by reducing inpatient medication waste. Our process results in 67 fewer wasted ICS devices ($15,960) per 1,000 admissions for asthma exacerbation, extrapolated using the average cost ($238.20, average wholesale price) of each ICS during the baseline period.

Our quality improvement study had several limitations. (1) The interventions occurred at a single center with an established culture that embraces quality improvement, which may limit the generalizability of the work. (2) Our process verified insurance coverage with a hospital-based outpatient pharmacy. Some ICS prescriptions continued to be dispensed from the inpatient pharmacy, limiting our ability to verify insurance coverage. Local factors, including regulatory restrictions and delivery requirements, may limit the generalizability of using an outpatient pharmacy in this manner. (3) We achieved our goal of decreasing medication waste, but our a priori goal was to maintain our commitment to our established practice of interactive patient education with an ICS device as well as medications-in-hand at time of discharge. Our balancing measure showed a decrease in the percent of patients with a discharge prescription for an ICS who also received an inpatient dose of that ICS. This implies a decreased fidelity in our previously established education protocols. We had postulated that this occurred when the patient-supplied medication arrived on the day of discharge, but not close to when the medication was scheduled on the medication administration report, preventing administration. However, this is not a direct measure of patients receiving medications-in-hand or interactive medication education. Both may have occurred without administration of the ICS. (4) Despite a hospital culture that embraces quality improvement, this project required a significant change in the workflow that required considerable education at the time of implementation to integrate the new process reliably. However, once the process was in place, we have been able to sustain our improvement with limited educational investment.

Implementation of a new process for ordering ICS that emphasized delaying treatment until all necessary information was available and using an outpatient pharmacy to confirm insurance formulary coverage reduced the waste associated with more than one ICS being prescribed during a single admission.

Acknowledgments

The authors thank Sally Pope, MPH and Dr. Michael Carlisle, MD for their contribution to the quality improvement project. Thank you to Drs. Karen McDowell, MD and Carolyn Kercsmar, MD for advisement of our quality improvement project.

The authors appreciate the following individuals for their invaluable contributions. Dr. Hoefgen conceptualized and designed the study, was a member of the primary improvement team, carried out initial analysis, drafted the initial manuscript, and reviewed and revised the manuscript. Drs. Jones and Torres Garcia, and Mr. Hare were members of the primary improvement team who contributed to the design of the quality improvement study and interventions, ongoing data interpretation, and critically reviewed the manuscript. Dr. Courter contributed to the conceptualization and designed the study, was a member of the primary improvement team, designed data collection instruments, and critically reviewed and revised the manuscript. Dr. Simmons conceptualized and designed the study, critically reviewed the manuscript for important intellectual content, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Disclaimer

The information or content and conclusions are those of the author and should not be construed as the official position or policy of, nor should any endorsements be inferred by the BHPR, HRSA, DHHS, or the U.S. Government.

Hip fractures in the elderly are associated with significant morbidity and mortality.¹ These are typically fragility fractures since they are caused by mechanical forces that would ordinarily not result in a serious injury, such as a fall from or below standing level. The incidence of hip fractures in the United States is expected to increase as the population ages; estimates project 512,000 hip fractures with an associated cost of $16 billion annually by the year 2040.² Timely surgery is recommended for hip fracture patients as delayed surgery beyond 24 to 48 hours of presentation is associated with increased morbidity and mortality.^3-6 Time to surgery (TTS) has been shown to be the major potentially modifiable risk factor in the management of a hip fracture.⁷

Factors that have been noted to influence TTS include the American Society of Anesthesiologists’ (ASA) score, the day of the week of hospital admission, and preoperative testing.^8,9 Preoperative cardiology consultation and subsequent cardiac testing, in particular, can increase the TTS and length of stay (LOS) without changing perioperative management.^9,10 In our review of literature, we could not identify any studies specifically looking at the impact of preoperative specialty consults on short-term mortality or comparison of care provided by hospitalists alone versus additionally involving subspecialists such as cardiologists. To our knowledge, there are no studies that have categorized recommendations from a preoperative specialty consult as minor, moderate, or major.

Our study evaluated whether preoperative specialty consults meaningfully change management and influence outcomes for hip fracture patients. At our institution, all hip fracture patients are admitted to the hospitalist service and comanaged with the orthopedic team. The hospitalist physician performs the preoperative evaluation as part of the admission history and physical exam. Preoperative specialty consult(s), if needed, are requested only by the hospitalist team. A consultant such as a cardiologist provides input; however, final management decisions are coordinated by the hospitalist physician.

Study Design

We performed a retrospective cohort study of patients aged 50 years and older who underwent surgery for an isolated fragility fracture of the hip at Hartford Hospital, a level one trauma and tertiary care medical center, within the 24-month period from April 2015 to March 2017. Fragility hip fracture is defined as one occurring from a fall of a height of standing or less. A consult referred to a specialty or subspecialty consultation, other than hospital medicine, obtained prior to surgery. Patients with additional skeletal trauma and periprosthetic fractures were excluded. A total of 491 unique patients met the inclusion criteria, and data were obtained from chart review and an orthopedic surgery registry. The Hartford Hospital Institutional Review Board approved this study.

Our primary predictor was the presence or absence of a preoperative specialty consultation requested by the hospitalist. We also analyzed the following: covariates of demographics (age, sex, race), the ASA score, and severity of comorbidities using the Charlson comorbidity index (CCI) with a Quan modification;¹¹ “R program package, International Classification of Disease (ICD)”¹² was used to calculate the CCI using ICD-9 and ICD-10 diagnostic codes.

The primary outcome measures were TTS (measured in hours), LOS (measured in days), complications, and preoperative specialty consult resulting in a change in perioperative management. TTS was defined as the time elapsed from the presentation at the emergency department (ED) to surgery start. For transfer or direct admission patients, the time of admission was used in place of time of presentation. The measured complications included postoperative venous thromboembolic events, surgical site infection, myocardial infarction, stroke, and sepsis. Secondary outcome measures included 30-day mortality, readmission rate, and rate of return to OR. There were no elective or planned readmissions postoperatively on review of our institution’s orthopedic surgery registry.

Our team performed an extensive chart review including reviewing the admission note, consulting physician notes, and relevant test results. Our senior investigator (MK) then rated each preoperative specialty consult on appropriateness, the relative strength of the consultant’s recommendation, and resulting change in perioperative management. Cardiology consultations were deemed reasonable if a patient’s cardiac risk was considered elevated by the admitting physician or an active cardiac condition was present (suggestion of or clear evidence for acute coronary syndrome, acute congestive heart failure, uncontrolled arrhythmia, or symptomatic valvular disease). The determination of “elevated cardiac risk” was made, if admit note contained verbiage expressing concern for further evaluation for cardiac issues or words such as “high risk” or “elevated risk”. A specific guideline-based score such as the revised cardiac risk index was not consistently available in this retrospective chart review. A noncardiology consult was deemed reasonable only if it would have been warranted for the specific clinical situation­—for example, a neurology consult for an acute stroke or a pulmonary consult for acute respiratory failure. Consult recommendations or outcomes were rated as minor, moderate, or major (see Table 1 for detailed criteria). Some consults may generate more than one recommendation, in these cases, we determined that a major recommendation supersedes a moderate or minor recommendation and only one was counted in the final analysis. Next we determined if a consult recommendation led to a change in perioperative or therapeutic management, defined as a medication or dosage change, need to delay surgery to stabilize an unstable medical condition, invasive procedures (such as thoracentesis or cardiac catheterization) or change in postoperative monitoring. As a way of clarification, a consult may have a minor recommendation such as an EKG but if no other recommendations were given and there was no change in therapeutic management such as a medication change, this would be considered as a “no change”.

Statistical Analysis

The main analyses compared the two patient subgroups (with or without preoperative specialty consults) around outcome measures. Primary outcome measures were TTS, LOS, complications, and consult resulting in a change in perioperative management. Secondary outcome measures were 30-day readmit, return to OR, and mortality. A preliminary analysis was conducted to explore distributions for TTS and LOS. As expected, none met the assumptions of normality and were thus analyzed with Wilcoxon ranked-sum tests. The other outcomes were dichotomous and analyzed with chi-square tests of proportion or Fisher’s exact test when the expected cell frequencies were too low. Dichotomized variables for TTS (within 24 hours and 48 hours) and LOS (within five days, the median LOS for this cohort) were calculated and subsequently analyzed with additional chi-square tests of proportion or Fisher’s exact test¹³. To explore the effect of preoperative specialty consults independent of potential confounders, logistic regression analyses predicting each of the dichotomous outcomes were conducted with age and CCI used as predictors in addition to the main variable of whether or not there was a preoperative specialty consult. Since the CCI and ASA scores were highly intercorrelated, only the former was chosen for the multivariate analyses based on the consistent algorithm used to calculate CCI.

Additional analyses with the subgroup of patients with a preoperative specialty consult explored whether the consult was reasonable, the relative strength of resulting recommendation and whether it resulted in a change in management. The statistical approach used was the same as for the other dichotomous outcomes. All analyses used 0.05 as the level of statistical significance; SPSSv21 (IBM, Armonk, New York) was the statistical software used.

The sample size for this retrospective analysis was determined by the available number of patients meeting the inclusion criteria. An a priori power calculation was done to determine if the expected volume would be sufficient for the multivariate analysis; the presence of a complication was selected for calculation. Based on an expected volume of approximately 500 and an estimate of a 10% serious complication rate, it was determined that the sample could support the analysis of up to five predictor variables, sufficient for the main variable and four potential confounders; this was considered adequate.¹⁴ Propensity scoring was considered but did not offer any advantages to logistic regression because we only had two observed covariates: CCI and age.

A total of 491 unique patients met our inclusion criteria, 177 patients had a preoperative specialty consult. Of these 177 patients, 24 patients had more than one consult; hence, the total number of consults was 201. Most of the consults were cardiology (159). Others were Infectious disease (11), Pulmonology (10), Neurology (7), and Miscellaneous (14, which included Nephrology, Gastroenterology, Hematology, and Oncology).

No significant differences were found between the consult and no-consult groups with respect to gender, race, body mass index, type of anesthesia, and day of the week of surgery. We did note that patients with a consult were older and had a significantly higher CCI and ASA score (Table 2).

Patients with preoperative specialty consults were older and had more comorbidities than patients without consults. Our findings suggest that consults contribute to delays to surgery and may lead to higher LOS and higher risk of 30-day readmission after controlling for age and comorbidities in a multivariate analysis. This observation is significant considering that consults were requested more frequently on patients with a higher comorbidity burden and included patients who did not get additional preoperative testing, suggesting that a delay from waiting for a consult alone may be deleterious. This was a unique observation in our study; prior studies examining this subject have attributed delays to additional testing and not consults alone. Even though most consult requests appear to be reasonable according to our criteria, the majority of recommendations were minor (72.9%), and 62.7% of consults resulted in no change in perioperative management. Major changes in perioperative management were noted in only 4.5% of patients.

Our finding that a majority of patients in the consult group had no significant change in perioperative management raises an important area of potential improvement in the care of hip fracture patients. We believe that narrowing indications for preoperative specialty consults may result in shorter TTS and LOS for this group of frail elderly patients without sacrificing the quality of care. Since all patients in our study were comanaged by hospitalists and patients without additional consults had similar or better outcomes, we believe that hospitalist physicians are well positioned to provide standardized comanagement to this patient group without additional consultation unless absolutely necessary.

The primary limitation of our study was that this was a retrospective case analysis. The designation of minor, moderate, or major recommendation was done after the consults were already completed, and it may not be possible to predict that a consult results in no change without it being actually performed. Additionally, our classification of recommendations is somewhat arbitrary and subjective; for example, some readers might argue that a medication change counts as a moderate recommendation. We rated a medication change to be minor as we believe that an experienced hospitalist may likely make such management decisions on their own, and if this is the only recommendation from a consult, it is not additional information critical to patient care. There may also be an “unmeasured complexity” noted by the admitting physician, which was not necessarily accounted for by multivariate analysis of age and CCI but one that led to higher mortality and readmissions. However, we feel that this “unmeasured complexity” is likely inconsequential as the vast majority of consults did not result in any change in management. We did adjust for covariates as noted, but some confounding by indication is likely to remain. Additionally, categorization of consult recommendations and consequent changes by one physician could be considered subjective. We did control for this by having another physician review the entire dataset and rate it independently for interrater reliability with excellent correlation and kappa, although these may be inflated to some degree because our chart review did not account for variability among chart extractors.

A prospective evaluation of a clinical protocol that delineates reasonable indications for a preoperative consult would be helpful to validate our findings. In our study, we noted moderate or major recommendations from a cardiologist only in cases where an active cardiac condition was suspected by the hospitalist requesting the consult; hence, limiting preoperative cardiology consults to active cardiac conditions may be a reasonable approach to evaluate in a prospective study.

In conclusion, a majority of preoperative specialty consults do not appear to meaningfully influence management and may indirectly increase morbidity by delaying surgery and extending hospital stays. Our data suggest that unless the patient is clinically unstable and likely to require active management by a consultant prior to hip fracture repair, consults may offer limited benefit. Appropriately standardized perioperative management of this patient group by hospitalist physicians appears to manage most hip fracture patients as effectively with faster TTS and shorter hospital LOS.

Acknowledgments

The authors would like to thank John Corradi, PhD (Research Department at Hartford Hospital) for his input in calculating the Charlson comorbidity index.^12,13

Hip fractures in the elderly are associated with significant morbidity and mortality.¹ These are typically fragility fractures since they are caused by mechanical forces that would ordinarily not result in a serious injury, such as a fall from or below standing level. The incidence of hip fractures in the United States is expected to increase as the population ages; estimates project 512,000 hip fractures with an associated cost of $16 billion annually by the year 2040.² Timely surgery is recommended for hip fracture patients as delayed surgery beyond 24 to 48 hours of presentation is associated with increased morbidity and mortality.^3-6 Time to surgery (TTS) has been shown to be the major potentially modifiable risk factor in the management of a hip fracture.⁷

Factors that have been noted to influence TTS include the American Society of Anesthesiologists’ (ASA) score, the day of the week of hospital admission, and preoperative testing.^8,9 Preoperative cardiology consultation and subsequent cardiac testing, in particular, can increase the TTS and length of stay (LOS) without changing perioperative management.^9,10 In our review of literature, we could not identify any studies specifically looking at the impact of preoperative specialty consults on short-term mortality or comparison of care provided by hospitalists alone versus additionally involving subspecialists such as cardiologists. To our knowledge, there are no studies that have categorized recommendations from a preoperative specialty consult as minor, moderate, or major.

Our study evaluated whether preoperative specialty consults meaningfully change management and influence outcomes for hip fracture patients. At our institution, all hip fracture patients are admitted to the hospitalist service and comanaged with the orthopedic team. The hospitalist physician performs the preoperative evaluation as part of the admission history and physical exam. Preoperative specialty consult(s), if needed, are requested only by the hospitalist team. A consultant such as a cardiologist provides input; however, final management decisions are coordinated by the hospitalist physician.

Study Design

We performed a retrospective cohort study of patients aged 50 years and older who underwent surgery for an isolated fragility fracture of the hip at Hartford Hospital, a level one trauma and tertiary care medical center, within the 24-month period from April 2015 to March 2017. Fragility hip fracture is defined as one occurring from a fall of a height of standing or less. A consult referred to a specialty or subspecialty consultation, other than hospital medicine, obtained prior to surgery. Patients with additional skeletal trauma and periprosthetic fractures were excluded. A total of 491 unique patients met the inclusion criteria, and data were obtained from chart review and an orthopedic surgery registry. The Hartford Hospital Institutional Review Board approved this study.

Our primary predictor was the presence or absence of a preoperative specialty consultation requested by the hospitalist. We also analyzed the following: covariates of demographics (age, sex, race), the ASA score, and severity of comorbidities using the Charlson comorbidity index (CCI) with a Quan modification;¹¹ “R program package, International Classification of Disease (ICD)”¹² was used to calculate the CCI using ICD-9 and ICD-10 diagnostic codes.

The primary outcome measures were TTS (measured in hours), LOS (measured in days), complications, and preoperative specialty consult resulting in a change in perioperative management. TTS was defined as the time elapsed from the presentation at the emergency department (ED) to surgery start. For transfer or direct admission patients, the time of admission was used in place of time of presentation. The measured complications included postoperative venous thromboembolic events, surgical site infection, myocardial infarction, stroke, and sepsis. Secondary outcome measures included 30-day mortality, readmission rate, and rate of return to OR. There were no elective or planned readmissions postoperatively on review of our institution’s orthopedic surgery registry.

Our team performed an extensive chart review including reviewing the admission note, consulting physician notes, and relevant test results. Our senior investigator (MK) then rated each preoperative specialty consult on appropriateness, the relative strength of the consultant’s recommendation, and resulting change in perioperative management. Cardiology consultations were deemed reasonable if a patient’s cardiac risk was considered elevated by the admitting physician or an active cardiac condition was present (suggestion of or clear evidence for acute coronary syndrome, acute congestive heart failure, uncontrolled arrhythmia, or symptomatic valvular disease). The determination of “elevated cardiac risk” was made, if admit note contained verbiage expressing concern for further evaluation for cardiac issues or words such as “high risk” or “elevated risk”. A specific guideline-based score such as the revised cardiac risk index was not consistently available in this retrospective chart review. A noncardiology consult was deemed reasonable only if it would have been warranted for the specific clinical situation­—for example, a neurology consult for an acute stroke or a pulmonary consult for acute respiratory failure. Consult recommendations or outcomes were rated as minor, moderate, or major (see Table 1 for detailed criteria). Some consults may generate more than one recommendation, in these cases, we determined that a major recommendation supersedes a moderate or minor recommendation and only one was counted in the final analysis. Next we determined if a consult recommendation led to a change in perioperative or therapeutic management, defined as a medication or dosage change, need to delay surgery to stabilize an unstable medical condition, invasive procedures (such as thoracentesis or cardiac catheterization) or change in postoperative monitoring. As a way of clarification, a consult may have a minor recommendation such as an EKG but if no other recommendations were given and there was no change in therapeutic management such as a medication change, this would be considered as a “no change”.

Statistical Analysis

The main analyses compared the two patient subgroups (with or without preoperative specialty consults) around outcome measures. Primary outcome measures were TTS, LOS, complications, and consult resulting in a change in perioperative management. Secondary outcome measures were 30-day readmit, return to OR, and mortality. A preliminary analysis was conducted to explore distributions for TTS and LOS. As expected, none met the assumptions of normality and were thus analyzed with Wilcoxon ranked-sum tests. The other outcomes were dichotomous and analyzed with chi-square tests of proportion or Fisher’s exact test when the expected cell frequencies were too low. Dichotomized variables for TTS (within 24 hours and 48 hours) and LOS (within five days, the median LOS for this cohort) were calculated and subsequently analyzed with additional chi-square tests of proportion or Fisher’s exact test¹³. To explore the effect of preoperative specialty consults independent of potential confounders, logistic regression analyses predicting each of the dichotomous outcomes were conducted with age and CCI used as predictors in addition to the main variable of whether or not there was a preoperative specialty consult. Since the CCI and ASA scores were highly intercorrelated, only the former was chosen for the multivariate analyses based on the consistent algorithm used to calculate CCI.

Additional analyses with the subgroup of patients with a preoperative specialty consult explored whether the consult was reasonable, the relative strength of resulting recommendation and whether it resulted in a change in management. The statistical approach used was the same as for the other dichotomous outcomes. All analyses used 0.05 as the level of statistical significance; SPSSv21 (IBM, Armonk, New York) was the statistical software used.

The sample size for this retrospective analysis was determined by the available number of patients meeting the inclusion criteria. An a priori power calculation was done to determine if the expected volume would be sufficient for the multivariate analysis; the presence of a complication was selected for calculation. Based on an expected volume of approximately 500 and an estimate of a 10% serious complication rate, it was determined that the sample could support the analysis of up to five predictor variables, sufficient for the main variable and four potential confounders; this was considered adequate.¹⁴ Propensity scoring was considered but did not offer any advantages to logistic regression because we only had two observed covariates: CCI and age.

A total of 491 unique patients met our inclusion criteria, 177 patients had a preoperative specialty consult. Of these 177 patients, 24 patients had more than one consult; hence, the total number of consults was 201. Most of the consults were cardiology (159). Others were Infectious disease (11), Pulmonology (10), Neurology (7), and Miscellaneous (14, which included Nephrology, Gastroenterology, Hematology, and Oncology).

No significant differences were found between the consult and no-consult groups with respect to gender, race, body mass index, type of anesthesia, and day of the week of surgery. We did note that patients with a consult were older and had a significantly higher CCI and ASA score (Table 2).

Patients with preoperative specialty consults were older and had more comorbidities than patients without consults. Our findings suggest that consults contribute to delays to surgery and may lead to higher LOS and higher risk of 30-day readmission after controlling for age and comorbidities in a multivariate analysis. This observation is significant considering that consults were requested more frequently on patients with a higher comorbidity burden and included patients who did not get additional preoperative testing, suggesting that a delay from waiting for a consult alone may be deleterious. This was a unique observation in our study; prior studies examining this subject have attributed delays to additional testing and not consults alone. Even though most consult requests appear to be reasonable according to our criteria, the majority of recommendations were minor (72.9%), and 62.7% of consults resulted in no change in perioperative management. Major changes in perioperative management were noted in only 4.5% of patients.

Our finding that a majority of patients in the consult group had no significant change in perioperative management raises an important area of potential improvement in the care of hip fracture patients. We believe that narrowing indications for preoperative specialty consults may result in shorter TTS and LOS for this group of frail elderly patients without sacrificing the quality of care. Since all patients in our study were comanaged by hospitalists and patients without additional consults had similar or better outcomes, we believe that hospitalist physicians are well positioned to provide standardized comanagement to this patient group without additional consultation unless absolutely necessary.

The primary limitation of our study was that this was a retrospective case analysis. The designation of minor, moderate, or major recommendation was done after the consults were already completed, and it may not be possible to predict that a consult results in no change without it being actually performed. Additionally, our classification of recommendations is somewhat arbitrary and subjective; for example, some readers might argue that a medication change counts as a moderate recommendation. We rated a medication change to be minor as we believe that an experienced hospitalist may likely make such management decisions on their own, and if this is the only recommendation from a consult, it is not additional information critical to patient care. There may also be an “unmeasured complexity” noted by the admitting physician, which was not necessarily accounted for by multivariate analysis of age and CCI but one that led to higher mortality and readmissions. However, we feel that this “unmeasured complexity” is likely inconsequential as the vast majority of consults did not result in any change in management. We did adjust for covariates as noted, but some confounding by indication is likely to remain. Additionally, categorization of consult recommendations and consequent changes by one physician could be considered subjective. We did control for this by having another physician review the entire dataset and rate it independently for interrater reliability with excellent correlation and kappa, although these may be inflated to some degree because our chart review did not account for variability among chart extractors.

A prospective evaluation of a clinical protocol that delineates reasonable indications for a preoperative consult would be helpful to validate our findings. In our study, we noted moderate or major recommendations from a cardiologist only in cases where an active cardiac condition was suspected by the hospitalist requesting the consult; hence, limiting preoperative cardiology consults to active cardiac conditions may be a reasonable approach to evaluate in a prospective study.

In conclusion, a majority of preoperative specialty consults do not appear to meaningfully influence management and may indirectly increase morbidity by delaying surgery and extending hospital stays. Our data suggest that unless the patient is clinically unstable and likely to require active management by a consultant prior to hip fracture repair, consults may offer limited benefit. Appropriately standardized perioperative management of this patient group by hospitalist physicians appears to manage most hip fracture patients as effectively with faster TTS and shorter hospital LOS.

Acknowledgments

The authors would like to thank John Corradi, PhD (Research Department at Hartford Hospital) for his input in calculating the Charlson comorbidity index.^12,13

Hip fractures in the elderly are associated with significant morbidity and mortality.¹ These are typically fragility fractures since they are caused by mechanical forces that would ordinarily not result in a serious injury, such as a fall from or below standing level. The incidence of hip fractures in the United States is expected to increase as the population ages; estimates project 512,000 hip fractures with an associated cost of $16 billion annually by the year 2040.² Timely surgery is recommended for hip fracture patients as delayed surgery beyond 24 to 48 hours of presentation is associated with increased morbidity and mortality.^3-6 Time to surgery (TTS) has been shown to be the major potentially modifiable risk factor in the management of a hip fracture.⁷

Factors that have been noted to influence TTS include the American Society of Anesthesiologists’ (ASA) score, the day of the week of hospital admission, and preoperative testing.^8,9 Preoperative cardiology consultation and subsequent cardiac testing, in particular, can increase the TTS and length of stay (LOS) without changing perioperative management.^9,10 In our review of literature, we could not identify any studies specifically looking at the impact of preoperative specialty consults on short-term mortality or comparison of care provided by hospitalists alone versus additionally involving subspecialists such as cardiologists. To our knowledge, there are no studies that have categorized recommendations from a preoperative specialty consult as minor, moderate, or major.

Our study evaluated whether preoperative specialty consults meaningfully change management and influence outcomes for hip fracture patients. At our institution, all hip fracture patients are admitted to the hospitalist service and comanaged with the orthopedic team. The hospitalist physician performs the preoperative evaluation as part of the admission history and physical exam. Preoperative specialty consult(s), if needed, are requested only by the hospitalist team. A consultant such as a cardiologist provides input; however, final management decisions are coordinated by the hospitalist physician.

Study Design

We performed a retrospective cohort study of patients aged 50 years and older who underwent surgery for an isolated fragility fracture of the hip at Hartford Hospital, a level one trauma and tertiary care medical center, within the 24-month period from April 2015 to March 2017. Fragility hip fracture is defined as one occurring from a fall of a height of standing or less. A consult referred to a specialty or subspecialty consultation, other than hospital medicine, obtained prior to surgery. Patients with additional skeletal trauma and periprosthetic fractures were excluded. A total of 491 unique patients met the inclusion criteria, and data were obtained from chart review and an orthopedic surgery registry. The Hartford Hospital Institutional Review Board approved this study.

Our primary predictor was the presence or absence of a preoperative specialty consultation requested by the hospitalist. We also analyzed the following: covariates of demographics (age, sex, race), the ASA score, and severity of comorbidities using the Charlson comorbidity index (CCI) with a Quan modification;¹¹ “R program package, International Classification of Disease (ICD)”¹² was used to calculate the CCI using ICD-9 and ICD-10 diagnostic codes.

The primary outcome measures were TTS (measured in hours), LOS (measured in days), complications, and preoperative specialty consult resulting in a change in perioperative management. TTS was defined as the time elapsed from the presentation at the emergency department (ED) to surgery start. For transfer or direct admission patients, the time of admission was used in place of time of presentation. The measured complications included postoperative venous thromboembolic events, surgical site infection, myocardial infarction, stroke, and sepsis. Secondary outcome measures included 30-day mortality, readmission rate, and rate of return to OR. There were no elective or planned readmissions postoperatively on review of our institution’s orthopedic surgery registry.

Our team performed an extensive chart review including reviewing the admission note, consulting physician notes, and relevant test results. Our senior investigator (MK) then rated each preoperative specialty consult on appropriateness, the relative strength of the consultant’s recommendation, and resulting change in perioperative management. Cardiology consultations were deemed reasonable if a patient’s cardiac risk was considered elevated by the admitting physician or an active cardiac condition was present (suggestion of or clear evidence for acute coronary syndrome, acute congestive heart failure, uncontrolled arrhythmia, or symptomatic valvular disease). The determination of “elevated cardiac risk” was made, if admit note contained verbiage expressing concern for further evaluation for cardiac issues or words such as “high risk” or “elevated risk”. A specific guideline-based score such as the revised cardiac risk index was not consistently available in this retrospective chart review. A noncardiology consult was deemed reasonable only if it would have been warranted for the specific clinical situation­—for example, a neurology consult for an acute stroke or a pulmonary consult for acute respiratory failure. Consult recommendations or outcomes were rated as minor, moderate, or major (see Table 1 for detailed criteria). Some consults may generate more than one recommendation, in these cases, we determined that a major recommendation supersedes a moderate or minor recommendation and only one was counted in the final analysis. Next we determined if a consult recommendation led to a change in perioperative or therapeutic management, defined as a medication or dosage change, need to delay surgery to stabilize an unstable medical condition, invasive procedures (such as thoracentesis or cardiac catheterization) or change in postoperative monitoring. As a way of clarification, a consult may have a minor recommendation such as an EKG but if no other recommendations were given and there was no change in therapeutic management such as a medication change, this would be considered as a “no change”.

Statistical Analysis

The main analyses compared the two patient subgroups (with or without preoperative specialty consults) around outcome measures. Primary outcome measures were TTS, LOS, complications, and consult resulting in a change in perioperative management. Secondary outcome measures were 30-day readmit, return to OR, and mortality. A preliminary analysis was conducted to explore distributions for TTS and LOS. As expected, none met the assumptions of normality and were thus analyzed with Wilcoxon ranked-sum tests. The other outcomes were dichotomous and analyzed with chi-square tests of proportion or Fisher’s exact test when the expected cell frequencies were too low. Dichotomized variables for TTS (within 24 hours and 48 hours) and LOS (within five days, the median LOS for this cohort) were calculated and subsequently analyzed with additional chi-square tests of proportion or Fisher’s exact test¹³. To explore the effect of preoperative specialty consults independent of potential confounders, logistic regression analyses predicting each of the dichotomous outcomes were conducted with age and CCI used as predictors in addition to the main variable of whether or not there was a preoperative specialty consult. Since the CCI and ASA scores were highly intercorrelated, only the former was chosen for the multivariate analyses based on the consistent algorithm used to calculate CCI.

Additional analyses with the subgroup of patients with a preoperative specialty consult explored whether the consult was reasonable, the relative strength of resulting recommendation and whether it resulted in a change in management. The statistical approach used was the same as for the other dichotomous outcomes. All analyses used 0.05 as the level of statistical significance; SPSSv21 (IBM, Armonk, New York) was the statistical software used.

The sample size for this retrospective analysis was determined by the available number of patients meeting the inclusion criteria. An a priori power calculation was done to determine if the expected volume would be sufficient for the multivariate analysis; the presence of a complication was selected for calculation. Based on an expected volume of approximately 500 and an estimate of a 10% serious complication rate, it was determined that the sample could support the analysis of up to five predictor variables, sufficient for the main variable and four potential confounders; this was considered adequate.¹⁴ Propensity scoring was considered but did not offer any advantages to logistic regression because we only had two observed covariates: CCI and age.

A total of 491 unique patients met our inclusion criteria, 177 patients had a preoperative specialty consult. Of these 177 patients, 24 patients had more than one consult; hence, the total number of consults was 201. Most of the consults were cardiology (159). Others were Infectious disease (11), Pulmonology (10), Neurology (7), and Miscellaneous (14, which included Nephrology, Gastroenterology, Hematology, and Oncology).

No significant differences were found between the consult and no-consult groups with respect to gender, race, body mass index, type of anesthesia, and day of the week of surgery. We did note that patients with a consult were older and had a significantly higher CCI and ASA score (Table 2).

Patients with preoperative specialty consults were older and had more comorbidities than patients without consults. Our findings suggest that consults contribute to delays to surgery and may lead to higher LOS and higher risk of 30-day readmission after controlling for age and comorbidities in a multivariate analysis. This observation is significant considering that consults were requested more frequently on patients with a higher comorbidity burden and included patients who did not get additional preoperative testing, suggesting that a delay from waiting for a consult alone may be deleterious. This was a unique observation in our study; prior studies examining this subject have attributed delays to additional testing and not consults alone. Even though most consult requests appear to be reasonable according to our criteria, the majority of recommendations were minor (72.9%), and 62.7% of consults resulted in no change in perioperative management. Major changes in perioperative management were noted in only 4.5% of patients.

Our finding that a majority of patients in the consult group had no significant change in perioperative management raises an important area of potential improvement in the care of hip fracture patients. We believe that narrowing indications for preoperative specialty consults may result in shorter TTS and LOS for this group of frail elderly patients without sacrificing the quality of care. Since all patients in our study were comanaged by hospitalists and patients without additional consults had similar or better outcomes, we believe that hospitalist physicians are well positioned to provide standardized comanagement to this patient group without additional consultation unless absolutely necessary.

The primary limitation of our study was that this was a retrospective case analysis. The designation of minor, moderate, or major recommendation was done after the consults were already completed, and it may not be possible to predict that a consult results in no change without it being actually performed. Additionally, our classification of recommendations is somewhat arbitrary and subjective; for example, some readers might argue that a medication change counts as a moderate recommendation. We rated a medication change to be minor as we believe that an experienced hospitalist may likely make such management decisions on their own, and if this is the only recommendation from a consult, it is not additional information critical to patient care. There may also be an “unmeasured complexity” noted by the admitting physician, which was not necessarily accounted for by multivariate analysis of age and CCI but one that led to higher mortality and readmissions. However, we feel that this “unmeasured complexity” is likely inconsequential as the vast majority of consults did not result in any change in management. We did adjust for covariates as noted, but some confounding by indication is likely to remain. Additionally, categorization of consult recommendations and consequent changes by one physician could be considered subjective. We did control for this by having another physician review the entire dataset and rate it independently for interrater reliability with excellent correlation and kappa, although these may be inflated to some degree because our chart review did not account for variability among chart extractors.

A prospective evaluation of a clinical protocol that delineates reasonable indications for a preoperative consult would be helpful to validate our findings. In our study, we noted moderate or major recommendations from a cardiologist only in cases where an active cardiac condition was suspected by the hospitalist requesting the consult; hence, limiting preoperative cardiology consults to active cardiac conditions may be a reasonable approach to evaluate in a prospective study.

In conclusion, a majority of preoperative specialty consults do not appear to meaningfully influence management and may indirectly increase morbidity by delaying surgery and extending hospital stays. Our data suggest that unless the patient is clinically unstable and likely to require active management by a consultant prior to hip fracture repair, consults may offer limited benefit. Appropriately standardized perioperative management of this patient group by hospitalist physicians appears to manage most hip fracture patients as effectively with faster TTS and shorter hospital LOS.

Acknowledgments

The authors would like to thank John Corradi, PhD (Research Department at Hartford Hospital) for his input in calculating the Charlson comorbidity index.^12,13

As a newly recognized specialty, pediatric hospital medicine (PHM) continues to expand and diversify.¹ Pediatric hospitalists care for children in hospitals ranging from small, rural community hospitals to large, free-standing quaternary children’s hospitals.^2-4 In addition, more than 10% of graduating pediatric residents are seeking future careers within PHM.⁵

In 2018, Fromme et al. published a study describing clinical workload for pediatric hospitalists within university-based settings.⁶ They characterized the diversity of work models and programmatic sustainability but limited the study to university-based programs. With over half of children receiving care within community hospitals,⁷ workforce patterns for community-based pediatric hospitalists should be characterized to maximize sustainability and minimize attrition across the field.

In this study, we describe programmatic variability in clinical work expectations of 70 community-based PHM programs. We aimed to describe existing work models and expectations of community-based program leaders as they relate to their unique clinical setting.

We conducted a cross-sectional survey of community-based PHM site directors through structured interviews. Community hospital programs were self-defined by the study participants, although typically defined as general hospitals that admit pediatric patients and are not free-standing or children’s hospitals within a general hospital. Survey respondents were asked to answer questions only reflecting expectations at their community hospital.

Survey Design and Content

Building from a tool used by Fromme et al.⁶ we created a 12-question structured interview questionnaire focused on three areas: (1) full-time employment (FTE) metrics including definitions of a 1.0 FTE, “typical” shifts, and weekend responsibilities; (2) work volume including census parameters, service-line coverage expectations, back-up systems, and overnight call responsibilities; and (3) programmatic model including sense of sustainability (eg, minimizing burnout and attrition), support for activities such as administrative or research time, and employer model (Appendix).

We modified the survey through research team consensus. After pilot-testing by research team members at their own sites, the survey was refined for item clarity, structural design, and length. We chose to administer surveys through phone interviews over a traditional distribution due to anticipated variability in work models. The research team discussed how each question should be asked, and responses were clarified to maintain consistency.

Survey Administration

Given the absence of a national registry or database for community-based PHM programs, study participation was solicited through an invitation posted on the American Academy of Pediatrics Section on Hospital Medicine (AAP SOHM) Listserv and the AAP SOHM Community Hospitalist Listserv in May 2018. Invitations were posted twice at two weeks apart. Each research team member completed 6-19 interviews. Responses to survey questions were recorded in REDCap, a secure, web-based data capture instrument.⁸

Participating in the study was considered implied consent, and participants did not receive a monetary incentive, although respondents were offered deidentified survey data for participation. The study was exempted through the University of Chicago Institutional Review Board.

Data Analysis

Employers were dichotomized as community hospital employer (including primary community hospital employment/private organization) or noncommunity hospital employer (including children’s/university hospital employment or school of medicine). Descriptive statistics were reported to compare the demographics of two employer groups. P values were calculated using two-sample t-tests for the continuous variables and chi-square or Fisher-exact tests for the categorical variables. Mann–Whitney U-test was performed for continuous variables without normality. Analyses were performed using the R Statistical Programming Language (R Foundation for Statistical Computing, Vienna, Austria), version 3.4.3.

Participation and Program Characteristics

We interviewed 70 community-based PHM site directors representing programs across 29 states (Table 1) and five geographic regions: Midwest (34.3%), Northeast (11.4%), Southeast (15.7%), Southwest (4.3%), and West (34.3%). Employer models varied across groups, with more noncommunity hospital employers (57%) than community hospital employers (43%). The top three services covered by pediatric hospitalists were pediatric inpatient or observation bed admissions (97%), emergency department consults (89%), and general newborns (67%). PHM programs also provided coverage for other services, including newborn deliveries (43%), Special Care Nursery/Level II Neonatal Intensive Care Unit (41%), step-down unit (20%), and mental health units (13%). About 59% of programs provided education for family medicine residents, 36% were for pediatric residents, and 70% worked with advanced practice providers. The majority of programs (70%) provided in-house coverage overnight.

Clinical Work Expectations and Employer Model

Clinical work expectations varied broadly across programs (Table 2). The median expected hours for a 1.0 FTE was 1,882 hours per year (interquartile range [IQR] 1,805, 2,016), and the median expected weekend coverage/year (defined as covering two days or two nights of the weekend) was 21 (IQR 14, 24). Most programs did not expand staff coverage based on seasonality (73%), and less than 20% of programs operated with a census cap. Median support for nondirect patient care activities was 4% (IQR 0,10) of a program’s total FTE (ie, a 5.0 FTE program would have 0.20 FTE support). Programs with community hospital employers had an 8% higher expectation of 1.0 FTE hours/year (P = .01) and viewed an appropriate pediatric morning census as 20% higher (P = .01; Table 2).

Program Sustainability

To our knowledge, this study is the first to describe clinical work models exclusively for pediatric community hospitalist programs. We found that expectations for clinical FTE hours, weekend coverage, appropriate morning census, support for nondirect patient care activities, and perception of sustainability varied broadly across programs. The only variable affecting some of these differences was employer model, with those employed by a community hospital employer having a higher expectation for hours/year and appropriate morning pediatric census than those employed by noncommunity hospital employers.

With a growing emphasis on physician burnout and career satisfaction,^9-11 understanding the characteristics of community hospital work settings is critical for identifying and building sustainable employment models. Previous studies have identified that the balance of clinical and nonclinical responsibilities and the setting of community versus university-based programs are major contributors to burnout and career satisfaction.^9,11 Interestingly, although community hospital-based programs have limited FTE for nondirect patient care activities, we found that a higher percentage of program site directors perceived their program models as sustainable when compared with university-based programs in prior research (63% versus 50%).⁶ Elucidating why community hospital PHM programs are perceived as more sustainable provides an opportunity for future research. Potential reasons may include fewer academic requirements for promotion or an increased connection to a local community.

We also found that the employer model had a statistically significant impact on expected FTE hours per year but not on perception of sustainability. Programs employed by community hospitals worked 8% more hours per year than those employed by noncommunity hospital employers and accepted a higher morning pediatric census. This variation in hours and census level appropriateness is likely multifactorial, potentially from higher nonclinical expectations for promotion (eg, academic or scholarly production) at school of medicine or children’s hospital employed programs versus limited reimbursement for administrative responsibilities within community hospital employment models.

There are several potential next steps for our findings. As our data are the first attempt (to our knowledge) at describing the current practice and expectations exclusively within community hospital programs, this study can be used as a starting point for the development of workload expectation standards. Increasing transparency nationally for individual community programs potentially promotes discussions around burnout and attrition. Having objective data to compare program models may assist in advocating with local hospital leadership for restructuring that better aligns with national norms.

Our study has several limitations. First, our sampling frame was based upon a self-selection of program directors. This may have led to a biased representation of programs with higher workloads motivated to develop a standard to compare with other programs, which may have potentially led to an overestimation of hours. Second, without a registry or database for community-based pediatric hospitalist programs, we do not know the percentage of community-based programs that our sample represents. Although our results cannot speak for all community PHM programs, we attempted to mitigate nonresponse bias through the breadth of programs represented, which spanned 29 states, five geographic regions, and teaching and nonteaching programs. The interview-based method for data collection allowed the research team to clarify questions and responses across sites, thereby improving the quality and consistency of the data for the represented study sample. Finally, other factors possibly contributed to sustainability that we did not address in this study, such as programs that are dependent on billable encounters as part of their salary support.

As a newly recognized subspecialty, creating a reference for community-based program leaders to determine and discuss individual models and expectations with hospital administrators may help address programmatic sustainability. It may also allow for the analysis of long-term career satisfaction and longevity within community PHM programs based on workload. Future studies should further explore root causes for workload discrepancies between community and university employed programs along with establishing potential standards for PHM program development.

Acknowledgments

We would like to thank the Stanford School of Medicine Quantitative Sciences Unit staff for their assistance in statistical analysis.

Disclosure

The authors have nothing to disclose.

As a newly recognized specialty, pediatric hospital medicine (PHM) continues to expand and diversify.¹ Pediatric hospitalists care for children in hospitals ranging from small, rural community hospitals to large, free-standing quaternary children’s hospitals.^2-4 In addition, more than 10% of graduating pediatric residents are seeking future careers within PHM.⁵

In 2018, Fromme et al. published a study describing clinical workload for pediatric hospitalists within university-based settings.⁶ They characterized the diversity of work models and programmatic sustainability but limited the study to university-based programs. With over half of children receiving care within community hospitals,⁷ workforce patterns for community-based pediatric hospitalists should be characterized to maximize sustainability and minimize attrition across the field.

In this study, we describe programmatic variability in clinical work expectations of 70 community-based PHM programs. We aimed to describe existing work models and expectations of community-based program leaders as they relate to their unique clinical setting.

We conducted a cross-sectional survey of community-based PHM site directors through structured interviews. Community hospital programs were self-defined by the study participants, although typically defined as general hospitals that admit pediatric patients and are not free-standing or children’s hospitals within a general hospital. Survey respondents were asked to answer questions only reflecting expectations at their community hospital.

Survey Design and Content

Building from a tool used by Fromme et al.⁶ we created a 12-question structured interview questionnaire focused on three areas: (1) full-time employment (FTE) metrics including definitions of a 1.0 FTE, “typical” shifts, and weekend responsibilities; (2) work volume including census parameters, service-line coverage expectations, back-up systems, and overnight call responsibilities; and (3) programmatic model including sense of sustainability (eg, minimizing burnout and attrition), support for activities such as administrative or research time, and employer model (Appendix).

We modified the survey through research team consensus. After pilot-testing by research team members at their own sites, the survey was refined for item clarity, structural design, and length. We chose to administer surveys through phone interviews over a traditional distribution due to anticipated variability in work models. The research team discussed how each question should be asked, and responses were clarified to maintain consistency.

Survey Administration

Given the absence of a national registry or database for community-based PHM programs, study participation was solicited through an invitation posted on the American Academy of Pediatrics Section on Hospital Medicine (AAP SOHM) Listserv and the AAP SOHM Community Hospitalist Listserv in May 2018. Invitations were posted twice at two weeks apart. Each research team member completed 6-19 interviews. Responses to survey questions were recorded in REDCap, a secure, web-based data capture instrument.⁸

Participating in the study was considered implied consent, and participants did not receive a monetary incentive, although respondents were offered deidentified survey data for participation. The study was exempted through the University of Chicago Institutional Review Board.

Data Analysis

Employers were dichotomized as community hospital employer (including primary community hospital employment/private organization) or noncommunity hospital employer (including children’s/university hospital employment or school of medicine). Descriptive statistics were reported to compare the demographics of two employer groups. P values were calculated using two-sample t-tests for the continuous variables and chi-square or Fisher-exact tests for the categorical variables. Mann–Whitney U-test was performed for continuous variables without normality. Analyses were performed using the R Statistical Programming Language (R Foundation for Statistical Computing, Vienna, Austria), version 3.4.3.

Participation and Program Characteristics

We interviewed 70 community-based PHM site directors representing programs across 29 states (Table 1) and five geographic regions: Midwest (34.3%), Northeast (11.4%), Southeast (15.7%), Southwest (4.3%), and West (34.3%). Employer models varied across groups, with more noncommunity hospital employers (57%) than community hospital employers (43%). The top three services covered by pediatric hospitalists were pediatric inpatient or observation bed admissions (97%), emergency department consults (89%), and general newborns (67%). PHM programs also provided coverage for other services, including newborn deliveries (43%), Special Care Nursery/Level II Neonatal Intensive Care Unit (41%), step-down unit (20%), and mental health units (13%). About 59% of programs provided education for family medicine residents, 36% were for pediatric residents, and 70% worked with advanced practice providers. The majority of programs (70%) provided in-house coverage overnight.

Clinical Work Expectations and Employer Model

Clinical work expectations varied broadly across programs (Table 2). The median expected hours for a 1.0 FTE was 1,882 hours per year (interquartile range [IQR] 1,805, 2,016), and the median expected weekend coverage/year (defined as covering two days or two nights of the weekend) was 21 (IQR 14, 24). Most programs did not expand staff coverage based on seasonality (73%), and less than 20% of programs operated with a census cap. Median support for nondirect patient care activities was 4% (IQR 0,10) of a program’s total FTE (ie, a 5.0 FTE program would have 0.20 FTE support). Programs with community hospital employers had an 8% higher expectation of 1.0 FTE hours/year (P = .01) and viewed an appropriate pediatric morning census as 20% higher (P = .01; Table 2).

Program Sustainability

To our knowledge, this study is the first to describe clinical work models exclusively for pediatric community hospitalist programs. We found that expectations for clinical FTE hours, weekend coverage, appropriate morning census, support for nondirect patient care activities, and perception of sustainability varied broadly across programs. The only variable affecting some of these differences was employer model, with those employed by a community hospital employer having a higher expectation for hours/year and appropriate morning pediatric census than those employed by noncommunity hospital employers.

With a growing emphasis on physician burnout and career satisfaction,^9-11 understanding the characteristics of community hospital work settings is critical for identifying and building sustainable employment models. Previous studies have identified that the balance of clinical and nonclinical responsibilities and the setting of community versus university-based programs are major contributors to burnout and career satisfaction.^9,11 Interestingly, although community hospital-based programs have limited FTE for nondirect patient care activities, we found that a higher percentage of program site directors perceived their program models as sustainable when compared with university-based programs in prior research (63% versus 50%).⁶ Elucidating why community hospital PHM programs are perceived as more sustainable provides an opportunity for future research. Potential reasons may include fewer academic requirements for promotion or an increased connection to a local community.

We also found that the employer model had a statistically significant impact on expected FTE hours per year but not on perception of sustainability. Programs employed by community hospitals worked 8% more hours per year than those employed by noncommunity hospital employers and accepted a higher morning pediatric census. This variation in hours and census level appropriateness is likely multifactorial, potentially from higher nonclinical expectations for promotion (eg, academic or scholarly production) at school of medicine or children’s hospital employed programs versus limited reimbursement for administrative responsibilities within community hospital employment models.

There are several potential next steps for our findings. As our data are the first attempt (to our knowledge) at describing the current practice and expectations exclusively within community hospital programs, this study can be used as a starting point for the development of workload expectation standards. Increasing transparency nationally for individual community programs potentially promotes discussions around burnout and attrition. Having objective data to compare program models may assist in advocating with local hospital leadership for restructuring that better aligns with national norms.

Our study has several limitations. First, our sampling frame was based upon a self-selection of program directors. This may have led to a biased representation of programs with higher workloads motivated to develop a standard to compare with other programs, which may have potentially led to an overestimation of hours. Second, without a registry or database for community-based pediatric hospitalist programs, we do not know the percentage of community-based programs that our sample represents. Although our results cannot speak for all community PHM programs, we attempted to mitigate nonresponse bias through the breadth of programs represented, which spanned 29 states, five geographic regions, and teaching and nonteaching programs. The interview-based method for data collection allowed the research team to clarify questions and responses across sites, thereby improving the quality and consistency of the data for the represented study sample. Finally, other factors possibly contributed to sustainability that we did not address in this study, such as programs that are dependent on billable encounters as part of their salary support.

As a newly recognized subspecialty, creating a reference for community-based program leaders to determine and discuss individual models and expectations with hospital administrators may help address programmatic sustainability. It may also allow for the analysis of long-term career satisfaction and longevity within community PHM programs based on workload. Future studies should further explore root causes for workload discrepancies between community and university employed programs along with establishing potential standards for PHM program development.

Acknowledgments

We would like to thank the Stanford School of Medicine Quantitative Sciences Unit staff for their assistance in statistical analysis.

Disclosure

The authors have nothing to disclose.

As a newly recognized specialty, pediatric hospital medicine (PHM) continues to expand and diversify.¹ Pediatric hospitalists care for children in hospitals ranging from small, rural community hospitals to large, free-standing quaternary children’s hospitals.^2-4 In addition, more than 10% of graduating pediatric residents are seeking future careers within PHM.⁵

In 2018, Fromme et al. published a study describing clinical workload for pediatric hospitalists within university-based settings.⁶ They characterized the diversity of work models and programmatic sustainability but limited the study to university-based programs. With over half of children receiving care within community hospitals,⁷ workforce patterns for community-based pediatric hospitalists should be characterized to maximize sustainability and minimize attrition across the field.

In this study, we describe programmatic variability in clinical work expectations of 70 community-based PHM programs. We aimed to describe existing work models and expectations of community-based program leaders as they relate to their unique clinical setting.

We conducted a cross-sectional survey of community-based PHM site directors through structured interviews. Community hospital programs were self-defined by the study participants, although typically defined as general hospitals that admit pediatric patients and are not free-standing or children’s hospitals within a general hospital. Survey respondents were asked to answer questions only reflecting expectations at their community hospital.

Survey Design and Content

Building from a tool used by Fromme et al.⁶ we created a 12-question structured interview questionnaire focused on three areas: (1) full-time employment (FTE) metrics including definitions of a 1.0 FTE, “typical” shifts, and weekend responsibilities; (2) work volume including census parameters, service-line coverage expectations, back-up systems, and overnight call responsibilities; and (3) programmatic model including sense of sustainability (eg, minimizing burnout and attrition), support for activities such as administrative or research time, and employer model (Appendix).

We modified the survey through research team consensus. After pilot-testing by research team members at their own sites, the survey was refined for item clarity, structural design, and length. We chose to administer surveys through phone interviews over a traditional distribution due to anticipated variability in work models. The research team discussed how each question should be asked, and responses were clarified to maintain consistency.

Survey Administration

Given the absence of a national registry or database for community-based PHM programs, study participation was solicited through an invitation posted on the American Academy of Pediatrics Section on Hospital Medicine (AAP SOHM) Listserv and the AAP SOHM Community Hospitalist Listserv in May 2018. Invitations were posted twice at two weeks apart. Each research team member completed 6-19 interviews. Responses to survey questions were recorded in REDCap, a secure, web-based data capture instrument.⁸

Participating in the study was considered implied consent, and participants did not receive a monetary incentive, although respondents were offered deidentified survey data for participation. The study was exempted through the University of Chicago Institutional Review Board.

Data Analysis

Employers were dichotomized as community hospital employer (including primary community hospital employment/private organization) or noncommunity hospital employer (including children’s/university hospital employment or school of medicine). Descriptive statistics were reported to compare the demographics of two employer groups. P values were calculated using two-sample t-tests for the continuous variables and chi-square or Fisher-exact tests for the categorical variables. Mann–Whitney U-test was performed for continuous variables without normality. Analyses were performed using the R Statistical Programming Language (R Foundation for Statistical Computing, Vienna, Austria), version 3.4.3.

Participation and Program Characteristics

We interviewed 70 community-based PHM site directors representing programs across 29 states (Table 1) and five geographic regions: Midwest (34.3%), Northeast (11.4%), Southeast (15.7%), Southwest (4.3%), and West (34.3%). Employer models varied across groups, with more noncommunity hospital employers (57%) than community hospital employers (43%). The top three services covered by pediatric hospitalists were pediatric inpatient or observation bed admissions (97%), emergency department consults (89%), and general newborns (67%). PHM programs also provided coverage for other services, including newborn deliveries (43%), Special Care Nursery/Level II Neonatal Intensive Care Unit (41%), step-down unit (20%), and mental health units (13%). About 59% of programs provided education for family medicine residents, 36% were for pediatric residents, and 70% worked with advanced practice providers. The majority of programs (70%) provided in-house coverage overnight.

Clinical Work Expectations and Employer Model

Clinical work expectations varied broadly across programs (Table 2). The median expected hours for a 1.0 FTE was 1,882 hours per year (interquartile range [IQR] 1,805, 2,016), and the median expected weekend coverage/year (defined as covering two days or two nights of the weekend) was 21 (IQR 14, 24). Most programs did not expand staff coverage based on seasonality (73%), and less than 20% of programs operated with a census cap. Median support for nondirect patient care activities was 4% (IQR 0,10) of a program’s total FTE (ie, a 5.0 FTE program would have 0.20 FTE support). Programs with community hospital employers had an 8% higher expectation of 1.0 FTE hours/year (P = .01) and viewed an appropriate pediatric morning census as 20% higher (P = .01; Table 2).

Program Sustainability

To our knowledge, this study is the first to describe clinical work models exclusively for pediatric community hospitalist programs. We found that expectations for clinical FTE hours, weekend coverage, appropriate morning census, support for nondirect patient care activities, and perception of sustainability varied broadly across programs. The only variable affecting some of these differences was employer model, with those employed by a community hospital employer having a higher expectation for hours/year and appropriate morning pediatric census than those employed by noncommunity hospital employers.

With a growing emphasis on physician burnout and career satisfaction,^9-11 understanding the characteristics of community hospital work settings is critical for identifying and building sustainable employment models. Previous studies have identified that the balance of clinical and nonclinical responsibilities and the setting of community versus university-based programs are major contributors to burnout and career satisfaction.^9,11 Interestingly, although community hospital-based programs have limited FTE for nondirect patient care activities, we found that a higher percentage of program site directors perceived their program models as sustainable when compared with university-based programs in prior research (63% versus 50%).⁶ Elucidating why community hospital PHM programs are perceived as more sustainable provides an opportunity for future research. Potential reasons may include fewer academic requirements for promotion or an increased connection to a local community.

We also found that the employer model had a statistically significant impact on expected FTE hours per year but not on perception of sustainability. Programs employed by community hospitals worked 8% more hours per year than those employed by noncommunity hospital employers and accepted a higher morning pediatric census. This variation in hours and census level appropriateness is likely multifactorial, potentially from higher nonclinical expectations for promotion (eg, academic or scholarly production) at school of medicine or children’s hospital employed programs versus limited reimbursement for administrative responsibilities within community hospital employment models.

There are several potential next steps for our findings. As our data are the first attempt (to our knowledge) at describing the current practice and expectations exclusively within community hospital programs, this study can be used as a starting point for the development of workload expectation standards. Increasing transparency nationally for individual community programs potentially promotes discussions around burnout and attrition. Having objective data to compare program models may assist in advocating with local hospital leadership for restructuring that better aligns with national norms.

Our study has several limitations. First, our sampling frame was based upon a self-selection of program directors. This may have led to a biased representation of programs with higher workloads motivated to develop a standard to compare with other programs, which may have potentially led to an overestimation of hours. Second, without a registry or database for community-based pediatric hospitalist programs, we do not know the percentage of community-based programs that our sample represents. Although our results cannot speak for all community PHM programs, we attempted to mitigate nonresponse bias through the breadth of programs represented, which spanned 29 states, five geographic regions, and teaching and nonteaching programs. The interview-based method for data collection allowed the research team to clarify questions and responses across sites, thereby improving the quality and consistency of the data for the represented study sample. Finally, other factors possibly contributed to sustainability that we did not address in this study, such as programs that are dependent on billable encounters as part of their salary support.

As a newly recognized subspecialty, creating a reference for community-based program leaders to determine and discuss individual models and expectations with hospital administrators may help address programmatic sustainability. It may also allow for the analysis of long-term career satisfaction and longevity within community PHM programs based on workload. Future studies should further explore root causes for workload discrepancies between community and university employed programs along with establishing potential standards for PHM program development.

Acknowledgments

We would like to thank the Stanford School of Medicine Quantitative Sciences Unit staff for their assistance in statistical analysis.

Disclosure

The authors have nothing to disclose.

One of the most promising methods for improving medical decision-making is learning from the outcomes of one’s decisions and either maintaining or modifying future decision-making based on those outcomes.^1-3 This process of iterative improvement over time based on feedback is called calibration and is one of the most important drivers of lifelong learning and improvement.¹

Despite the importance of knowing the outcomes of one’s decisions, this seldom occurs in modern medical education.⁴ Learners do not often obtain specific feedback about the decisions they make within a short enough time frame to intentionally reflect upon and modify that decision-making process.^3,5 In addition, almost every patient admitted to a teaching hospital will be cared for by multiple physicians over the course of a hospitalization. These care transitions may be seen as barriers to high-quality care and education, but we suggest a different paradigm: transitions of care present opportunities for trainees to be teammates in each other’s calibration. Peers can provide specific feedback about the diagnostic process and inform one another about patient outcomes. Transitions of care allow for built-in “second opinions,” and trainees can intentionally learn by comparing the clinical reasoning involved at different points in a patient’s course. The diagnostic process is dynamic and complex; it is fundamental that trainees have the opportunity to reflect on the process to identify how and why the diagnostic process evolved throughout a patient’s hospitalization. Most inpatient diagnoses are “working diagnoses” that are likely to change. Thus, identifying the twists and turns in a patient’s diagnostic journey provides invaluable learning for future practice.

Herein, we describe the implementation and impact of a multisite initiative to engage residents in delivering feedback to their peers about medical decisions around transitions of care.

The LOOP Project is a prospective clinical educational study that aimed to engage resident physicians to deliver feedback and updates about their colleagues’ diagnostic decision-making around care transitions. This study was deemed exempt from review by the University of Minnesota Institutional Review Board and either approved or deemed exempt by the corresponding Institutional Review Boards at all participating institutions. The study was conducted by seven programs at six institutions and included Internal Medicine, Pediatrics, and Internal Medicine–Pediatrics (PGY 1-4) residents from February 2017 to June 2017. Residents rotating through participating clinical services during the study period were invited to participate and given further information by site leads via informational presentations, written handouts, and/or emails.

The intervention entailed residents delivering structured feedback to their colleagues regarding their patients’ diagnoses after transitions of care. The predominant setting was the inpatient hospital medicine day-shift team providing feedback to the night-shift team regarding overnight admissions. Feedback about patients (usually chosen by the day-shift team) was delivered through completion of a standard templated form (Figure) usually sent within 24 hours after hospital admission through secure messaging (ie, EPIC In-Basket message utilizing a Smartphrase of the LOOP feedback form). A 24-hour time period was chosen to allow for rapid cycling of feedback focusing on initial diagnostic assessment. Site leads and resident champions promoted the project through presentations, informal discussions, and prizes for high completion rates of forms and surveys (ie, coffee cards and pizza).

Overall, there were 716 current residents in the training programs that participated in this study; one site planned on participating but did not complete any forms. A total of 405 residents were eligible to participate during the study period. Overall, 221 (54.5%) presurveys and 90 postsurveys were completed (22.2%); 54 residents (13.3%) completed both pre- and postsurveys and were included in the analysis. Of the 54 survey respondents, 26 (48.15%) were female.

Survey results (Table) indicated significantly improved self-efficacy in identifying cognitive errors in residents’ own practice, identifying why those errors occurred, and identifying strategies to decrease future diagnostic errors. Participants noted increased frequency of discussions within teams regarding differential diagnoses, diagnostic errors, and why diagnoses changed over time. The feedback process was viewed positively by participants, who were also generally satisfied with the overall quality, frequency, and value of the feedback received. After the intervention, participants reported an increase in the amount of feedback received for night admissions and an overall increase in the perception that nighttime admissions were as “educational” as daytime admissions.

This study suggests that an intervention to operationalize standardized, structured feedback about diagnostic decision-making around transitions of care is a promising approach to improve residents’ understanding of changes in, and evolution of, the diagnostic process, as well as improve the perceived educational value of overnight admissions. In our results, over 40% of the patients admitted by residents had some change in their diagnoses after a transition of care during their early hospitalization. This finding highlights the importance of ensuring that trainees have the opportunity to know the outcomes of their decisions. Indeed, residents should be encouraged to follow-up on their own patients without prompting; however, studies show that this practice is uncommon and interventions beyond admonition are necessary.⁴

The diagnostic change rate observed in this study confirms that diagnosis is an iterative process and that the concept of a working diagnosis is key—a diagnosis made at admission will very likely be modified by time, the natural history of the disease, and new clinical information. When diagnoses are viewed as working diagnoses, trainees may be empowered to better understand the diagnostic process. As learners and teachers adopt this perspective, training programs are more likely to be successful in helping learners calibrate toward expertise.

Previous studies have questioned whether resident physicians view overnight admissions as valuable.⁶ After our intervention, we found an increase in both the amount of feedback received and the proportion of participants who agreed that night and day admissions were equally educational, suggesting that targeted diagnostic reasoning feedback can bolster educational value of nighttime admissions.

This study presents a number of limitations. First, the survey response rate was low, which could potentially lead to biased results. We excluded those respondents who did not respond to both the pre- and postsurveys from the analysis. Second, we did not measure actual change in diagnostic performance. While learners did report learning and saw feedback as valuable, self-identified learning points may not always translate to improved patient care. Additionally, residents chose the patients for whom feedback was provided, and the diagnostic change rate described may be overestimated. We did not track the total number of admissions for which feedback could have been delivered during the study. We did not include a control group, and the intervention may not be responsible for changing learners’ perceptions. However, the included programs were not implementing other new protocols focused on diagnostic reasoning during the study period. In addition, we addressed diagnostic changes early in a hospital course; a comprehensive program should address more feedback loops (eg, discharging team to admitting team).

This work is a pilot study; for future interventions focused on improving calibration to be sustainable, they should be congruent with existing clinical workflows and avoid adding to the stress and/or cognitive load of an already-busy clinical experience. The most optimal strategies for delivering feedback about clinical reasoning remain unclear.

In summary, a program to deliver structured feedback among resident physicians about diagnostic reasoning across care transitions for selected hospitalized patients is viewed positively by trainees, is feasible, and leads to changes in resident perception and self-efficacy. Future studies and interventions should aim to provide feedback more systematically, rather than just for selected patients, and objectively track diagnostic changes over time in hospitalized patients. While truly objective diagnostic information is challenging to obtain, comparing admission and other inpatient diagnoses to discharge diagnoses or diagnoses from primary care follow-up visits may be helpful. In addition, studies should aim to track trainees’ clinical decision-making over time and determine the effectiveness of feedback at improving diagnostic performance through calibration.

Acknowledgments

The authors thank the trainees who participated in this study, as well as the residency leadership at participating institutions. The authors also thank Qi Wang, PhD, for providing statistical analysis.

Disclosures

The authors have nothing to disclose.

Funding

The study was funded by an AAIM Innovation Grant and local support at each participating institution.

One of the most promising methods for improving medical decision-making is learning from the outcomes of one’s decisions and either maintaining or modifying future decision-making based on those outcomes.^1-3 This process of iterative improvement over time based on feedback is called calibration and is one of the most important drivers of lifelong learning and improvement.¹

Despite the importance of knowing the outcomes of one’s decisions, this seldom occurs in modern medical education.⁴ Learners do not often obtain specific feedback about the decisions they make within a short enough time frame to intentionally reflect upon and modify that decision-making process.^3,5 In addition, almost every patient admitted to a teaching hospital will be cared for by multiple physicians over the course of a hospitalization. These care transitions may be seen as barriers to high-quality care and education, but we suggest a different paradigm: transitions of care present opportunities for trainees to be teammates in each other’s calibration. Peers can provide specific feedback about the diagnostic process and inform one another about patient outcomes. Transitions of care allow for built-in “second opinions,” and trainees can intentionally learn by comparing the clinical reasoning involved at different points in a patient’s course. The diagnostic process is dynamic and complex; it is fundamental that trainees have the opportunity to reflect on the process to identify how and why the diagnostic process evolved throughout a patient’s hospitalization. Most inpatient diagnoses are “working diagnoses” that are likely to change. Thus, identifying the twists and turns in a patient’s diagnostic journey provides invaluable learning for future practice.

Herein, we describe the implementation and impact of a multisite initiative to engage residents in delivering feedback to their peers about medical decisions around transitions of care.

The LOOP Project is a prospective clinical educational study that aimed to engage resident physicians to deliver feedback and updates about their colleagues’ diagnostic decision-making around care transitions. This study was deemed exempt from review by the University of Minnesota Institutional Review Board and either approved or deemed exempt by the corresponding Institutional Review Boards at all participating institutions. The study was conducted by seven programs at six institutions and included Internal Medicine, Pediatrics, and Internal Medicine–Pediatrics (PGY 1-4) residents from February 2017 to June 2017. Residents rotating through participating clinical services during the study period were invited to participate and given further information by site leads via informational presentations, written handouts, and/or emails.

The intervention entailed residents delivering structured feedback to their colleagues regarding their patients’ diagnoses after transitions of care. The predominant setting was the inpatient hospital medicine day-shift team providing feedback to the night-shift team regarding overnight admissions. Feedback about patients (usually chosen by the day-shift team) was delivered through completion of a standard templated form (Figure) usually sent within 24 hours after hospital admission through secure messaging (ie, EPIC In-Basket message utilizing a Smartphrase of the LOOP feedback form). A 24-hour time period was chosen to allow for rapid cycling of feedback focusing on initial diagnostic assessment. Site leads and resident champions promoted the project through presentations, informal discussions, and prizes for high completion rates of forms and surveys (ie, coffee cards and pizza).

Overall, there were 716 current residents in the training programs that participated in this study; one site planned on participating but did not complete any forms. A total of 405 residents were eligible to participate during the study period. Overall, 221 (54.5%) presurveys and 90 postsurveys were completed (22.2%); 54 residents (13.3%) completed both pre- and postsurveys and were included in the analysis. Of the 54 survey respondents, 26 (48.15%) were female.

Survey results (Table) indicated significantly improved self-efficacy in identifying cognitive errors in residents’ own practice, identifying why those errors occurred, and identifying strategies to decrease future diagnostic errors. Participants noted increased frequency of discussions within teams regarding differential diagnoses, diagnostic errors, and why diagnoses changed over time. The feedback process was viewed positively by participants, who were also generally satisfied with the overall quality, frequency, and value of the feedback received. After the intervention, participants reported an increase in the amount of feedback received for night admissions and an overall increase in the perception that nighttime admissions were as “educational” as daytime admissions.

This study suggests that an intervention to operationalize standardized, structured feedback about diagnostic decision-making around transitions of care is a promising approach to improve residents’ understanding of changes in, and evolution of, the diagnostic process, as well as improve the perceived educational value of overnight admissions. In our results, over 40% of the patients admitted by residents had some change in their diagnoses after a transition of care during their early hospitalization. This finding highlights the importance of ensuring that trainees have the opportunity to know the outcomes of their decisions. Indeed, residents should be encouraged to follow-up on their own patients without prompting; however, studies show that this practice is uncommon and interventions beyond admonition are necessary.⁴

The diagnostic change rate observed in this study confirms that diagnosis is an iterative process and that the concept of a working diagnosis is key—a diagnosis made at admission will very likely be modified by time, the natural history of the disease, and new clinical information. When diagnoses are viewed as working diagnoses, trainees may be empowered to better understand the diagnostic process. As learners and teachers adopt this perspective, training programs are more likely to be successful in helping learners calibrate toward expertise.

Previous studies have questioned whether resident physicians view overnight admissions as valuable.⁶ After our intervention, we found an increase in both the amount of feedback received and the proportion of participants who agreed that night and day admissions were equally educational, suggesting that targeted diagnostic reasoning feedback can bolster educational value of nighttime admissions.

This study presents a number of limitations. First, the survey response rate was low, which could potentially lead to biased results. We excluded those respondents who did not respond to both the pre- and postsurveys from the analysis. Second, we did not measure actual change in diagnostic performance. While learners did report learning and saw feedback as valuable, self-identified learning points may not always translate to improved patient care. Additionally, residents chose the patients for whom feedback was provided, and the diagnostic change rate described may be overestimated. We did not track the total number of admissions for which feedback could have been delivered during the study. We did not include a control group, and the intervention may not be responsible for changing learners’ perceptions. However, the included programs were not implementing other new protocols focused on diagnostic reasoning during the study period. In addition, we addressed diagnostic changes early in a hospital course; a comprehensive program should address more feedback loops (eg, discharging team to admitting team).

This work is a pilot study; for future interventions focused on improving calibration to be sustainable, they should be congruent with existing clinical workflows and avoid adding to the stress and/or cognitive load of an already-busy clinical experience. The most optimal strategies for delivering feedback about clinical reasoning remain unclear.

In summary, a program to deliver structured feedback among resident physicians about diagnostic reasoning across care transitions for selected hospitalized patients is viewed positively by trainees, is feasible, and leads to changes in resident perception and self-efficacy. Future studies and interventions should aim to provide feedback more systematically, rather than just for selected patients, and objectively track diagnostic changes over time in hospitalized patients. While truly objective diagnostic information is challenging to obtain, comparing admission and other inpatient diagnoses to discharge diagnoses or diagnoses from primary care follow-up visits may be helpful. In addition, studies should aim to track trainees’ clinical decision-making over time and determine the effectiveness of feedback at improving diagnostic performance through calibration.

Acknowledgments

The authors thank the trainees who participated in this study, as well as the residency leadership at participating institutions. The authors also thank Qi Wang, PhD, for providing statistical analysis.

Disclosures

The authors have nothing to disclose.

Funding

The study was funded by an AAIM Innovation Grant and local support at each participating institution.

One of the most promising methods for improving medical decision-making is learning from the outcomes of one’s decisions and either maintaining or modifying future decision-making based on those outcomes.^1-3 This process of iterative improvement over time based on feedback is called calibration and is one of the most important drivers of lifelong learning and improvement.¹

Despite the importance of knowing the outcomes of one’s decisions, this seldom occurs in modern medical education.⁴ Learners do not often obtain specific feedback about the decisions they make within a short enough time frame to intentionally reflect upon and modify that decision-making process.^3,5 In addition, almost every patient admitted to a teaching hospital will be cared for by multiple physicians over the course of a hospitalization. These care transitions may be seen as barriers to high-quality care and education, but we suggest a different paradigm: transitions of care present opportunities for trainees to be teammates in each other’s calibration. Peers can provide specific feedback about the diagnostic process and inform one another about patient outcomes. Transitions of care allow for built-in “second opinions,” and trainees can intentionally learn by comparing the clinical reasoning involved at different points in a patient’s course. The diagnostic process is dynamic and complex; it is fundamental that trainees have the opportunity to reflect on the process to identify how and why the diagnostic process evolved throughout a patient’s hospitalization. Most inpatient diagnoses are “working diagnoses” that are likely to change. Thus, identifying the twists and turns in a patient’s diagnostic journey provides invaluable learning for future practice.

Herein, we describe the implementation and impact of a multisite initiative to engage residents in delivering feedback to their peers about medical decisions around transitions of care.

The LOOP Project is a prospective clinical educational study that aimed to engage resident physicians to deliver feedback and updates about their colleagues’ diagnostic decision-making around care transitions. This study was deemed exempt from review by the University of Minnesota Institutional Review Board and either approved or deemed exempt by the corresponding Institutional Review Boards at all participating institutions. The study was conducted by seven programs at six institutions and included Internal Medicine, Pediatrics, and Internal Medicine–Pediatrics (PGY 1-4) residents from February 2017 to June 2017. Residents rotating through participating clinical services during the study period were invited to participate and given further information by site leads via informational presentations, written handouts, and/or emails.

The intervention entailed residents delivering structured feedback to their colleagues regarding their patients’ diagnoses after transitions of care. The predominant setting was the inpatient hospital medicine day-shift team providing feedback to the night-shift team regarding overnight admissions. Feedback about patients (usually chosen by the day-shift team) was delivered through completion of a standard templated form (Figure) usually sent within 24 hours after hospital admission through secure messaging (ie, EPIC In-Basket message utilizing a Smartphrase of the LOOP feedback form). A 24-hour time period was chosen to allow for rapid cycling of feedback focusing on initial diagnostic assessment. Site leads and resident champions promoted the project through presentations, informal discussions, and prizes for high completion rates of forms and surveys (ie, coffee cards and pizza).

Overall, there were 716 current residents in the training programs that participated in this study; one site planned on participating but did not complete any forms. A total of 405 residents were eligible to participate during the study period. Overall, 221 (54.5%) presurveys and 90 postsurveys were completed (22.2%); 54 residents (13.3%) completed both pre- and postsurveys and were included in the analysis. Of the 54 survey respondents, 26 (48.15%) were female.

Survey results (Table) indicated significantly improved self-efficacy in identifying cognitive errors in residents’ own practice, identifying why those errors occurred, and identifying strategies to decrease future diagnostic errors. Participants noted increased frequency of discussions within teams regarding differential diagnoses, diagnostic errors, and why diagnoses changed over time. The feedback process was viewed positively by participants, who were also generally satisfied with the overall quality, frequency, and value of the feedback received. After the intervention, participants reported an increase in the amount of feedback received for night admissions and an overall increase in the perception that nighttime admissions were as “educational” as daytime admissions.

This study suggests that an intervention to operationalize standardized, structured feedback about diagnostic decision-making around transitions of care is a promising approach to improve residents’ understanding of changes in, and evolution of, the diagnostic process, as well as improve the perceived educational value of overnight admissions. In our results, over 40% of the patients admitted by residents had some change in their diagnoses after a transition of care during their early hospitalization. This finding highlights the importance of ensuring that trainees have the opportunity to know the outcomes of their decisions. Indeed, residents should be encouraged to follow-up on their own patients without prompting; however, studies show that this practice is uncommon and interventions beyond admonition are necessary.⁴

The diagnostic change rate observed in this study confirms that diagnosis is an iterative process and that the concept of a working diagnosis is key—a diagnosis made at admission will very likely be modified by time, the natural history of the disease, and new clinical information. When diagnoses are viewed as working diagnoses, trainees may be empowered to better understand the diagnostic process. As learners and teachers adopt this perspective, training programs are more likely to be successful in helping learners calibrate toward expertise.

Previous studies have questioned whether resident physicians view overnight admissions as valuable.⁶ After our intervention, we found an increase in both the amount of feedback received and the proportion of participants who agreed that night and day admissions were equally educational, suggesting that targeted diagnostic reasoning feedback can bolster educational value of nighttime admissions.

This study presents a number of limitations. First, the survey response rate was low, which could potentially lead to biased results. We excluded those respondents who did not respond to both the pre- and postsurveys from the analysis. Second, we did not measure actual change in diagnostic performance. While learners did report learning and saw feedback as valuable, self-identified learning points may not always translate to improved patient care. Additionally, residents chose the patients for whom feedback was provided, and the diagnostic change rate described may be overestimated. We did not track the total number of admissions for which feedback could have been delivered during the study. We did not include a control group, and the intervention may not be responsible for changing learners’ perceptions. However, the included programs were not implementing other new protocols focused on diagnostic reasoning during the study period. In addition, we addressed diagnostic changes early in a hospital course; a comprehensive program should address more feedback loops (eg, discharging team to admitting team).

This work is a pilot study; for future interventions focused on improving calibration to be sustainable, they should be congruent with existing clinical workflows and avoid adding to the stress and/or cognitive load of an already-busy clinical experience. The most optimal strategies for delivering feedback about clinical reasoning remain unclear.

In summary, a program to deliver structured feedback among resident physicians about diagnostic reasoning across care transitions for selected hospitalized patients is viewed positively by trainees, is feasible, and leads to changes in resident perception and self-efficacy. Future studies and interventions should aim to provide feedback more systematically, rather than just for selected patients, and objectively track diagnostic changes over time in hospitalized patients. While truly objective diagnostic information is challenging to obtain, comparing admission and other inpatient diagnoses to discharge diagnoses or diagnoses from primary care follow-up visits may be helpful. In addition, studies should aim to track trainees’ clinical decision-making over time and determine the effectiveness of feedback at improving diagnostic performance through calibration.

Acknowledgments

The authors thank the trainees who participated in this study, as well as the residency leadership at participating institutions. The authors also thank Qi Wang, PhD, for providing statistical analysis.

Disclosures

The authors have nothing to disclose.

Funding

The study was funded by an AAIM Innovation Grant and local support at each participating institution.

Unintentional medication discrepancies in the hospital setting are common and contribute to adverse drug events, resulting in patient harm.¹ Discrepancies can be resolved by implementing high-quality medication reconciliation, but there are insufficient data to guide hospitals as to which interventions are most effective at improving medication reconciliation processes and reducing harm.² We recently reported that implementation of a best practices toolkit reduced total medication discrepancies in the Multi-Center Medication Reconciliation Quality Improvement Study (MARQUIS).³ This report describes the effect of individual toolkit components on rates of medication discrepancies with the potential for patient harm.

Detailed descriptions of the intervention toolkit and study design of MARQUIS are published.^4,5 Briefly, MARQUIS was a pragmatic, mentored, quality improvement (QI) study in which five hospitals in the United States implemented interventions from a best practices toolkit to improve medication reconciliation on noncritical care medical and surgical units from September 2011 to July 2014. We used a mentored implementation approach, in which each site identified the leaders of their local quality improvement team (ie, mentees) who received mentorship from a trained physician with QI and medication safety experience.⁶ Mentors conducted monthly calls with their mentees and two site visits. Sites adapted and implemented one or more components from the MARQUIS toolkit, a compilation of evidence-based best practices in medication reconciliation.^5,7

The primary outcome was unintentional medication discrepancies in admission and discharge orders with the potential for causing harm, as previously described.⁴ Trained study pharmacists at each site took “gold standard” medication histories on a random sample of up to 22 patients per month. These medications were then compared with admission and discharge medication orders, and all unintentional discrepancies were identified. The discrepancies were then adjudicated by physicians blinded to the treatment arm, who confirmed whether discrepancies were unintentional and carried the potential for patient harm.

We employed a modification of a stepped wedge methodology to measure the incremental effect of implementing nine different intervention components, introduced at different sites over the course of the study, on the number of potentially harmful discrepancies per patient. These analyses were restricted to the postimplementation period on hospital units that implemented at least one intervention. All interventions conducted at each site were categorized by component, including dates of implementation. Each intervention component could be applied more than once per site (eg, when involving a new group of providers) or implemented on a new hospital unit or service, in which case, all dates were included in the analysis. We conducted a multivariable Poisson regression (with time divided into months) adjusted for patient factors, season, and site, with the number of potentially harmful discrepancies as the dependent variable, and the total number of gold standard medications as a model offset. The model was designed to analyze changes in the y-intercept each time an intervention component was either implemented or spread and assumed the change in the y-intercept was the same for each of these events for any given component. The model also assumes that combinations of interventions had independent additive effects.

Across the five participating sites, 1,648 patients were enrolled from September 2011 to July 2014. This number included 613 patients during the preimplementation period and 1,035 patients during the postimplementation period, of which 791 were on intervention units and comprised the study population. Table 1 displays the intervention components implemented by site. Sites implemented between one and seven components. The most frequently implemented intervention component was training existing staff to take the best possible medication histories (BPMHs), implemented at four sites. The regression results are displayed in Table 2. Three interventions were associated with significant decreases in potentially harmful discrepancy rates: (1) clearly defining roles and responsibilities and communicating this with clinical staff (hazard ratio [HR] 0.53, 95% CI: 0.32–0.87); (2) training existing staff to perform discharge medication reconciliation and patient counseling (HR 0.64, 95% CI: 0.46–0.89); and (3) hiring additional staff to perform discharge medication reconciliation and patient counseling (HR 0.48, 95% CI: 0.31–0.77). Two interventions were associated with significant increases in potentially harmful discrepancy rates: training existing staff to take BPMHs (HR 1.38, 95% CI: 1.21–1.57) and implementing a new electronic health record (EHR; HR 2.21, 95% CI: 1.64–2.97).

We noted that three intervention components were associated with decreased rates of unintentional medication discrepancies with potential for harm, whereas two were associated with increased rates. The components with a beneficial effect were not surprising. A prior qualitative study demonstrated the confusion related to clinicians’ roles and responsibilities during medication reconciliation; therefore, clear delineations should reduce rework and improve the medication reconciliation process.⁸ Other studies have shown the benefits of pharmacist involvement in the inpatient setting, particularly in reducing errors at discharge.⁹ However, we did not anticipate that training staff to take BPMHs would be detrimental. Possible reasons for this finding that are based on direct observations by mentors at site visits or noted during monthly calls include (1) training personnel on this task without certification of competency may not sufficiently improve their skills, leading instead to diffusion of responsibility; (2) training personnel without sufficient time to perform the task well (eg, frontline nurses with many other responsibilities) may be counterproductive compared with training a few personnel with time dedicated to this task; and (3) training existing personnel in history-taking may have been used to delay the necessary hiring of more staff to take BPMHs. Future studies could address several of these shortcomings in both the design and implementation of medication history-training intervention components.

Several reasons may explain the association we found between implementing a new EHR and increased rates of discrepancies. Based on mentors’ experiences, we suspect it is because sitewide EHR implementation requires significant resources, time, and effort. Therefore, sitewide EHR implementation pulls attention away from a focus on medication safety. Most large vendor EHRs have design flaws in their medication reconciliation modules, with the overarching problem being that their systems are not designed for an interdisciplinary team approach to medication reconciliation (unpublished material). In addition, problems may also exist with the local implementation of these modules and the way they are used by clinicians (eg, bypassing critical steps in the medication reconciliation process that lead to new medication errors). We have updated the MARQUIS toolkit to include pros and cons of EHR software and ideal features and functions of medication reconciliation information technology. We should note that this finding contrasts with previous studies that showed beneficial effects of dedicated medication reconciliation applications, which used proprietary technology, often combined with process redesign, in a focused QI effort.^10-13 These findings suggest the need for improvements in the design, local customization, and use of medication reconciliation modules in vendor EHRs.

Our study has several limitations. We conducted an on-treatment analysis, which may be confounded by characteristics of sites that chose to implement different intervention components; however, we adjusted for sites in the analysis. Some results are based on a limited number of sites implementing an intervention component (eg, defining roles and responsibilities). Although this was a longitudinal study, and we adjusted for seasonal effects, it is possible that temporal trends and cointerventions confounded our results. The adjudication of discrepancies for the potential for harm was somewhat subjective, although we used a rigorous process to ensure the reliability of adjudication, as in prior studies.^3,14 As in the main analysis of the MARQUIS study, this analysis did not measure intervention fidelity.

Based on these analyses and the literature base, we recommend that hospitals focus first on hiring and training dedicated staff (usually pharmacists) to assist with medication reconciliation at discharge.⁷ Hospitals should also be aware of potential increases in medication discrepancies when implementing a large vendor EHR across their institution. Further work is needed on the best ways to mitigate these adverse effects, at both the design and local site levels. Finally, the effect of medication history training on discrepancies warrants further study.

Disclosures

SK has served as a consultant to Verustat, a remote health monitoring company. All other authors have no disclosures or conflicts of interests.

Funding

This study was supported by the Agency for Healthcare Research and Quality (grant number: R18 HS019598). JLS has received funding from (1) Mallinckrodt Pharmaceuticals for an investigator-initiated study of opioid-related adverse drug events in postsurgical patients; (2) Horizon Blue Cross Blue Shield for an honorarium and travel expenses for workshop on medication reconciliation; (3) Island Peer Review Organization for honorarium and travel expenses for workshop on medication reconciliation; and, (4) Portola Pharmaceuticals for investigator-initiated study of inpatients who decline subcutaneous medications for venous thromboembolism prophylaxis. ASM was funded by a VA HSR&D Career Development Award (12-168).

Trial Registration

ClinicalTrials.gov NCT01337063

Unintentional medication discrepancies in the hospital setting are common and contribute to adverse drug events, resulting in patient harm.¹ Discrepancies can be resolved by implementing high-quality medication reconciliation, but there are insufficient data to guide hospitals as to which interventions are most effective at improving medication reconciliation processes and reducing harm.² We recently reported that implementation of a best practices toolkit reduced total medication discrepancies in the Multi-Center Medication Reconciliation Quality Improvement Study (MARQUIS).³ This report describes the effect of individual toolkit components on rates of medication discrepancies with the potential for patient harm.

Detailed descriptions of the intervention toolkit and study design of MARQUIS are published.^4,5 Briefly, MARQUIS was a pragmatic, mentored, quality improvement (QI) study in which five hospitals in the United States implemented interventions from a best practices toolkit to improve medication reconciliation on noncritical care medical and surgical units from September 2011 to July 2014. We used a mentored implementation approach, in which each site identified the leaders of their local quality improvement team (ie, mentees) who received mentorship from a trained physician with QI and medication safety experience.⁶ Mentors conducted monthly calls with their mentees and two site visits. Sites adapted and implemented one or more components from the MARQUIS toolkit, a compilation of evidence-based best practices in medication reconciliation.^5,7

The primary outcome was unintentional medication discrepancies in admission and discharge orders with the potential for causing harm, as previously described.⁴ Trained study pharmacists at each site took “gold standard” medication histories on a random sample of up to 22 patients per month. These medications were then compared with admission and discharge medication orders, and all unintentional discrepancies were identified. The discrepancies were then adjudicated by physicians blinded to the treatment arm, who confirmed whether discrepancies were unintentional and carried the potential for patient harm.

We employed a modification of a stepped wedge methodology to measure the incremental effect of implementing nine different intervention components, introduced at different sites over the course of the study, on the number of potentially harmful discrepancies per patient. These analyses were restricted to the postimplementation period on hospital units that implemented at least one intervention. All interventions conducted at each site were categorized by component, including dates of implementation. Each intervention component could be applied more than once per site (eg, when involving a new group of providers) or implemented on a new hospital unit or service, in which case, all dates were included in the analysis. We conducted a multivariable Poisson regression (with time divided into months) adjusted for patient factors, season, and site, with the number of potentially harmful discrepancies as the dependent variable, and the total number of gold standard medications as a model offset. The model was designed to analyze changes in the y-intercept each time an intervention component was either implemented or spread and assumed the change in the y-intercept was the same for each of these events for any given component. The model also assumes that combinations of interventions had independent additive effects.

Across the five participating sites, 1,648 patients were enrolled from September 2011 to July 2014. This number included 613 patients during the preimplementation period and 1,035 patients during the postimplementation period, of which 791 were on intervention units and comprised the study population. Table 1 displays the intervention components implemented by site. Sites implemented between one and seven components. The most frequently implemented intervention component was training existing staff to take the best possible medication histories (BPMHs), implemented at four sites. The regression results are displayed in Table 2. Three interventions were associated with significant decreases in potentially harmful discrepancy rates: (1) clearly defining roles and responsibilities and communicating this with clinical staff (hazard ratio [HR] 0.53, 95% CI: 0.32–0.87); (2) training existing staff to perform discharge medication reconciliation and patient counseling (HR 0.64, 95% CI: 0.46–0.89); and (3) hiring additional staff to perform discharge medication reconciliation and patient counseling (HR 0.48, 95% CI: 0.31–0.77). Two interventions were associated with significant increases in potentially harmful discrepancy rates: training existing staff to take BPMHs (HR 1.38, 95% CI: 1.21–1.57) and implementing a new electronic health record (EHR; HR 2.21, 95% CI: 1.64–2.97).

We noted that three intervention components were associated with decreased rates of unintentional medication discrepancies with potential for harm, whereas two were associated with increased rates. The components with a beneficial effect were not surprising. A prior qualitative study demonstrated the confusion related to clinicians’ roles and responsibilities during medication reconciliation; therefore, clear delineations should reduce rework and improve the medication reconciliation process.⁸ Other studies have shown the benefits of pharmacist involvement in the inpatient setting, particularly in reducing errors at discharge.⁹ However, we did not anticipate that training staff to take BPMHs would be detrimental. Possible reasons for this finding that are based on direct observations by mentors at site visits or noted during monthly calls include (1) training personnel on this task without certification of competency may not sufficiently improve their skills, leading instead to diffusion of responsibility; (2) training personnel without sufficient time to perform the task well (eg, frontline nurses with many other responsibilities) may be counterproductive compared with training a few personnel with time dedicated to this task; and (3) training existing personnel in history-taking may have been used to delay the necessary hiring of more staff to take BPMHs. Future studies could address several of these shortcomings in both the design and implementation of medication history-training intervention components.

Several reasons may explain the association we found between implementing a new EHR and increased rates of discrepancies. Based on mentors’ experiences, we suspect it is because sitewide EHR implementation requires significant resources, time, and effort. Therefore, sitewide EHR implementation pulls attention away from a focus on medication safety. Most large vendor EHRs have design flaws in their medication reconciliation modules, with the overarching problem being that their systems are not designed for an interdisciplinary team approach to medication reconciliation (unpublished material). In addition, problems may also exist with the local implementation of these modules and the way they are used by clinicians (eg, bypassing critical steps in the medication reconciliation process that lead to new medication errors). We have updated the MARQUIS toolkit to include pros and cons of EHR software and ideal features and functions of medication reconciliation information technology. We should note that this finding contrasts with previous studies that showed beneficial effects of dedicated medication reconciliation applications, which used proprietary technology, often combined with process redesign, in a focused QI effort.^10-13 These findings suggest the need for improvements in the design, local customization, and use of medication reconciliation modules in vendor EHRs.

Our study has several limitations. We conducted an on-treatment analysis, which may be confounded by characteristics of sites that chose to implement different intervention components; however, we adjusted for sites in the analysis. Some results are based on a limited number of sites implementing an intervention component (eg, defining roles and responsibilities). Although this was a longitudinal study, and we adjusted for seasonal effects, it is possible that temporal trends and cointerventions confounded our results. The adjudication of discrepancies for the potential for harm was somewhat subjective, although we used a rigorous process to ensure the reliability of adjudication, as in prior studies.^3,14 As in the main analysis of the MARQUIS study, this analysis did not measure intervention fidelity.

Based on these analyses and the literature base, we recommend that hospitals focus first on hiring and training dedicated staff (usually pharmacists) to assist with medication reconciliation at discharge.⁷ Hospitals should also be aware of potential increases in medication discrepancies when implementing a large vendor EHR across their institution. Further work is needed on the best ways to mitigate these adverse effects, at both the design and local site levels. Finally, the effect of medication history training on discrepancies warrants further study.

Disclosures

SK has served as a consultant to Verustat, a remote health monitoring company. All other authors have no disclosures or conflicts of interests.

Funding

This study was supported by the Agency for Healthcare Research and Quality (grant number: R18 HS019598). JLS has received funding from (1) Mallinckrodt Pharmaceuticals for an investigator-initiated study of opioid-related adverse drug events in postsurgical patients; (2) Horizon Blue Cross Blue Shield for an honorarium and travel expenses for workshop on medication reconciliation; (3) Island Peer Review Organization for honorarium and travel expenses for workshop on medication reconciliation; and, (4) Portola Pharmaceuticals for investigator-initiated study of inpatients who decline subcutaneous medications for venous thromboembolism prophylaxis. ASM was funded by a VA HSR&D Career Development Award (12-168).

Trial Registration

ClinicalTrials.gov NCT01337063

Unintentional medication discrepancies in the hospital setting are common and contribute to adverse drug events, resulting in patient harm.¹ Discrepancies can be resolved by implementing high-quality medication reconciliation, but there are insufficient data to guide hospitals as to which interventions are most effective at improving medication reconciliation processes and reducing harm.² We recently reported that implementation of a best practices toolkit reduced total medication discrepancies in the Multi-Center Medication Reconciliation Quality Improvement Study (MARQUIS).³ This report describes the effect of individual toolkit components on rates of medication discrepancies with the potential for patient harm.

Detailed descriptions of the intervention toolkit and study design of MARQUIS are published.^4,5 Briefly, MARQUIS was a pragmatic, mentored, quality improvement (QI) study in which five hospitals in the United States implemented interventions from a best practices toolkit to improve medication reconciliation on noncritical care medical and surgical units from September 2011 to July 2014. We used a mentored implementation approach, in which each site identified the leaders of their local quality improvement team (ie, mentees) who received mentorship from a trained physician with QI and medication safety experience.⁶ Mentors conducted monthly calls with their mentees and two site visits. Sites adapted and implemented one or more components from the MARQUIS toolkit, a compilation of evidence-based best practices in medication reconciliation.^5,7

The primary outcome was unintentional medication discrepancies in admission and discharge orders with the potential for causing harm, as previously described.⁴ Trained study pharmacists at each site took “gold standard” medication histories on a random sample of up to 22 patients per month. These medications were then compared with admission and discharge medication orders, and all unintentional discrepancies were identified. The discrepancies were then adjudicated by physicians blinded to the treatment arm, who confirmed whether discrepancies were unintentional and carried the potential for patient harm.

We employed a modification of a stepped wedge methodology to measure the incremental effect of implementing nine different intervention components, introduced at different sites over the course of the study, on the number of potentially harmful discrepancies per patient. These analyses were restricted to the postimplementation period on hospital units that implemented at least one intervention. All interventions conducted at each site were categorized by component, including dates of implementation. Each intervention component could be applied more than once per site (eg, when involving a new group of providers) or implemented on a new hospital unit or service, in which case, all dates were included in the analysis. We conducted a multivariable Poisson regression (with time divided into months) adjusted for patient factors, season, and site, with the number of potentially harmful discrepancies as the dependent variable, and the total number of gold standard medications as a model offset. The model was designed to analyze changes in the y-intercept each time an intervention component was either implemented or spread and assumed the change in the y-intercept was the same for each of these events for any given component. The model also assumes that combinations of interventions had independent additive effects.

Across the five participating sites, 1,648 patients were enrolled from September 2011 to July 2014. This number included 613 patients during the preimplementation period and 1,035 patients during the postimplementation period, of which 791 were on intervention units and comprised the study population. Table 1 displays the intervention components implemented by site. Sites implemented between one and seven components. The most frequently implemented intervention component was training existing staff to take the best possible medication histories (BPMHs), implemented at four sites. The regression results are displayed in Table 2. Three interventions were associated with significant decreases in potentially harmful discrepancy rates: (1) clearly defining roles and responsibilities and communicating this with clinical staff (hazard ratio [HR] 0.53, 95% CI: 0.32–0.87); (2) training existing staff to perform discharge medication reconciliation and patient counseling (HR 0.64, 95% CI: 0.46–0.89); and (3) hiring additional staff to perform discharge medication reconciliation and patient counseling (HR 0.48, 95% CI: 0.31–0.77). Two interventions were associated with significant increases in potentially harmful discrepancy rates: training existing staff to take BPMHs (HR 1.38, 95% CI: 1.21–1.57) and implementing a new electronic health record (EHR; HR 2.21, 95% CI: 1.64–2.97).

We noted that three intervention components were associated with decreased rates of unintentional medication discrepancies with potential for harm, whereas two were associated with increased rates. The components with a beneficial effect were not surprising. A prior qualitative study demonstrated the confusion related to clinicians’ roles and responsibilities during medication reconciliation; therefore, clear delineations should reduce rework and improve the medication reconciliation process.⁸ Other studies have shown the benefits of pharmacist involvement in the inpatient setting, particularly in reducing errors at discharge.⁹ However, we did not anticipate that training staff to take BPMHs would be detrimental. Possible reasons for this finding that are based on direct observations by mentors at site visits or noted during monthly calls include (1) training personnel on this task without certification of competency may not sufficiently improve their skills, leading instead to diffusion of responsibility; (2) training personnel without sufficient time to perform the task well (eg, frontline nurses with many other responsibilities) may be counterproductive compared with training a few personnel with time dedicated to this task; and (3) training existing personnel in history-taking may have been used to delay the necessary hiring of more staff to take BPMHs. Future studies could address several of these shortcomings in both the design and implementation of medication history-training intervention components.

Several reasons may explain the association we found between implementing a new EHR and increased rates of discrepancies. Based on mentors’ experiences, we suspect it is because sitewide EHR implementation requires significant resources, time, and effort. Therefore, sitewide EHR implementation pulls attention away from a focus on medication safety. Most large vendor EHRs have design flaws in their medication reconciliation modules, with the overarching problem being that their systems are not designed for an interdisciplinary team approach to medication reconciliation (unpublished material). In addition, problems may also exist with the local implementation of these modules and the way they are used by clinicians (eg, bypassing critical steps in the medication reconciliation process that lead to new medication errors). We have updated the MARQUIS toolkit to include pros and cons of EHR software and ideal features and functions of medication reconciliation information technology. We should note that this finding contrasts with previous studies that showed beneficial effects of dedicated medication reconciliation applications, which used proprietary technology, often combined with process redesign, in a focused QI effort.^10-13 These findings suggest the need for improvements in the design, local customization, and use of medication reconciliation modules in vendor EHRs.

Our study has several limitations. We conducted an on-treatment analysis, which may be confounded by characteristics of sites that chose to implement different intervention components; however, we adjusted for sites in the analysis. Some results are based on a limited number of sites implementing an intervention component (eg, defining roles and responsibilities). Although this was a longitudinal study, and we adjusted for seasonal effects, it is possible that temporal trends and cointerventions confounded our results. The adjudication of discrepancies for the potential for harm was somewhat subjective, although we used a rigorous process to ensure the reliability of adjudication, as in prior studies.^3,14 As in the main analysis of the MARQUIS study, this analysis did not measure intervention fidelity.

Based on these analyses and the literature base, we recommend that hospitals focus first on hiring and training dedicated staff (usually pharmacists) to assist with medication reconciliation at discharge.⁷ Hospitals should also be aware of potential increases in medication discrepancies when implementing a large vendor EHR across their institution. Further work is needed on the best ways to mitigate these adverse effects, at both the design and local site levels. Finally, the effect of medication history training on discrepancies warrants further study.

Disclosures

SK has served as a consultant to Verustat, a remote health monitoring company. All other authors have no disclosures or conflicts of interests.

Funding

This study was supported by the Agency for Healthcare Research and Quality (grant number: R18 HS019598). JLS has received funding from (1) Mallinckrodt Pharmaceuticals for an investigator-initiated study of opioid-related adverse drug events in postsurgical patients; (2) Horizon Blue Cross Blue Shield for an honorarium and travel expenses for workshop on medication reconciliation; (3) Island Peer Review Organization for honorarium and travel expenses for workshop on medication reconciliation; and, (4) Portola Pharmaceuticals for investigator-initiated study of inpatients who decline subcutaneous medications for venous thromboembolism prophylaxis. ASM was funded by a VA HSR&D Career Development Award (12-168).

Trial Registration

ClinicalTrials.gov NCT01337063

N o matter the era, few aspects of residency are more defining or memorable than overnight call. Nights can be a time of growth and learning but also of fear and uncertainty, as residents take on the responsibility of managing sick patients on their own. One of us (ASD) started his residency in 1978 at the Massachusetts General Hospital in Boston; the other two (ST and BCY) started theirs in 2016 and 2017, respectively, at the University of Toronto. In this essay, we reflect on our experiences of night call separated by 40 years, highlighting what has changed and what has stayed the same.

At 6 pm, a calm overtook the hospital as the daytime staff emptied out of the building. The only remaining residents were those on call for the night. Each team had one resident and one medical student responsible for about 20 inpatients. Teams also admitted new patients from the emergency department (ED). Since there were three teams who rotated the admissions, we were able to spend a part of our night on the inpatient ward. Most of the new patients had been thoroughly evaluated by a medical resident in the ED before we saw them, giving us a head start on eliciting their stories, performing physical examinations, reviewing the laboratory workup, and generating plans to present to the attending physician (who was called “the visit”) the next morning at 10 am .

We carried one pager that was about 7 inches long and 2 inches wide clipped to the waist of our pants. It could only make a beep; we then had to call the page operator to find out who wanted us. However, the pages were relatively few. Nurses called only when a patient was unstable, and other residents called only when a new patient was ready in the emergency department. At 9 pm , the laboratory data for the day were delivered to the ward nursing stations on computer-generated paper. Our job was to separate the pages connected by serrated breaks, review the results, and then file the pages in the charts. The nurses were aware of this routine, so they saved their questions for our presence on the wards, reducing the need to page us. At 10 pm , residents from all services went to the cafeteria for the “10 o’clock meal” when the food was free.We learned early in the year to drop everything at 10 pm ; otherwise, we did not eat. The social aspect of the group meal was comforting, but compared with today, the group was much more homogeneous and therefore less interesting. There were several women medical residents (although almost none in surgery), but very few minorities, and no openly gay residents.

Gathering data about patients prior to the current hospitalization required reviewing the “old chart,” which had to be delivered from patient records but was generally available when the patient was still in the ED. It contained typed discharge summaries and progress notes often handwritten by coresidents whom we knew. The handwriting was often difficult to read, outpatient notes were not included, and information from other hospitals was absent—but despite these deficiencies, we somehow managed just fine.

The patients on the inpatient ward were mostly stable, but more importantly, we had very few medications and tests to order. I recall prescribing fewer than 20 drugs—furosemide, hydrochlorothiazide, penicillins, cephalosporins, gentamicin, isoniazid, lidocaine, nitroglycerin, aminophylline, alpha-methyldopa, clonidine, propranolol, digoxin, hydralazine, indomethacin, steroids, and morphine. Orders for tests and imaging had to be physically written in the chart and could not be inputted remotely, which was a nuisance when we were away from the ward. However, we rarely ordered any imaging beyond plain radiographs at night. We did draw arterial blood gases and venous blood, administer oxygen, insert intravenous and central lines, take electrocardiograms, and perform urinalyses by microscopy. We did all these tasks ourselves for patients on the “ward service” (as opposed to the “private service”, which had to do with the type of insurance the patients possessed). As a result, we became experts in both blood drawing and intravenous line insertion—skills that might be less familiar to today’s residents.

Of course, patients did get acutely ill during the night. I recall intubating, cardioverting, performing phlebotomy to alleviate pulmonary edema, sending patients to surgery, and pronouncing death. Nevertheless, we often got sleep, and sometimes, several hours in a row. I had a rule; I always took a shower the next morning and put on clean clothes (we stayed until 5 pm , making the shifts 33.5 hours long). There was a camaraderie that existed between all of us at night. We were supportive, friendly, and knew each other by name, and more.

We were often frightened by the responsibility of managing sick patients alone. On particularly challenging nights, we would record our fears and feelings in a “night call diary” in one of the conference rooms—generally at 4 am . Some entries became legendary as people read and reread those months and years later. Mornings always brought a sense of relief and accomplishment, because when the sun came up, we knew that the other residents would not be far behind; when they arrived, we could tell our stories and get help.

There was definitely competitiveness to the work. Those who responded quickly to deteriorating patients were applauded; those who did not really know what to do were subtly disdained. However, over time, we all got the hang of it, and this led to a growing confidence that we were indeed doctors. The graded autonomy afforded by night call was a crucial part of that journey.

At 6 pm , the on-call residents assemble in the ED, where we would spend the remainder of the night and early morning admitting new patients to the hospital. A night team consists of a senior resident, three junior residents, and two medical students, with each resident being responsible for approximately 20 inpatients. Overnight coverage of the ward mostly occurs remotely; since the ED is often so busy, we address most of the issues through a computer or over the phone. Only in rare cases, such as when a patient is unstable or a death has occurred, do we deal with the matter in person.

To enable rapid remote responses, we each carry an assortment of devices on our waists or lanyards and in our pockets, such as a personal pager, ED consult pager, code blue pager, and hospital-issued smartphones capable of receiving pages, text messages, phone calls, and e-mails. Nurses, pharmacists, and other consultants communicate with us through all of these channels. Few of these interactions occur face-to-face. To our frustration, encounters with patients are frequently interrupted by a stream of beeps, rings, and vibrations—irrespective of whether we are having a difficult discussion about goals of care or performing a delicate procedure.

The ED contains a work space dedicated for residents to enter electronic orders, type notes, and review new admissions. Between consults, we try to discuss exciting cases and provide teaching to the medical students and interns, which we enjoy. Dinner is generally devoured while inputting orders. In exceptional circumstances, a brief reprieve from pages may allow the on-call team to share a meal. Depending on our role, sleep may be possible on certain nights but is never guaranteed. Moments spent with the on-call team—all of us learning, commiserating, and growing together—are some of the most memorable of residency, and many of us become close friends by the end of the rotation.

However, apart from these few familiar faces, we rarely get acquainted with the nurses or residents from other services. Many often refer to themselves by specialty rather than name and phone calls that begin with “Are you Medicine?” can end with “You should really call Orthopedics.” Meanwhile, “Medicine” and “Orthopedics” may pass each other in the hallway without recognition beyond a vague familiarity of a voice heard on the phone.

Every 10 minutes spent with a new patient is accompanied by approximately one hour of “electronic” time, which includes reading through previous medical records, reviewing laboratory data and imaging, and creating an admission note. Interns might groan as they pull up a patient’s electronic health record (EHR); irrelevant details often arise with each click of the mouse, and the cursed “copy-paste” function means that new notes often duplicate older ones. However, with time, we learn to look past the EHR’s shortcomings and appreciate several of its advantages. For example, we are now able to access test results performed outside our hospital and thus limit our repetition of investigations. We can also use the EHR to rapidly glean salient information about a patient in time-critical scenarios. This is always a satisfying process, and it makes us wonder how physicians ever practiced in the era before computers.

Today’s patients are older and sicker than ever before. Many are receiving treatments that did not exist even a decade ago. As residents, we must recognize a seemingly endless variety of drugs—a challenging but intellectually satisfying responsibility. We must also decide whether the patient’s current health state permits their continuation, or whether safer alternatives exist. Some of these decisions cannot wait until the morning.

During handover at 8 am , we often recount moments from our call shift with a sense of vigor that is only partly dulled by fatigue. We may share with pride our management of a sick patient. We may relay a touching exchange with a concerned family member. Or we may recall with satisfaction our handling of a tense situation with a colleague. Taking part in these experiences is one of the most character-building aspects of being on call. The absence of our supervisors can be unnerving at first, but we gradually begin to enjoy the sense of independence. Moreover, we feel empowered to make our patients better with the right combination of carefully selected treatments. Nothing makes us feel more like doctors.

No doubt, being on call is difficult. The next day brings a feeling of relief and accomplishment, knowing that we got through it—whether by floundering or flourishing—in one piece.

The two passages described here are personal descriptions of a typical night on-call in two different eras. Readers around the world may have a very different recollection of their own experience. Nevertheless, several aspects of being on call remain constant, such as anxiety about caring for sick patients alone, fond recollections of friends made, and relief when the morning comes. Most important, however, might be the tremendous satisfaction at the opportunity to learn and grow—to become a competent physician by testing one’s physical and intellectual limits through graded autonomy

On the other hand, certain elements of night call have undeniably changed—partly a consequence of the increased number of people involved in patient care and changing communication technology. Residents today encounter a greater number of interruptions to their work flow. Tasks that require long, continuous periods of full attention are now punctuated by texts, e-mails, calls, and pages. The EHR is often clumsy to navigate, but it can also be a veritable mine of information. Finally, although residents from the same specialty may be close friends, duty hour restrictions and remote asynchronous communication may reduce familiarity with residents from other programs.

Do these descriptions resonate with your experience of night call? Keeping in mind that the 1979 vignette is described through the rose-colored lens of nostalgia, both eras have their advantages and disadvantages. We leave it to the reader to decide what has changed (plus ça change) and what has stayed the same (plus c’est la même chose).

Acknowledgments

The authors thank Micheal A. Fifer, MD (Massachusetts General Hospital), and Timothy J. Judson, MD (UCSF), for their comments on an earlier draft of this essay.

Disclosures

The authors have nothing to disclose.

N o matter the era, few aspects of residency are more defining or memorable than overnight call. Nights can be a time of growth and learning but also of fear and uncertainty, as residents take on the responsibility of managing sick patients on their own. One of us (ASD) started his residency in 1978 at the Massachusetts General Hospital in Boston; the other two (ST and BCY) started theirs in 2016 and 2017, respectively, at the University of Toronto. In this essay, we reflect on our experiences of night call separated by 40 years, highlighting what has changed and what has stayed the same.

At 6 pm, a calm overtook the hospital as the daytime staff emptied out of the building. The only remaining residents were those on call for the night. Each team had one resident and one medical student responsible for about 20 inpatients. Teams also admitted new patients from the emergency department (ED). Since there were three teams who rotated the admissions, we were able to spend a part of our night on the inpatient ward. Most of the new patients had been thoroughly evaluated by a medical resident in the ED before we saw them, giving us a head start on eliciting their stories, performing physical examinations, reviewing the laboratory workup, and generating plans to present to the attending physician (who was called “the visit”) the next morning at 10 am .

We carried one pager that was about 7 inches long and 2 inches wide clipped to the waist of our pants. It could only make a beep; we then had to call the page operator to find out who wanted us. However, the pages were relatively few. Nurses called only when a patient was unstable, and other residents called only when a new patient was ready in the emergency department. At 9 pm , the laboratory data for the day were delivered to the ward nursing stations on computer-generated paper. Our job was to separate the pages connected by serrated breaks, review the results, and then file the pages in the charts. The nurses were aware of this routine, so they saved their questions for our presence on the wards, reducing the need to page us. At 10 pm , residents from all services went to the cafeteria for the “10 o’clock meal” when the food was free.We learned early in the year to drop everything at 10 pm ; otherwise, we did not eat. The social aspect of the group meal was comforting, but compared with today, the group was much more homogeneous and therefore less interesting. There were several women medical residents (although almost none in surgery), but very few minorities, and no openly gay residents.

Gathering data about patients prior to the current hospitalization required reviewing the “old chart,” which had to be delivered from patient records but was generally available when the patient was still in the ED. It contained typed discharge summaries and progress notes often handwritten by coresidents whom we knew. The handwriting was often difficult to read, outpatient notes were not included, and information from other hospitals was absent—but despite these deficiencies, we somehow managed just fine.

The patients on the inpatient ward were mostly stable, but more importantly, we had very few medications and tests to order. I recall prescribing fewer than 20 drugs—furosemide, hydrochlorothiazide, penicillins, cephalosporins, gentamicin, isoniazid, lidocaine, nitroglycerin, aminophylline, alpha-methyldopa, clonidine, propranolol, digoxin, hydralazine, indomethacin, steroids, and morphine. Orders for tests and imaging had to be physically written in the chart and could not be inputted remotely, which was a nuisance when we were away from the ward. However, we rarely ordered any imaging beyond plain radiographs at night. We did draw arterial blood gases and venous blood, administer oxygen, insert intravenous and central lines, take electrocardiograms, and perform urinalyses by microscopy. We did all these tasks ourselves for patients on the “ward service” (as opposed to the “private service”, which had to do with the type of insurance the patients possessed). As a result, we became experts in both blood drawing and intravenous line insertion—skills that might be less familiar to today’s residents.

Of course, patients did get acutely ill during the night. I recall intubating, cardioverting, performing phlebotomy to alleviate pulmonary edema, sending patients to surgery, and pronouncing death. Nevertheless, we often got sleep, and sometimes, several hours in a row. I had a rule; I always took a shower the next morning and put on clean clothes (we stayed until 5 pm , making the shifts 33.5 hours long). There was a camaraderie that existed between all of us at night. We were supportive, friendly, and knew each other by name, and more.

We were often frightened by the responsibility of managing sick patients alone. On particularly challenging nights, we would record our fears and feelings in a “night call diary” in one of the conference rooms—generally at 4 am . Some entries became legendary as people read and reread those months and years later. Mornings always brought a sense of relief and accomplishment, because when the sun came up, we knew that the other residents would not be far behind; when they arrived, we could tell our stories and get help.

There was definitely competitiveness to the work. Those who responded quickly to deteriorating patients were applauded; those who did not really know what to do were subtly disdained. However, over time, we all got the hang of it, and this led to a growing confidence that we were indeed doctors. The graded autonomy afforded by night call was a crucial part of that journey.

At 6 pm , the on-call residents assemble in the ED, where we would spend the remainder of the night and early morning admitting new patients to the hospital. A night team consists of a senior resident, three junior residents, and two medical students, with each resident being responsible for approximately 20 inpatients. Overnight coverage of the ward mostly occurs remotely; since the ED is often so busy, we address most of the issues through a computer or over the phone. Only in rare cases, such as when a patient is unstable or a death has occurred, do we deal with the matter in person.

To enable rapid remote responses, we each carry an assortment of devices on our waists or lanyards and in our pockets, such as a personal pager, ED consult pager, code blue pager, and hospital-issued smartphones capable of receiving pages, text messages, phone calls, and e-mails. Nurses, pharmacists, and other consultants communicate with us through all of these channels. Few of these interactions occur face-to-face. To our frustration, encounters with patients are frequently interrupted by a stream of beeps, rings, and vibrations—irrespective of whether we are having a difficult discussion about goals of care or performing a delicate procedure.

The ED contains a work space dedicated for residents to enter electronic orders, type notes, and review new admissions. Between consults, we try to discuss exciting cases and provide teaching to the medical students and interns, which we enjoy. Dinner is generally devoured while inputting orders. In exceptional circumstances, a brief reprieve from pages may allow the on-call team to share a meal. Depending on our role, sleep may be possible on certain nights but is never guaranteed. Moments spent with the on-call team—all of us learning, commiserating, and growing together—are some of the most memorable of residency, and many of us become close friends by the end of the rotation.

However, apart from these few familiar faces, we rarely get acquainted with the nurses or residents from other services. Many often refer to themselves by specialty rather than name and phone calls that begin with “Are you Medicine?” can end with “You should really call Orthopedics.” Meanwhile, “Medicine” and “Orthopedics” may pass each other in the hallway without recognition beyond a vague familiarity of a voice heard on the phone.

Every 10 minutes spent with a new patient is accompanied by approximately one hour of “electronic” time, which includes reading through previous medical records, reviewing laboratory data and imaging, and creating an admission note. Interns might groan as they pull up a patient’s electronic health record (EHR); irrelevant details often arise with each click of the mouse, and the cursed “copy-paste” function means that new notes often duplicate older ones. However, with time, we learn to look past the EHR’s shortcomings and appreciate several of its advantages. For example, we are now able to access test results performed outside our hospital and thus limit our repetition of investigations. We can also use the EHR to rapidly glean salient information about a patient in time-critical scenarios. This is always a satisfying process, and it makes us wonder how physicians ever practiced in the era before computers.

Today’s patients are older and sicker than ever before. Many are receiving treatments that did not exist even a decade ago. As residents, we must recognize a seemingly endless variety of drugs—a challenging but intellectually satisfying responsibility. We must also decide whether the patient’s current health state permits their continuation, or whether safer alternatives exist. Some of these decisions cannot wait until the morning.

During handover at 8 am , we often recount moments from our call shift with a sense of vigor that is only partly dulled by fatigue. We may share with pride our management of a sick patient. We may relay a touching exchange with a concerned family member. Or we may recall with satisfaction our handling of a tense situation with a colleague. Taking part in these experiences is one of the most character-building aspects of being on call. The absence of our supervisors can be unnerving at first, but we gradually begin to enjoy the sense of independence. Moreover, we feel empowered to make our patients better with the right combination of carefully selected treatments. Nothing makes us feel more like doctors.

No doubt, being on call is difficult. The next day brings a feeling of relief and accomplishment, knowing that we got through it—whether by floundering or flourishing—in one piece.

The two passages described here are personal descriptions of a typical night on-call in two different eras. Readers around the world may have a very different recollection of their own experience. Nevertheless, several aspects of being on call remain constant, such as anxiety about caring for sick patients alone, fond recollections of friends made, and relief when the morning comes. Most important, however, might be the tremendous satisfaction at the opportunity to learn and grow—to become a competent physician by testing one’s physical and intellectual limits through graded autonomy

On the other hand, certain elements of night call have undeniably changed—partly a consequence of the increased number of people involved in patient care and changing communication technology. Residents today encounter a greater number of interruptions to their work flow. Tasks that require long, continuous periods of full attention are now punctuated by texts, e-mails, calls, and pages. The EHR is often clumsy to navigate, but it can also be a veritable mine of information. Finally, although residents from the same specialty may be close friends, duty hour restrictions and remote asynchronous communication may reduce familiarity with residents from other programs.

Do these descriptions resonate with your experience of night call? Keeping in mind that the 1979 vignette is described through the rose-colored lens of nostalgia, both eras have their advantages and disadvantages. We leave it to the reader to decide what has changed (plus ça change) and what has stayed the same (plus c’est la même chose).

Acknowledgments

The authors thank Micheal A. Fifer, MD (Massachusetts General Hospital), and Timothy J. Judson, MD (UCSF), for their comments on an earlier draft of this essay.

Disclosures

The authors have nothing to disclose.

N o matter the era, few aspects of residency are more defining or memorable than overnight call. Nights can be a time of growth and learning but also of fear and uncertainty, as residents take on the responsibility of managing sick patients on their own. One of us (ASD) started his residency in 1978 at the Massachusetts General Hospital in Boston; the other two (ST and BCY) started theirs in 2016 and 2017, respectively, at the University of Toronto. In this essay, we reflect on our experiences of night call separated by 40 years, highlighting what has changed and what has stayed the same.

At 6 pm, a calm overtook the hospital as the daytime staff emptied out of the building. The only remaining residents were those on call for the night. Each team had one resident and one medical student responsible for about 20 inpatients. Teams also admitted new patients from the emergency department (ED). Since there were three teams who rotated the admissions, we were able to spend a part of our night on the inpatient ward. Most of the new patients had been thoroughly evaluated by a medical resident in the ED before we saw them, giving us a head start on eliciting their stories, performing physical examinations, reviewing the laboratory workup, and generating plans to present to the attending physician (who was called “the visit”) the next morning at 10 am .

We carried one pager that was about 7 inches long and 2 inches wide clipped to the waist of our pants. It could only make a beep; we then had to call the page operator to find out who wanted us. However, the pages were relatively few. Nurses called only when a patient was unstable, and other residents called only when a new patient was ready in the emergency department. At 9 pm , the laboratory data for the day were delivered to the ward nursing stations on computer-generated paper. Our job was to separate the pages connected by serrated breaks, review the results, and then file the pages in the charts. The nurses were aware of this routine, so they saved their questions for our presence on the wards, reducing the need to page us. At 10 pm , residents from all services went to the cafeteria for the “10 o’clock meal” when the food was free.We learned early in the year to drop everything at 10 pm ; otherwise, we did not eat. The social aspect of the group meal was comforting, but compared with today, the group was much more homogeneous and therefore less interesting. There were several women medical residents (although almost none in surgery), but very few minorities, and no openly gay residents.

Gathering data about patients prior to the current hospitalization required reviewing the “old chart,” which had to be delivered from patient records but was generally available when the patient was still in the ED. It contained typed discharge summaries and progress notes often handwritten by coresidents whom we knew. The handwriting was often difficult to read, outpatient notes were not included, and information from other hospitals was absent—but despite these deficiencies, we somehow managed just fine.

The patients on the inpatient ward were mostly stable, but more importantly, we had very few medications and tests to order. I recall prescribing fewer than 20 drugs—furosemide, hydrochlorothiazide, penicillins, cephalosporins, gentamicin, isoniazid, lidocaine, nitroglycerin, aminophylline, alpha-methyldopa, clonidine, propranolol, digoxin, hydralazine, indomethacin, steroids, and morphine. Orders for tests and imaging had to be physically written in the chart and could not be inputted remotely, which was a nuisance when we were away from the ward. However, we rarely ordered any imaging beyond plain radiographs at night. We did draw arterial blood gases and venous blood, administer oxygen, insert intravenous and central lines, take electrocardiograms, and perform urinalyses by microscopy. We did all these tasks ourselves for patients on the “ward service” (as opposed to the “private service”, which had to do with the type of insurance the patients possessed). As a result, we became experts in both blood drawing and intravenous line insertion—skills that might be less familiar to today’s residents.

Of course, patients did get acutely ill during the night. I recall intubating, cardioverting, performing phlebotomy to alleviate pulmonary edema, sending patients to surgery, and pronouncing death. Nevertheless, we often got sleep, and sometimes, several hours in a row. I had a rule; I always took a shower the next morning and put on clean clothes (we stayed until 5 pm , making the shifts 33.5 hours long). There was a camaraderie that existed between all of us at night. We were supportive, friendly, and knew each other by name, and more.

We were often frightened by the responsibility of managing sick patients alone. On particularly challenging nights, we would record our fears and feelings in a “night call diary” in one of the conference rooms—generally at 4 am . Some entries became legendary as people read and reread those months and years later. Mornings always brought a sense of relief and accomplishment, because when the sun came up, we knew that the other residents would not be far behind; when they arrived, we could tell our stories and get help.

There was definitely competitiveness to the work. Those who responded quickly to deteriorating patients were applauded; those who did not really know what to do were subtly disdained. However, over time, we all got the hang of it, and this led to a growing confidence that we were indeed doctors. The graded autonomy afforded by night call was a crucial part of that journey.

At 6 pm , the on-call residents assemble in the ED, where we would spend the remainder of the night and early morning admitting new patients to the hospital. A night team consists of a senior resident, three junior residents, and two medical students, with each resident being responsible for approximately 20 inpatients. Overnight coverage of the ward mostly occurs remotely; since the ED is often so busy, we address most of the issues through a computer or over the phone. Only in rare cases, such as when a patient is unstable or a death has occurred, do we deal with the matter in person.

To enable rapid remote responses, we each carry an assortment of devices on our waists or lanyards and in our pockets, such as a personal pager, ED consult pager, code blue pager, and hospital-issued smartphones capable of receiving pages, text messages, phone calls, and e-mails. Nurses, pharmacists, and other consultants communicate with us through all of these channels. Few of these interactions occur face-to-face. To our frustration, encounters with patients are frequently interrupted by a stream of beeps, rings, and vibrations—irrespective of whether we are having a difficult discussion about goals of care or performing a delicate procedure.

The ED contains a work space dedicated for residents to enter electronic orders, type notes, and review new admissions. Between consults, we try to discuss exciting cases and provide teaching to the medical students and interns, which we enjoy. Dinner is generally devoured while inputting orders. In exceptional circumstances, a brief reprieve from pages may allow the on-call team to share a meal. Depending on our role, sleep may be possible on certain nights but is never guaranteed. Moments spent with the on-call team—all of us learning, commiserating, and growing together—are some of the most memorable of residency, and many of us become close friends by the end of the rotation.

However, apart from these few familiar faces, we rarely get acquainted with the nurses or residents from other services. Many often refer to themselves by specialty rather than name and phone calls that begin with “Are you Medicine?” can end with “You should really call Orthopedics.” Meanwhile, “Medicine” and “Orthopedics” may pass each other in the hallway without recognition beyond a vague familiarity of a voice heard on the phone.

Every 10 minutes spent with a new patient is accompanied by approximately one hour of “electronic” time, which includes reading through previous medical records, reviewing laboratory data and imaging, and creating an admission note. Interns might groan as they pull up a patient’s electronic health record (EHR); irrelevant details often arise with each click of the mouse, and the cursed “copy-paste” function means that new notes often duplicate older ones. However, with time, we learn to look past the EHR’s shortcomings and appreciate several of its advantages. For example, we are now able to access test results performed outside our hospital and thus limit our repetition of investigations. We can also use the EHR to rapidly glean salient information about a patient in time-critical scenarios. This is always a satisfying process, and it makes us wonder how physicians ever practiced in the era before computers.

Today’s patients are older and sicker than ever before. Many are receiving treatments that did not exist even a decade ago. As residents, we must recognize a seemingly endless variety of drugs—a challenging but intellectually satisfying responsibility. We must also decide whether the patient’s current health state permits their continuation, or whether safer alternatives exist. Some of these decisions cannot wait until the morning.

During handover at 8 am , we often recount moments from our call shift with a sense of vigor that is only partly dulled by fatigue. We may share with pride our management of a sick patient. We may relay a touching exchange with a concerned family member. Or we may recall with satisfaction our handling of a tense situation with a colleague. Taking part in these experiences is one of the most character-building aspects of being on call. The absence of our supervisors can be unnerving at first, but we gradually begin to enjoy the sense of independence. Moreover, we feel empowered to make our patients better with the right combination of carefully selected treatments. Nothing makes us feel more like doctors.

No doubt, being on call is difficult. The next day brings a feeling of relief and accomplishment, knowing that we got through it—whether by floundering or flourishing—in one piece.

The two passages described here are personal descriptions of a typical night on-call in two different eras. Readers around the world may have a very different recollection of their own experience. Nevertheless, several aspects of being on call remain constant, such as anxiety about caring for sick patients alone, fond recollections of friends made, and relief when the morning comes. Most important, however, might be the tremendous satisfaction at the opportunity to learn and grow—to become a competent physician by testing one’s physical and intellectual limits through graded autonomy

On the other hand, certain elements of night call have undeniably changed—partly a consequence of the increased number of people involved in patient care and changing communication technology. Residents today encounter a greater number of interruptions to their work flow. Tasks that require long, continuous periods of full attention are now punctuated by texts, e-mails, calls, and pages. The EHR is often clumsy to navigate, but it can also be a veritable mine of information. Finally, although residents from the same specialty may be close friends, duty hour restrictions and remote asynchronous communication may reduce familiarity with residents from other programs.

Do these descriptions resonate with your experience of night call? Keeping in mind that the 1979 vignette is described through the rose-colored lens of nostalgia, both eras have their advantages and disadvantages. We leave it to the reader to decide what has changed (plus ça change) and what has stayed the same (plus c’est la même chose).

Acknowledgments

The authors thank Micheal A. Fifer, MD (Massachusetts General Hospital), and Timothy J. Judson, MD (UCSF), for their comments on an earlier draft of this essay.

Disclosures

The authors have nothing to disclose.

Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.¹ It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.^2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.⁷ GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).⁸

Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.¹ Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.¹

Stakeholder Recruitment

GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.⁸

Session Logistics

Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.⁵ Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.

Prompt Development

Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.¹ This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. ⁸ Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.

Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.

GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.

Description of the Seven Steps in GLA

In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.

In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.

Data Analysis

Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.

Complementary Data and Synthesis

Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.⁸

Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).¹ When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.

Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”¹ Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.

Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.

Disclosures

The authors have nothing to disclose.

Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.¹ It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.^2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.⁷ GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).⁸

Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.¹ Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.¹

Stakeholder Recruitment

GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.⁸

Session Logistics

Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.⁵ Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.

Prompt Development

Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.¹ This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. ⁸ Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.

Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.

GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.

Description of the Seven Steps in GLA

In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.

In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.

Data Analysis

Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.

Complementary Data and Synthesis

Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.⁸

Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).¹ When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.

Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”¹ Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.

Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.

Disclosures

The authors have nothing to disclose.

Group Level Assessment (GLA) is a qualitative research methodology designed to enable groups of stakeholders to generate and evaluate data in participatory sessions.¹ It has been used in diverse health-related settings for multiple research purposes, including needs/resource assessment, program evaluation, quality improvement, intervention development, feasibility/acceptability testing, knowledge generation, and prioritization.^2-6 Unlike traditional qualitative research methods in which participants provide data and researchers analyze it, GLA uses a seven-step structured process (Table) that actively involves a large group of stakeholders in the generation, interpretation, and synthesis of data and allows salient themes to be identified from stakeholders’ perspectives.⁷ GLA deliverables include a set of action items that are relevant to the target issue and representative of the collective view of stakeholders. In this issue of the Journal of Hospital Medicine, Choe and colleagues used GLA methodology to identify the perspectives of pediatric medical providers and interpreters with regard to the use of interpreter services for hospitalized children having limited English proficiency (LEP).⁸

Each individual GLA session is intended for a group of 15-60 stakeholders. Ideally, a GLA session is scheduled for approximately three hours with a skilled facilitator guiding the group through the steps of the session.¹ Depending on the study scope and research questions, modifications to GLA can be made when engaging fewer stakeholders, conducting the GLA across several shorter sessions with the same group, or conducting multiple sessions with different stakeholder groups wherein results are integrated across the groups.¹

Stakeholder Recruitment

GLAs are designed to bring diverse groups together to be able to generate and evaluate ideas collectively, which in turn helps to reduce potential power differentials between or among participants. Depending on the research question(s), relevant stakeholders may include local community residents, patients, caregivers, community leaders, practitioners, providers, community-based organizations, and even CEOs. The use of purposeful sampling techniques can obtain a diverse group of stakeholders, thus helping ensure a wide range of ideas and perspectives. Choe and colleagues used flyers and announcements at staff meetings to recruit physicians, nursing staff, and interpreters who were subsequently assigned to GLA sessions to ensure engagement from a range of stakeholder roles at each session.⁸

Session Logistics

Strategies to create an open, equitable atmosphere in GLA sessions include role-based assigning of individuals to specific groups, avoiding introductions that emphasize status, pre-education for any leaders and supervisors about the participatory and equitable nature of GLA, and minimizing cliques and overly dominant voices throughout the session. Stakeholders who take part in activities in a GLA session typically receive an incentive for participating. Additional supports such as food and childcare may be considered. GLA sessions involving children may require providing the young participants assistance in writing their responses and/or the use of additional facilitators to keep the small groups on track.⁵ Interpreters and facilitators can be incorporated into GLA sessions to assist stakeholders who may need assistance with understanding and responding to prompts, such as language interpretation and translation services.

Prompt Development

Similar to the development of questions for interview and focus group guides, creating effective prompts is a critical component of data collection in GLA. Prompts are statements worded as incomplete or fill-in-the-blank sentences that should be open ended to allow participants to respond with their own thoughts and experiences. Prompts that resemble the beginning of a sentence (eg, “The biggest challenge we face is…”) encourage honest reflection rather than questions that can make participants feel like they are being evaluated. We recommend varying the number of prompts based on the group size: approximately one chart and prompt per person attending, with a maximum of 35 prompts at one session.¹ This allows for sufficient variability in the responses generated without being overwhelming or too time-consuming. For example, Choe et al. developed a pool of 51 unique prompts addressing their research questions and then used 15-32 prompts in each GLA session, depending on the number of participants. ⁸ Prompts should be written with some purposeful redundancy, targeting the research question from several angles. The emphasis should be on the content’s alignment with the research questions rather than the actual wording of the prompts as a way of ensuring that the generated data is both valid and useful.

Prompts should also vary in format, style (eg, different color markers, pictures, fonts, etc.), and placement on each flip chart page. An individual flip chart can include multiple related prompts: for example, “split-halves” in two columns or rows (ie, the best part/worst part). Taken as a whole, the flip charts and accompanying prompts create different lenses for gathering participant perspectives on the research questions. See Appendix Table for suggested prompt characteristics and examples from a hypothetical study related to pediatric healthcare.

GLA prompt development will ideally occur in collaboration with an advisory team comprised of representative members from each of the stakeholder groups. Using a participatory research approach in the research design and preparation phases ensures that GLA prompts are understandable and relevant to participants and are able to appropriately capture the underlying purpose of the study.

Description of the Seven Steps in GLA

In step one, climate setting, the facilitator provides an overview of the session, including a description of the GLA rationale and process. Typically, an icebreaker or brief introduction activity is conducted. Step two, generating, is a hallmark step of GLA in which participants walk around and respond to prompts prewritten on flip charts hung on walls in a large room. Participants use markers and respond to each prompt by either providing a unique comment and/or corroborating an existing comment by adding a checkmark or star. During this step, organizers typically play music and encourage participants to enjoy food, chat with fellow participants, and leisurely move from prompt to prompt in any order. Step three, appreciating, is a brief interim step where participants take a “gallery walk” and view responses written on the charts.

In step four, reflecting, participants reflect on the data and briefly write down their thoughts about the responses generated in the session. In step five, understanding, smaller groups synthesize responses across a subset of charts and report their findings to the larger group. Depending on the size and composition of the larger group, small groups of four to seven people are formed or assigned. Each small group is assigned a subset of approximately four to six charts. Using thematic analysis, participants look for relationships among the responses on their assigned charts, referring to individual responses as evidence for the main findings. Groups will take notes on the charts, circle key phrases, or draw arrows to show relationships in the data and thereafter develop themes. As each small group reports their findings, the facilitator will keep a running list of generated themes, ideally in the participants’ own words. Step six, selecting, involves participants discussing, further synthesizing, and prioritizing data. Step six can occur as a facilitated large group discussion or in a form in which participants can remain in the same small groups from step five and work together to complete this further step. Themes across all of the small groups are consolidated and developed into overarching themes. Step seven, action, includes planning the next steps to address priorities.

Data Analysis

Analyzing the data generated through a GLA is an iterative process incorporated into steps three to seven as described above and often continues after the GLA session is complete. Step seven can be scheduled as a separate action-planning session depending on time constraints and the study goals. This final step moves the group toward interpretation and dissemination as themes are prioritized and used to drive action steps toward a programmatic, policy, or community change. In some studies, themes will be aggregated across multiple GLAs to integrate the findings from several sessions. This step is sometimes completed with a smaller group of stakeholders, an advisory board, or the research team.

Complementary Data and Synthesis

Research teams often collect additional sources of data that are later used to analyze and interpret the initial stakeholder-developed findings (ie, demographic surveys) and to identify priority areas. Field notes, photographs of completed charts, and recorded participant quotes can also be incorporated into the thematic analysis. Small and large group discussions could be audio recorded and transcribed to capture participants’ individual comments and interpretations. In Choe et al. the team recorded detailed notes, including quotations from participants, and collected a demographic survey. After each GLA session, Choe and colleagues compiled all of the stakeholder-driven findings to develop an overarching set of themes related to communication with LEP families and priority areas that could inform subsequent action. Similar to the qualitative validation strategy of member checking, the authors shared and revised this overarching set of themes in discussion with stakeholders to ensure that participant ideas were adequately and accurately represented.⁸

Compared to traditional qualitative methods such as one-on-one interviews and focus groups, GLA is designed for large groups and is used to promote active engagement of diverse stakeholders in the participatory process. Unlike many other qualitative methods, GLA provides a stakeholder-driven, structured format to elicit diverse stakeholder viewpoints in the moment and build consensus in a participatory manner about priorities and subsequent actions. The progression of the GLA process is collaborative, with stakeholders generating, analyzing, and prioritizing data from their own perspectives. In a focus group or one-on-one interviews, researchers would conduct the analysis after the audio recordings were transcribed. In GLA, stakeholders conduct a thematic analysis in real time, an aspect that adds the stakeholder perspective to analysis of the findings, interpretation, and implications. GLA offers a fun and interactive experience that can build a sense of community among participants (eg, walking around, impromptu conversation, working in small groups, sharing perspectives on the same issue from different vantage points, etc.). GLA is a versatile, flexible methodology that can be used to address different research objectives, be modified for use with various size groups, and be adapted based on the needs and characteristics of stakeholders (eg, children, people with disabilities, etc.).¹ When used in recruitment, GLA is designed to include stakeholders representing different roles and levels of a system. GLA can be particularly useful when engaging underserved communities in research because the process is nonthreatening and promotive of shared perspectives and decision-making. Importantly, the final step of GLA provides interested stakeholders with a way to stay involved in the research through prioritization and action.

Like other self-report research methods, GLA relies on stakeholder comfort and willingness to share “public data.”¹ Thus, controversial or sensitive issues may not be brought forth. Since the final themes of GLA are consensus based in terms of what the group of stakeholders finds to be most important, nuances and outlier data can be missed. Successfully conducting a GLA requires a skilled, flexible facilitator who can manage group dynamics while also balancing the structure of the seven-step process, promoting an open and equitable environment, and ensuring the research process remains rigorous. Large groups can be more difficult for facilitators to manage especially when there are power differentials, conflict, and hidden agendas among stakeholders. The large group design, multiple steps of GLA, and participatory atmosphere with music and food can be off-putting for some stakeholders who find the process too noisy, overwhelming, or unstructured. In addition, large groups can be challenging to schedule at times and to find locations that are convenient for stakeholders.

Compared to researcher-driven qualitative methods that can be resource-intensive and are limited by researcher perspective, GLA emphasizes the contextual, “lived” expertise of stakeholders and relies on them in real time to identify and prioritize matters relevant to the participants. The participatory process of GLA promotes stakeholder buy-in and builds on the collective wisdom of the stakeholder group. This is ideally seen in Choe et al.’s study where GLA offered the researchers a structured qualitative methodology that engaged a large number of medical providers and interpreters to identify effective practices that should ultimately enhance communication with families of hospitalized LEP children.

Disclosures

The authors have nothing to disclose.

Acute kidney injury (AKI) occurs in 5%-30% of noncritically ill hospitalized children.¹ Initially thought to be simply a symptom of more severe pathologies, it is now recognized that AKI independently increases mortality and is associated with the development of chronic kidney disease (CKD), even in children.² The wide acceptance of the Kidney Disease Improving Global Outcome (KDIGO) diagnostic criteria has enabled a more uniform definition of AKI from both clinical and research perspectives.² A better understanding of the pathophysiology and risk factors for AKI has led to new methods for early detection and prevention efforts. While serum creatinine (SCr) was historically one of the sole markers of AKI, novel biomarkers can facilitate earlier diagnosis of AKI, identify subclinical AKI, and guide clinical management. This clinical practice update addresses the latest clinical advances in risk assessment, diagnosis, and prevention of pediatric AKI, with a focus on AKI biomarkers.

Several sets of criteria have been used to diagnose AKI. The KDIGO classification, based on a systematic review of the literature and developed through expert consensus, is the current recommended definition.³ Increasing AKI stage, as defined by the KDIGO classification, is associated with increased mortality, the need for renal replacement therapy, length of stay, and CKD, thus underscoring the importance of accurate classification.³ Stage 1 AKI is defined by a rise in SCr of ≥0.3 mg/dL,1.5-1.9 times the baseline SCr, or urine output <0.5 ml/kg/h for six to 12 hours; stage 2 by a rise of ≥2.0-2.9 times the baseline SCr or urine output <0.5 ml/kg/h for >12 hours; and stage 3 by a rise of ≥4.0 mg/dL, ≥three times the baseline SCr, initiation of renal replacement therapy, urine output <0.3 ml/kg/h for ≥24 hours, or anuria ≥12 hours. However, these criteria rely on SCr, which is a suboptimal marker of renal dysfunction, as it rises only once the glomerular filtration rate (GFR) has already decreased, in some cases by as much as 50%. Additionally, interpretation of SCr in the diagnosis of AKI requires a prior Scr measurement to determine the magnitude of change from the baseline value, which is often lacking in children. To mitigate this limitation, different formulas exist to estimate a baseline SCr value based on height or age, an approach that assumes patients have preexisting normal renal function.

The limitations of SCr have led to interest in identifying more accurate biomarkers of AKI. Although many candidates have been identified, we will limit our discussion to those currently available for clinical use: serum cystatin C, urine neutrophil gelatinase-associated lipocalin (NGAL), urine TIMP-2, and urine IGFBP7 (Table).^4-8 While urine NGAL and cystatin C are measured individually, TIMP-2 and IGFBP7 are measured on the same panel and the product of their multiplied values is used for clinical guidance. While each of these biomarkers have good predictive accuracy for AKI when used independently, their combined use increases the accuracy of AKI diagnosis. These biomarkers can be divided into broad categories based on their utility as either functional markers or markers of injury.⁶ Serum cystatin C is a functional marker and as such can be used to estimate GFR more accurately than SCr.⁹ Comparatively, urine NGAL is a marker of renal injury, while TIMP2 and IGFBP7 are markers of renal stress. These markers are not useful in estimating GFR, but rather aid in the prediction and diagnosis of AKI (Figure). Despite the limitations of SCr, these biomarkers have yet to be incorporated into the diagnostic criteria. They have, however, helped to refine our understanding of the pathophysiology of AKI.

The renal angina index (RAI) identifies critically ill children at high risk for AKI. The RAI combines traditional markers of AKI, such as a change in estimated creatinine clearance and fluid overload, with patient factors, including need for ventilation, inotropic support, and history of transplantation (solid organ or bone marrow) to identify those patients who are at high risk for severe AKI. Patients identified as high risk by the patient factors component of the RAI have a much lower threshold for both a decrease in creatinine clearance and fluid overload to be considered at risk for severe AKI, as these early signs are more likely to reflect an early impending severe AKI in this high-risk group. Conversely, patients that do not meet these patient factors are more likely to simply have a transient or functional AKI, and therefore have a higher threshold for both a change in creatinine clearance and fluid overload in order to be considered at high risk for severe AKI.

The RAI has been validated in the critical care setting as a method to predict severe AKI at day three of admission to the pediatric intensive care unit, with a negative predictive value of 92%-99% when the score is negative in the first 12 hours.¹⁰ In selected high-risk patients (RAI ≥ 8), biomarkers become even more reliable for AKI prediction (eg, injury markers have an excellent area under the receiver operating characteristic curve (AUC) of 0.97 for severe AKI prediction in this high-risk group).¹¹ While only validated for critically ill patients, the concept of renal angina is still applicable in the complex populations managed by hospitalists who practice outside of the intensive care unit setting. Early signs of renal dysfunction (eg, rising SCr, fluid overload ≥5%) in patients with risk factors (see below) should prompt a thorough evaluation, including urinalysis, daily SCr, nephrotoxin avoidance, and tissue injury biomarkers, if available.

The risk factors for AKI are numerous and tend to potentiate one another. The most frequent predisposing comorbidities include CKD, heart failure or congenital heart diseases, transplantation (bone marrow or solid organs), and diabetes. Disease-related factors include sepsis, cardiac surgery, cardio-pulmonary bypass, mechanical ventilation, and vasopressor use. Potentially modifiable factors include hypovolemia and multiple nephrotoxic exposures. ^2,3

Nephrotoxic medications are now among the most common causes of AKI in hospitalized children.¹² Approximately 80% of children are exposed to at least one nephrotoxin during an inpatient admission.¹² Exposure to a single nephrotoxic medication is sufficient to place a child at risk of AKI, and each additional nephrotoxin further increases the risk.¹² While some drugs are routinely recognized to be nephrotoxic (eg, ibuprofen), others are commonly overlooked, notably certain antibiotics (eg, cefotaxime, ceftazidime, cefuroxime, nafcillin, and piperacillin) and anticonvulsants (eg, zonisamide).¹² Furthermore, the combination of multiple nephrotoxins can potentiate the risk of AKI. For example, the combination of vancomycin and piperacillin/tazobactam increases the risk of AKI by 3.4 times compared with the combination of vancomycin with another antipseudomonal beta-lactam antibiotic.¹³

Adequate monitoring, including daily SCr measurements and risk awareness, are critical as nephrotoxin-associated AKI can be easily missed in the absence of routine SCr monitoring, especially since these children are typically nonoliguric¹². Quality improvement efforts focused on obtaining daily SCr in patients exposed to either three or more nephrotoxins or three days of either aminoglycoside or vancomycin, even without concomitant exposure to other nephrotoxins, have shown success in decreasing both the number of nephrotoxins and the rate of nephrotoxin-associated AKI.¹²

While a significant injury cannot always be avoided, a mindful clinical approach and management can help to prevent some complications of AKI. An awareness of fluid status is critical, as fluid overload greater than 10% of the patient’s weight independently increases the risk of mortality in both adults and children.¹⁴ To assess the risk of AKI progression and potential failure of conservative management with diuretics, a furosemide stress test (FST) is an easy, safe, and accessible functional assessment of tubular reserve in a patient without intravascular depletion.¹⁵ A growing body of literature in adults shows that FST-responders are less likely to progress to stage 3 AKI or need renal replacement therapy than nonresponders.¹⁵ The FST is currently being investigated and standardized in children.

Research in AKI has made significant strides over the last few years. Nevertheless, many areas of research remain to be explored (eg, the impact of IV fluid type in the pediatric population, AKD characterization and impact on CKD development). AKI is common, associated with significant morbidity and mortality and, in some instances, preventable. While no targeted therapeutic options are currently under investigation, recent advances allow for better identification of high-risk patients and offer opportunities for impactful preventive approaches. Thoughtful use of nephrotoxic medications, early identification of patients at high risk for AKI, and accurate diagnosis and appropriate management of AKI are the recommended best practice.

Disclosures

The authors have nothing to disclose.

Acute kidney injury (AKI) occurs in 5%-30% of noncritically ill hospitalized children.¹ Initially thought to be simply a symptom of more severe pathologies, it is now recognized that AKI independently increases mortality and is associated with the development of chronic kidney disease (CKD), even in children.² The wide acceptance of the Kidney Disease Improving Global Outcome (KDIGO) diagnostic criteria has enabled a more uniform definition of AKI from both clinical and research perspectives.² A better understanding of the pathophysiology and risk factors for AKI has led to new methods for early detection and prevention efforts. While serum creatinine (SCr) was historically one of the sole markers of AKI, novel biomarkers can facilitate earlier diagnosis of AKI, identify subclinical AKI, and guide clinical management. This clinical practice update addresses the latest clinical advances in risk assessment, diagnosis, and prevention of pediatric AKI, with a focus on AKI biomarkers.

Several sets of criteria have been used to diagnose AKI. The KDIGO classification, based on a systematic review of the literature and developed through expert consensus, is the current recommended definition.³ Increasing AKI stage, as defined by the KDIGO classification, is associated with increased mortality, the need for renal replacement therapy, length of stay, and CKD, thus underscoring the importance of accurate classification.³ Stage 1 AKI is defined by a rise in SCr of ≥0.3 mg/dL,1.5-1.9 times the baseline SCr, or urine output <0.5 ml/kg/h for six to 12 hours; stage 2 by a rise of ≥2.0-2.9 times the baseline SCr or urine output <0.5 ml/kg/h for >12 hours; and stage 3 by a rise of ≥4.0 mg/dL, ≥three times the baseline SCr, initiation of renal replacement therapy, urine output <0.3 ml/kg/h for ≥24 hours, or anuria ≥12 hours. However, these criteria rely on SCr, which is a suboptimal marker of renal dysfunction, as it rises only once the glomerular filtration rate (GFR) has already decreased, in some cases by as much as 50%. Additionally, interpretation of SCr in the diagnosis of AKI requires a prior Scr measurement to determine the magnitude of change from the baseline value, which is often lacking in children. To mitigate this limitation, different formulas exist to estimate a baseline SCr value based on height or age, an approach that assumes patients have preexisting normal renal function.

The limitations of SCr have led to interest in identifying more accurate biomarkers of AKI. Although many candidates have been identified, we will limit our discussion to those currently available for clinical use: serum cystatin C, urine neutrophil gelatinase-associated lipocalin (NGAL), urine TIMP-2, and urine IGFBP7 (Table).^4-8 While urine NGAL and cystatin C are measured individually, TIMP-2 and IGFBP7 are measured on the same panel and the product of their multiplied values is used for clinical guidance. While each of these biomarkers have good predictive accuracy for AKI when used independently, their combined use increases the accuracy of AKI diagnosis. These biomarkers can be divided into broad categories based on their utility as either functional markers or markers of injury.⁶ Serum cystatin C is a functional marker and as such can be used to estimate GFR more accurately than SCr.⁹ Comparatively, urine NGAL is a marker of renal injury, while TIMP2 and IGFBP7 are markers of renal stress. These markers are not useful in estimating GFR, but rather aid in the prediction and diagnosis of AKI (Figure). Despite the limitations of SCr, these biomarkers have yet to be incorporated into the diagnostic criteria. They have, however, helped to refine our understanding of the pathophysiology of AKI.

The renal angina index (RAI) identifies critically ill children at high risk for AKI. The RAI combines traditional markers of AKI, such as a change in estimated creatinine clearance and fluid overload, with patient factors, including need for ventilation, inotropic support, and history of transplantation (solid organ or bone marrow) to identify those patients who are at high risk for severe AKI. Patients identified as high risk by the patient factors component of the RAI have a much lower threshold for both a decrease in creatinine clearance and fluid overload to be considered at risk for severe AKI, as these early signs are more likely to reflect an early impending severe AKI in this high-risk group. Conversely, patients that do not meet these patient factors are more likely to simply have a transient or functional AKI, and therefore have a higher threshold for both a change in creatinine clearance and fluid overload in order to be considered at high risk for severe AKI.

The RAI has been validated in the critical care setting as a method to predict severe AKI at day three of admission to the pediatric intensive care unit, with a negative predictive value of 92%-99% when the score is negative in the first 12 hours.¹⁰ In selected high-risk patients (RAI ≥ 8), biomarkers become even more reliable for AKI prediction (eg, injury markers have an excellent area under the receiver operating characteristic curve (AUC) of 0.97 for severe AKI prediction in this high-risk group).¹¹ While only validated for critically ill patients, the concept of renal angina is still applicable in the complex populations managed by hospitalists who practice outside of the intensive care unit setting. Early signs of renal dysfunction (eg, rising SCr, fluid overload ≥5%) in patients with risk factors (see below) should prompt a thorough evaluation, including urinalysis, daily SCr, nephrotoxin avoidance, and tissue injury biomarkers, if available.

The risk factors for AKI are numerous and tend to potentiate one another. The most frequent predisposing comorbidities include CKD, heart failure or congenital heart diseases, transplantation (bone marrow or solid organs), and diabetes. Disease-related factors include sepsis, cardiac surgery, cardio-pulmonary bypass, mechanical ventilation, and vasopressor use. Potentially modifiable factors include hypovolemia and multiple nephrotoxic exposures. ^2,3

Nephrotoxic medications are now among the most common causes of AKI in hospitalized children.¹² Approximately 80% of children are exposed to at least one nephrotoxin during an inpatient admission.¹² Exposure to a single nephrotoxic medication is sufficient to place a child at risk of AKI, and each additional nephrotoxin further increases the risk.¹² While some drugs are routinely recognized to be nephrotoxic (eg, ibuprofen), others are commonly overlooked, notably certain antibiotics (eg, cefotaxime, ceftazidime, cefuroxime, nafcillin, and piperacillin) and anticonvulsants (eg, zonisamide).¹² Furthermore, the combination of multiple nephrotoxins can potentiate the risk of AKI. For example, the combination of vancomycin and piperacillin/tazobactam increases the risk of AKI by 3.4 times compared with the combination of vancomycin with another antipseudomonal beta-lactam antibiotic.¹³

Adequate monitoring, including daily SCr measurements and risk awareness, are critical as nephrotoxin-associated AKI can be easily missed in the absence of routine SCr monitoring, especially since these children are typically nonoliguric¹². Quality improvement efforts focused on obtaining daily SCr in patients exposed to either three or more nephrotoxins or three days of either aminoglycoside or vancomycin, even without concomitant exposure to other nephrotoxins, have shown success in decreasing both the number of nephrotoxins and the rate of nephrotoxin-associated AKI.¹²

While a significant injury cannot always be avoided, a mindful clinical approach and management can help to prevent some complications of AKI. An awareness of fluid status is critical, as fluid overload greater than 10% of the patient’s weight independently increases the risk of mortality in both adults and children.¹⁴ To assess the risk of AKI progression and potential failure of conservative management with diuretics, a furosemide stress test (FST) is an easy, safe, and accessible functional assessment of tubular reserve in a patient without intravascular depletion.¹⁵ A growing body of literature in adults shows that FST-responders are less likely to progress to stage 3 AKI or need renal replacement therapy than nonresponders.¹⁵ The FST is currently being investigated and standardized in children.

Research in AKI has made significant strides over the last few years. Nevertheless, many areas of research remain to be explored (eg, the impact of IV fluid type in the pediatric population, AKD characterization and impact on CKD development). AKI is common, associated with significant morbidity and mortality and, in some instances, preventable. While no targeted therapeutic options are currently under investigation, recent advances allow for better identification of high-risk patients and offer opportunities for impactful preventive approaches. Thoughtful use of nephrotoxic medications, early identification of patients at high risk for AKI, and accurate diagnosis and appropriate management of AKI are the recommended best practice.

Disclosures

The authors have nothing to disclose.

Acute kidney injury (AKI) occurs in 5%-30% of noncritically ill hospitalized children.¹ Initially thought to be simply a symptom of more severe pathologies, it is now recognized that AKI independently increases mortality and is associated with the development of chronic kidney disease (CKD), even in children.² The wide acceptance of the Kidney Disease Improving Global Outcome (KDIGO) diagnostic criteria has enabled a more uniform definition of AKI from both clinical and research perspectives.² A better understanding of the pathophysiology and risk factors for AKI has led to new methods for early detection and prevention efforts. While serum creatinine (SCr) was historically one of the sole markers of AKI, novel biomarkers can facilitate earlier diagnosis of AKI, identify subclinical AKI, and guide clinical management. This clinical practice update addresses the latest clinical advances in risk assessment, diagnosis, and prevention of pediatric AKI, with a focus on AKI biomarkers.

Several sets of criteria have been used to diagnose AKI. The KDIGO classification, based on a systematic review of the literature and developed through expert consensus, is the current recommended definition.³ Increasing AKI stage, as defined by the KDIGO classification, is associated with increased mortality, the need for renal replacement therapy, length of stay, and CKD, thus underscoring the importance of accurate classification.³ Stage 1 AKI is defined by a rise in SCr of ≥0.3 mg/dL,1.5-1.9 times the baseline SCr, or urine output <0.5 ml/kg/h for six to 12 hours; stage 2 by a rise of ≥2.0-2.9 times the baseline SCr or urine output <0.5 ml/kg/h for >12 hours; and stage 3 by a rise of ≥4.0 mg/dL, ≥three times the baseline SCr, initiation of renal replacement therapy, urine output <0.3 ml/kg/h for ≥24 hours, or anuria ≥12 hours. However, these criteria rely on SCr, which is a suboptimal marker of renal dysfunction, as it rises only once the glomerular filtration rate (GFR) has already decreased, in some cases by as much as 50%. Additionally, interpretation of SCr in the diagnosis of AKI requires a prior Scr measurement to determine the magnitude of change from the baseline value, which is often lacking in children. To mitigate this limitation, different formulas exist to estimate a baseline SCr value based on height or age, an approach that assumes patients have preexisting normal renal function.

The limitations of SCr have led to interest in identifying more accurate biomarkers of AKI. Although many candidates have been identified, we will limit our discussion to those currently available for clinical use: serum cystatin C, urine neutrophil gelatinase-associated lipocalin (NGAL), urine TIMP-2, and urine IGFBP7 (Table).^4-8 While urine NGAL and cystatin C are measured individually, TIMP-2 and IGFBP7 are measured on the same panel and the product of their multiplied values is used for clinical guidance. While each of these biomarkers have good predictive accuracy for AKI when used independently, their combined use increases the accuracy of AKI diagnosis. These biomarkers can be divided into broad categories based on their utility as either functional markers or markers of injury.⁶ Serum cystatin C is a functional marker and as such can be used to estimate GFR more accurately than SCr.⁹ Comparatively, urine NGAL is a marker of renal injury, while TIMP2 and IGFBP7 are markers of renal stress. These markers are not useful in estimating GFR, but rather aid in the prediction and diagnosis of AKI (Figure). Despite the limitations of SCr, these biomarkers have yet to be incorporated into the diagnostic criteria. They have, however, helped to refine our understanding of the pathophysiology of AKI.

The renal angina index (RAI) identifies critically ill children at high risk for AKI. The RAI combines traditional markers of AKI, such as a change in estimated creatinine clearance and fluid overload, with patient factors, including need for ventilation, inotropic support, and history of transplantation (solid organ or bone marrow) to identify those patients who are at high risk for severe AKI. Patients identified as high risk by the patient factors component of the RAI have a much lower threshold for both a decrease in creatinine clearance and fluid overload to be considered at risk for severe AKI, as these early signs are more likely to reflect an early impending severe AKI in this high-risk group. Conversely, patients that do not meet these patient factors are more likely to simply have a transient or functional AKI, and therefore have a higher threshold for both a change in creatinine clearance and fluid overload in order to be considered at high risk for severe AKI.

The RAI has been validated in the critical care setting as a method to predict severe AKI at day three of admission to the pediatric intensive care unit, with a negative predictive value of 92%-99% when the score is negative in the first 12 hours.¹⁰ In selected high-risk patients (RAI ≥ 8), biomarkers become even more reliable for AKI prediction (eg, injury markers have an excellent area under the receiver operating characteristic curve (AUC) of 0.97 for severe AKI prediction in this high-risk group).¹¹ While only validated for critically ill patients, the concept of renal angina is still applicable in the complex populations managed by hospitalists who practice outside of the intensive care unit setting. Early signs of renal dysfunction (eg, rising SCr, fluid overload ≥5%) in patients with risk factors (see below) should prompt a thorough evaluation, including urinalysis, daily SCr, nephrotoxin avoidance, and tissue injury biomarkers, if available.

The risk factors for AKI are numerous and tend to potentiate one another. The most frequent predisposing comorbidities include CKD, heart failure or congenital heart diseases, transplantation (bone marrow or solid organs), and diabetes. Disease-related factors include sepsis, cardiac surgery, cardio-pulmonary bypass, mechanical ventilation, and vasopressor use. Potentially modifiable factors include hypovolemia and multiple nephrotoxic exposures. ^2,3

Nephrotoxic medications are now among the most common causes of AKI in hospitalized children.¹² Approximately 80% of children are exposed to at least one nephrotoxin during an inpatient admission.¹² Exposure to a single nephrotoxic medication is sufficient to place a child at risk of AKI, and each additional nephrotoxin further increases the risk.¹² While some drugs are routinely recognized to be nephrotoxic (eg, ibuprofen), others are commonly overlooked, notably certain antibiotics (eg, cefotaxime, ceftazidime, cefuroxime, nafcillin, and piperacillin) and anticonvulsants (eg, zonisamide).¹² Furthermore, the combination of multiple nephrotoxins can potentiate the risk of AKI. For example, the combination of vancomycin and piperacillin/tazobactam increases the risk of AKI by 3.4 times compared with the combination of vancomycin with another antipseudomonal beta-lactam antibiotic.¹³

Adequate monitoring, including daily SCr measurements and risk awareness, are critical as nephrotoxin-associated AKI can be easily missed in the absence of routine SCr monitoring, especially since these children are typically nonoliguric¹². Quality improvement efforts focused on obtaining daily SCr in patients exposed to either three or more nephrotoxins or three days of either aminoglycoside or vancomycin, even without concomitant exposure to other nephrotoxins, have shown success in decreasing both the number of nephrotoxins and the rate of nephrotoxin-associated AKI.¹²

While a significant injury cannot always be avoided, a mindful clinical approach and management can help to prevent some complications of AKI. An awareness of fluid status is critical, as fluid overload greater than 10% of the patient’s weight independently increases the risk of mortality in both adults and children.¹⁴ To assess the risk of AKI progression and potential failure of conservative management with diuretics, a furosemide stress test (FST) is an easy, safe, and accessible functional assessment of tubular reserve in a patient without intravascular depletion.¹⁵ A growing body of literature in adults shows that FST-responders are less likely to progress to stage 3 AKI or need renal replacement therapy than nonresponders.¹⁵ The FST is currently being investigated and standardized in children.

Research in AKI has made significant strides over the last few years. Nevertheless, many areas of research remain to be explored (eg, the impact of IV fluid type in the pediatric population, AKD characterization and impact on CKD development). AKI is common, associated with significant morbidity and mortality and, in some instances, preventable. While no targeted therapeutic options are currently under investigation, recent advances allow for better identification of high-risk patients and offer opportunities for impactful preventive approaches. Thoughtful use of nephrotoxic medications, early identification of patients at high risk for AKI, and accurate diagnosis and appropriate management of AKI are the recommended best practice.

Disclosures

The authors have nothing to disclose.

Systematic efforts to measure and compare work hours emerged in the 19^th century as laborers shifted from artisanal shops to factories, sparking debate over the appropriate length and intensity of work.¹ Two centuries of unionization and regulation defined work hours for many United States employees, including graduate medical trainees, but left attending physicians largely untouched. Instead, the medical workforce has long relied on survey data to shape jobs that balance professional norms with local market demands. Leaders in young, dynamic specialties, such as pediatric hospital medicine (PHM), particularly require such data to recruit and retain talent.

PHM progressed swiftly from acknowledgment as a “distinct area of practice” in 1999 to a subspecialty recognition.² Currently, at least 3,000 pediatric hospitalists³ practice in more than 800 US sites (Snow C, Personal communication regarding community PHM workforce survey). Approximately half of them work at community hospitals, where PHM groups often comprise fewer than five full-time equivalents (FTEs) and face unique challenges. Community PHM practices may assume broader responsibilities than university/children’s hospital colleagues, including advocacy for the needs of children in predominantly adult-oriented hospitals.⁴ Although data regarding academic PHM work demands are available,⁵ there is little information pertaining to community hospitalists regarding typical workloads or other characteristics of thriving practices.

In this issue of the Journal of Hospital Medicine, Alvarez et al. present the findings of structured interviews with 70 community PHM group leaders.⁶ Each participant answered 12 questions about their group, addressing the definition of a full-time workload and hours, the design of backup systems, and the respondent’s perception of the program’s sustainability. The sample is robust, with the caveats that it disproportionately represents the Midwest and West (34.3% each) and more than half of the groups were employed by an academic institution. The authors found a median work expectation per FTE of 1,882 hours/year and 21 weekends per year, although they noted significant variability in employers’ demands and services provided. The majority of hospitalist groups lacked census caps, formal backup systems, or processes to expand coverage during busy seasons. Among the site leaders, 63% perceived their program as sustainable, but no program design or employer characteristic was clearly associated with this perception.

The importance of this study derives from aggregating data about the largest cross section of community PHM groups yet reported. For many PHM group leaders, this will offer a new point of reference for key practice characteristics. Furthermore, the authors should be commended for attempting to distinguish how program sustainability manifests in community PHM, where hospitalists shoulder longer patient care hours and many of them sustain academic endeavors. It is concerning that more than a third of leaders do not perceive their program as sustainable, but the implications for the field are unclear. Perhaps part of this uncertainty arises from the terminology, as sustainability lacks a technical or a consensus definition and the authors purposefully did not define the term for the respondents. While many respondents probably worried about physician burnout, others might have channeled fears about group finances or competition with adult service lines for beds. In addition, leaders’ fears about sustainability may not exactly represent the concerns of front-line employees.

Sustainable work environments are complex constructs with several inputs. For example, supportive leaders, efficient delivery systems, optimized EHRs, competitive pay, and confidence about service line stability might all mitigate higher workloads. Ultimately, this complexity underscores an important caution about all workplace surveys in medicine; ie, average values can inform practice design, but hospitalists and administrators should always consider the local context. Blindly applying medians as benchmarks and ignoring the myriad other contributors to sustainable practice risk disrupting successful PHM programs. In other words, surveys describe how the world is, not how it should be. The spectrum of academic work and norms permeating community PHM groups instead call for a nuanced approach.

How does the field build upon this useful paper? First, the Society of Hospital Medicine (SHM) should engage PHM leaders to increase participation in regular remeasurement, a critical endeavor for this dynamic field. SHM’s State of Hospital Medicine Report queries about a wider variety of practice characteristics, but it has a smaller sample size that must grow to fill this void.⁷ As the work of repeated surveys transitions from academic inquiry to professional society service, SHM’s Practice Analysis Committee can meet the needs of PHM through relevant questions and efforts to foster adequate participation. Second, all practice leaders should follow the ballooning bodies of literature about burnout and healthcare value. Just as labor leaders had discovered in the industrial revolution, sustainable careers require not only measuring work hours but also advocating for safe, meaningful, and engaging work conditions. By continuously creating value for patients, families, and hospitals, we can strengthen our claim to the resources needed to optimize the work environment.

Disclosure

Andrew White is Chair of the Society of Hospital Medicine’s Practice Analysis Committee, an unpaid position. Dr. Marek serves on the American Academy of Pediatrics Section on Hospital Medicine Executive Committee which is a voluntary, unpaid, elected position.

Systematic efforts to measure and compare work hours emerged in the 19^th century as laborers shifted from artisanal shops to factories, sparking debate over the appropriate length and intensity of work.¹ Two centuries of unionization and regulation defined work hours for many United States employees, including graduate medical trainees, but left attending physicians largely untouched. Instead, the medical workforce has long relied on survey data to shape jobs that balance professional norms with local market demands. Leaders in young, dynamic specialties, such as pediatric hospital medicine (PHM), particularly require such data to recruit and retain talent.

PHM progressed swiftly from acknowledgment as a “distinct area of practice” in 1999 to a subspecialty recognition.² Currently, at least 3,000 pediatric hospitalists³ practice in more than 800 US sites (Snow C, Personal communication regarding community PHM workforce survey). Approximately half of them work at community hospitals, where PHM groups often comprise fewer than five full-time equivalents (FTEs) and face unique challenges. Community PHM practices may assume broader responsibilities than university/children’s hospital colleagues, including advocacy for the needs of children in predominantly adult-oriented hospitals.⁴ Although data regarding academic PHM work demands are available,⁵ there is little information pertaining to community hospitalists regarding typical workloads or other characteristics of thriving practices.

In this issue of the Journal of Hospital Medicine, Alvarez et al. present the findings of structured interviews with 70 community PHM group leaders.⁶ Each participant answered 12 questions about their group, addressing the definition of a full-time workload and hours, the design of backup systems, and the respondent’s perception of the program’s sustainability. The sample is robust, with the caveats that it disproportionately represents the Midwest and West (34.3% each) and more than half of the groups were employed by an academic institution. The authors found a median work expectation per FTE of 1,882 hours/year and 21 weekends per year, although they noted significant variability in employers’ demands and services provided. The majority of hospitalist groups lacked census caps, formal backup systems, or processes to expand coverage during busy seasons. Among the site leaders, 63% perceived their program as sustainable, but no program design or employer characteristic was clearly associated with this perception.

The importance of this study derives from aggregating data about the largest cross section of community PHM groups yet reported. For many PHM group leaders, this will offer a new point of reference for key practice characteristics. Furthermore, the authors should be commended for attempting to distinguish how program sustainability manifests in community PHM, where hospitalists shoulder longer patient care hours and many of them sustain academic endeavors. It is concerning that more than a third of leaders do not perceive their program as sustainable, but the implications for the field are unclear. Perhaps part of this uncertainty arises from the terminology, as sustainability lacks a technical or a consensus definition and the authors purposefully did not define the term for the respondents. While many respondents probably worried about physician burnout, others might have channeled fears about group finances or competition with adult service lines for beds. In addition, leaders’ fears about sustainability may not exactly represent the concerns of front-line employees.

Sustainable work environments are complex constructs with several inputs. For example, supportive leaders, efficient delivery systems, optimized EHRs, competitive pay, and confidence about service line stability might all mitigate higher workloads. Ultimately, this complexity underscores an important caution about all workplace surveys in medicine; ie, average values can inform practice design, but hospitalists and administrators should always consider the local context. Blindly applying medians as benchmarks and ignoring the myriad other contributors to sustainable practice risk disrupting successful PHM programs. In other words, surveys describe how the world is, not how it should be. The spectrum of academic work and norms permeating community PHM groups instead call for a nuanced approach.

How does the field build upon this useful paper? First, the Society of Hospital Medicine (SHM) should engage PHM leaders to increase participation in regular remeasurement, a critical endeavor for this dynamic field. SHM’s State of Hospital Medicine Report queries about a wider variety of practice characteristics, but it has a smaller sample size that must grow to fill this void.⁷ As the work of repeated surveys transitions from academic inquiry to professional society service, SHM’s Practice Analysis Committee can meet the needs of PHM through relevant questions and efforts to foster adequate participation. Second, all practice leaders should follow the ballooning bodies of literature about burnout and healthcare value. Just as labor leaders had discovered in the industrial revolution, sustainable careers require not only measuring work hours but also advocating for safe, meaningful, and engaging work conditions. By continuously creating value for patients, families, and hospitals, we can strengthen our claim to the resources needed to optimize the work environment.

Disclosure

Andrew White is Chair of the Society of Hospital Medicine’s Practice Analysis Committee, an unpaid position. Dr. Marek serves on the American Academy of Pediatrics Section on Hospital Medicine Executive Committee which is a voluntary, unpaid, elected position.

Systematic efforts to measure and compare work hours emerged in the 19^th century as laborers shifted from artisanal shops to factories, sparking debate over the appropriate length and intensity of work.¹ Two centuries of unionization and regulation defined work hours for many United States employees, including graduate medical trainees, but left attending physicians largely untouched. Instead, the medical workforce has long relied on survey data to shape jobs that balance professional norms with local market demands. Leaders in young, dynamic specialties, such as pediatric hospital medicine (PHM), particularly require such data to recruit and retain talent.

PHM progressed swiftly from acknowledgment as a “distinct area of practice” in 1999 to a subspecialty recognition.² Currently, at least 3,000 pediatric hospitalists³ practice in more than 800 US sites (Snow C, Personal communication regarding community PHM workforce survey). Approximately half of them work at community hospitals, where PHM groups often comprise fewer than five full-time equivalents (FTEs) and face unique challenges. Community PHM practices may assume broader responsibilities than university/children’s hospital colleagues, including advocacy for the needs of children in predominantly adult-oriented hospitals.⁴ Although data regarding academic PHM work demands are available,⁵ there is little information pertaining to community hospitalists regarding typical workloads or other characteristics of thriving practices.

In this issue of the Journal of Hospital Medicine, Alvarez et al. present the findings of structured interviews with 70 community PHM group leaders.⁶ Each participant answered 12 questions about their group, addressing the definition of a full-time workload and hours, the design of backup systems, and the respondent’s perception of the program’s sustainability. The sample is robust, with the caveats that it disproportionately represents the Midwest and West (34.3% each) and more than half of the groups were employed by an academic institution. The authors found a median work expectation per FTE of 1,882 hours/year and 21 weekends per year, although they noted significant variability in employers’ demands and services provided. The majority of hospitalist groups lacked census caps, formal backup systems, or processes to expand coverage during busy seasons. Among the site leaders, 63% perceived their program as sustainable, but no program design or employer characteristic was clearly associated with this perception.

The importance of this study derives from aggregating data about the largest cross section of community PHM groups yet reported. For many PHM group leaders, this will offer a new point of reference for key practice characteristics. Furthermore, the authors should be commended for attempting to distinguish how program sustainability manifests in community PHM, where hospitalists shoulder longer patient care hours and many of them sustain academic endeavors. It is concerning that more than a third of leaders do not perceive their program as sustainable, but the implications for the field are unclear. Perhaps part of this uncertainty arises from the terminology, as sustainability lacks a technical or a consensus definition and the authors purposefully did not define the term for the respondents. While many respondents probably worried about physician burnout, others might have channeled fears about group finances or competition with adult service lines for beds. In addition, leaders’ fears about sustainability may not exactly represent the concerns of front-line employees.

Sustainable work environments are complex constructs with several inputs. For example, supportive leaders, efficient delivery systems, optimized EHRs, competitive pay, and confidence about service line stability might all mitigate higher workloads. Ultimately, this complexity underscores an important caution about all workplace surveys in medicine; ie, average values can inform practice design, but hospitalists and administrators should always consider the local context. Blindly applying medians as benchmarks and ignoring the myriad other contributors to sustainable practice risk disrupting successful PHM programs. In other words, surveys describe how the world is, not how it should be. The spectrum of academic work and norms permeating community PHM groups instead call for a nuanced approach.

How does the field build upon this useful paper? First, the Society of Hospital Medicine (SHM) should engage PHM leaders to increase participation in regular remeasurement, a critical endeavor for this dynamic field. SHM’s State of Hospital Medicine Report queries about a wider variety of practice characteristics, but it has a smaller sample size that must grow to fill this void.⁷ As the work of repeated surveys transitions from academic inquiry to professional society service, SHM’s Practice Analysis Committee can meet the needs of PHM through relevant questions and efforts to foster adequate participation. Second, all practice leaders should follow the ballooning bodies of literature about burnout and healthcare value. Just as labor leaders had discovered in the industrial revolution, sustainable careers require not only measuring work hours but also advocating for safe, meaningful, and engaging work conditions. By continuously creating value for patients, families, and hospitals, we can strengthen our claim to the resources needed to optimize the work environment.

Disclosure

Andrew White is Chair of the Society of Hospital Medicine’s Practice Analysis Committee, an unpaid position. Dr. Marek serves on the American Academy of Pediatrics Section on Hospital Medicine Executive Committee which is a voluntary, unpaid, elected position.