The finding by Kachman et al. that consultations have decreased at their institution is an interesting and important observation.¹ In contrast, our study found that more than a third of hospitalists reported an increase in consultation requests.² There may be several explanations for this discrepancy. First, as Kachman et al. suggest, there may be differences between hospitalist perception and actual consultation use. Second, a significant variability in consultation may exist between hospitals. Although our study examined four institutions, we were unable to examine the variability between them, which requires further study. Third, there may be considerable variability between individual hospitalist practices, which is consistent with the findings reported by Kachman et al. Finally, the fact that our study examined only nonteaching services may be another explanation as Kachman et al. found that hospitalists on nonteaching services ordered more consultations than those on teaching services. These findings are consistent with a recent study conducted by Perez et al., who found that hospitalists on teaching services utilized fewer consultations and had lower direct care costs and shorter lengths of stay compared with those on nonteaching services.³ This finding raises the question of whether consultations impact care costs and lengths of stay, a topic that should be explored in future studies.

Disclosures

The authors report no conflicts of interest.

The finding by Kachman et al. that consultations have decreased at their institution is an interesting and important observation.¹ In contrast, our study found that more than a third of hospitalists reported an increase in consultation requests.² There may be several explanations for this discrepancy. First, as Kachman et al. suggest, there may be differences between hospitalist perception and actual consultation use. Second, a significant variability in consultation may exist between hospitals. Although our study examined four institutions, we were unable to examine the variability between them, which requires further study. Third, there may be considerable variability between individual hospitalist practices, which is consistent with the findings reported by Kachman et al. Finally, the fact that our study examined only nonteaching services may be another explanation as Kachman et al. found that hospitalists on nonteaching services ordered more consultations than those on teaching services. These findings are consistent with a recent study conducted by Perez et al., who found that hospitalists on teaching services utilized fewer consultations and had lower direct care costs and shorter lengths of stay compared with those on nonteaching services.³ This finding raises the question of whether consultations impact care costs and lengths of stay, a topic that should be explored in future studies.

Disclosures

The authors report no conflicts of interest.

The finding by Kachman et al. that consultations have decreased at their institution is an interesting and important observation.¹ In contrast, our study found that more than a third of hospitalists reported an increase in consultation requests.² There may be several explanations for this discrepancy. First, as Kachman et al. suggest, there may be differences between hospitalist perception and actual consultation use. Second, a significant variability in consultation may exist between hospitals. Although our study examined four institutions, we were unable to examine the variability between them, which requires further study. Third, there may be considerable variability between individual hospitalist practices, which is consistent with the findings reported by Kachman et al. Finally, the fact that our study examined only nonteaching services may be another explanation as Kachman et al. found that hospitalists on nonteaching services ordered more consultations than those on teaching services. These findings are consistent with a recent study conducted by Perez et al., who found that hospitalists on teaching services utilized fewer consultations and had lower direct care costs and shorter lengths of stay compared with those on nonteaching services.³ This finding raises the question of whether consultations impact care costs and lengths of stay, a topic that should be explored in future studies.

Disclosures

The authors report no conflicts of interest.

We thank Dr. Santhosh and colleagues for their letter concerning our article.¹ We agree that the diagnostic journey includes interactions both between and across teams, not just those within the patient’s team. In an article currently in press in Diagnosis, we examine how systems and cognitive factors interact during the process of diagnosis. Specifically, we reported on how communication between consultants can be both a barrier and facilitator to the diagnostic process.² We found that the frequency, quality, and pace of communication between and across inpatient teams and specialists are essential to timely diagnoses. As diagnostic errors remain a costly and morbid issue in the hospital setting, efforts to improve communication are clearly needed.³

Santhosh et al. raise an interesting point regarding cognitive load in evaluating diagnosis. Cognitive load is a multidimensional construct that represents the load that performing a specific task poses on a learner’s cognitive system.⁴ Components often used for measuring load include (a) task characteristics such as format, complexity, and time pressure; (b) subject characteristics such as expertise level, age, and spatial abilities; and (c) mental load and effort that originate from the interaction between task and subject characteristics.⁵ While there is little doubt that measuring these constructs has face value in diagnosis, we know of no instruments that are nimble, straightforward, or suitable for such measurement in the clinical setting. Furthermore, unlike handoffs (which lend themselves to structured frameworks), diagnostic evolution occurs across multiple individuals (from attendings to house staff and students), specialties (from emergency physicians to medical and surgical specialists), and over time. A unifying framework and tool to measure cognitive load across these elements would not only be novel, but a welcomed and much-needed component to facilitate diagnostic efforts. We hope that our ethnographic work will spur the development of these types of instruments and highlight opportunities for implementation. A future that both measures cognitive load and targets interventions to reduce or balance these across members of the diagnostic team would be welcomed.

Disclosures

The authors have nothing to disclose.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data.

We thank Dr. Santhosh and colleagues for their letter concerning our article.¹ We agree that the diagnostic journey includes interactions both between and across teams, not just those within the patient’s team. In an article currently in press in Diagnosis, we examine how systems and cognitive factors interact during the process of diagnosis. Specifically, we reported on how communication between consultants can be both a barrier and facilitator to the diagnostic process.² We found that the frequency, quality, and pace of communication between and across inpatient teams and specialists are essential to timely diagnoses. As diagnostic errors remain a costly and morbid issue in the hospital setting, efforts to improve communication are clearly needed.³

Santhosh et al. raise an interesting point regarding cognitive load in evaluating diagnosis. Cognitive load is a multidimensional construct that represents the load that performing a specific task poses on a learner’s cognitive system.⁴ Components often used for measuring load include (a) task characteristics such as format, complexity, and time pressure; (b) subject characteristics such as expertise level, age, and spatial abilities; and (c) mental load and effort that originate from the interaction between task and subject characteristics.⁵ While there is little doubt that measuring these constructs has face value in diagnosis, we know of no instruments that are nimble, straightforward, or suitable for such measurement in the clinical setting. Furthermore, unlike handoffs (which lend themselves to structured frameworks), diagnostic evolution occurs across multiple individuals (from attendings to house staff and students), specialties (from emergency physicians to medical and surgical specialists), and over time. A unifying framework and tool to measure cognitive load across these elements would not only be novel, but a welcomed and much-needed component to facilitate diagnostic efforts. We hope that our ethnographic work will spur the development of these types of instruments and highlight opportunities for implementation. A future that both measures cognitive load and targets interventions to reduce or balance these across members of the diagnostic team would be welcomed.

Disclosures

The authors have nothing to disclose.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data.

We thank Dr. Santhosh and colleagues for their letter concerning our article.¹ We agree that the diagnostic journey includes interactions both between and across teams, not just those within the patient’s team. In an article currently in press in Diagnosis, we examine how systems and cognitive factors interact during the process of diagnosis. Specifically, we reported on how communication between consultants can be both a barrier and facilitator to the diagnostic process.² We found that the frequency, quality, and pace of communication between and across inpatient teams and specialists are essential to timely diagnoses. As diagnostic errors remain a costly and morbid issue in the hospital setting, efforts to improve communication are clearly needed.³

Santhosh et al. raise an interesting point regarding cognitive load in evaluating diagnosis. Cognitive load is a multidimensional construct that represents the load that performing a specific task poses on a learner’s cognitive system.⁴ Components often used for measuring load include (a) task characteristics such as format, complexity, and time pressure; (b) subject characteristics such as expertise level, age, and spatial abilities; and (c) mental load and effort that originate from the interaction between task and subject characteristics.⁵ While there is little doubt that measuring these constructs has face value in diagnosis, we know of no instruments that are nimble, straightforward, or suitable for such measurement in the clinical setting. Furthermore, unlike handoffs (which lend themselves to structured frameworks), diagnostic evolution occurs across multiple individuals (from attendings to house staff and students), specialties (from emergency physicians to medical and surgical specialists), and over time. A unifying framework and tool to measure cognitive load across these elements would not only be novel, but a welcomed and much-needed component to facilitate diagnostic efforts. We hope that our ethnographic work will spur the development of these types of instruments and highlight opportunities for implementation. A future that both measures cognitive load and targets interventions to reduce or balance these across members of the diagnostic team would be welcomed.

Disclosures

The authors have nothing to disclose.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data.

We read the article by Chopra et al. “Focused Ethnography of Diagnosis in Academic Medical Centers” with great interest.¹ This ethnographic study provided valuable insights into possible interventions to encourage diagnostic thinking.

Duty hour regulations and the resulting increase in handoffs have shifted the social experience of diagnosis from one that occurs within teams to one that often occurs between teams during handoffs between providers.² While the article highlighted barriers to diagnosis, including distractions and time pressure, it did not explicitly discuss cognitive load theory. Cognitive load theory is an educational framework that has been described by Young et al.³ to improve instructions in the handoff process. These investigators showed how progressively experienced learners retain more information when using a structured scaffold or framework for information, such as the IPASS mnemonic,⁴ for example.

To mitigate the effects of distraction on the transfer of information, especially in cases with high diagnostic uncertainty, cognitive load must be explicitly considered. A structured framework for communication about diagnostic uncertainty informed by cognitive load theory would be a novel innovation that would help not only graduate medical education but could also improve diagnostic accuracy.

Disclosures

The authors have no conflicts of interest to disclose

We read the article by Chopra et al. “Focused Ethnography of Diagnosis in Academic Medical Centers” with great interest.¹ This ethnographic study provided valuable insights into possible interventions to encourage diagnostic thinking.

Duty hour regulations and the resulting increase in handoffs have shifted the social experience of diagnosis from one that occurs within teams to one that often occurs between teams during handoffs between providers.² While the article highlighted barriers to diagnosis, including distractions and time pressure, it did not explicitly discuss cognitive load theory. Cognitive load theory is an educational framework that has been described by Young et al.³ to improve instructions in the handoff process. These investigators showed how progressively experienced learners retain more information when using a structured scaffold or framework for information, such as the IPASS mnemonic,⁴ for example.

To mitigate the effects of distraction on the transfer of information, especially in cases with high diagnostic uncertainty, cognitive load must be explicitly considered. A structured framework for communication about diagnostic uncertainty informed by cognitive load theory would be a novel innovation that would help not only graduate medical education but could also improve diagnostic accuracy.

Disclosures

The authors have no conflicts of interest to disclose

We read the article by Chopra et al. “Focused Ethnography of Diagnosis in Academic Medical Centers” with great interest.¹ This ethnographic study provided valuable insights into possible interventions to encourage diagnostic thinking.

Duty hour regulations and the resulting increase in handoffs have shifted the social experience of diagnosis from one that occurs within teams to one that often occurs between teams during handoffs between providers.² While the article highlighted barriers to diagnosis, including distractions and time pressure, it did not explicitly discuss cognitive load theory. Cognitive load theory is an educational framework that has been described by Young et al.³ to improve instructions in the handoff process. These investigators showed how progressively experienced learners retain more information when using a structured scaffold or framework for information, such as the IPASS mnemonic,⁴ for example.

To mitigate the effects of distraction on the transfer of information, especially in cases with high diagnostic uncertainty, cognitive load must be explicitly considered. A structured framework for communication about diagnostic uncertainty informed by cognitive load theory would be a novel innovation that would help not only graduate medical education but could also improve diagnostic accuracy.

Disclosures

The authors have no conflicts of interest to disclose

We read with interest the article, “Hospitalist Perspective of Interactions with Medicine Subspecialty Consult Services.”¹ We applaud the authors for their work, but were surprised by the authors’ findings of hospitalist perceptions of consultation utilization. The authors reported that more hospitalists felt that their personal use of consultation was increasing (38.5%) versus those who reported that use was decreasing (30.3%).¹ The lack of true consensus on this issue may hint at significant variability in hospitalist use of inpatient consultation. We examined consultation use in 4,023 general medicine admissions to the University of Chicago from 2011 to 2015. Consultation use varied widely, with a 3.5-fold difference between the lowest and the highest quartiles of use (P < .01).² Contrary to the survey findings, we found that the number of consultations per admission actually decreased with each year in our sample.² In addition, a particularly interesting effect was observed in hospitalists; in multivariate regression, hospitalists on nonteaching services ordered more consultations than those on teaching services.² These findings suggest a gap between hospitalist self-reported perceptions of consultation use and actual use, which is important to understand, and highlight the need for further characterization of factors driving the use of this valuable resource.

Disclosures

The authors have no conflicts of interest to disclose.

We read with interest the article, “Hospitalist Perspective of Interactions with Medicine Subspecialty Consult Services.”¹ We applaud the authors for their work, but were surprised by the authors’ findings of hospitalist perceptions of consultation utilization. The authors reported that more hospitalists felt that their personal use of consultation was increasing (38.5%) versus those who reported that use was decreasing (30.3%).¹ The lack of true consensus on this issue may hint at significant variability in hospitalist use of inpatient consultation. We examined consultation use in 4,023 general medicine admissions to the University of Chicago from 2011 to 2015. Consultation use varied widely, with a 3.5-fold difference between the lowest and the highest quartiles of use (P < .01).² Contrary to the survey findings, we found that the number of consultations per admission actually decreased with each year in our sample.² In addition, a particularly interesting effect was observed in hospitalists; in multivariate regression, hospitalists on nonteaching services ordered more consultations than those on teaching services.² These findings suggest a gap between hospitalist self-reported perceptions of consultation use and actual use, which is important to understand, and highlight the need for further characterization of factors driving the use of this valuable resource.

Disclosures

The authors have no conflicts of interest to disclose.

We read with interest the article, “Hospitalist Perspective of Interactions with Medicine Subspecialty Consult Services.”¹ We applaud the authors for their work, but were surprised by the authors’ findings of hospitalist perceptions of consultation utilization. The authors reported that more hospitalists felt that their personal use of consultation was increasing (38.5%) versus those who reported that use was decreasing (30.3%).¹ The lack of true consensus on this issue may hint at significant variability in hospitalist use of inpatient consultation. We examined consultation use in 4,023 general medicine admissions to the University of Chicago from 2011 to 2015. Consultation use varied widely, with a 3.5-fold difference between the lowest and the highest quartiles of use (P < .01).² Contrary to the survey findings, we found that the number of consultations per admission actually decreased with each year in our sample.² In addition, a particularly interesting effect was observed in hospitalists; in multivariate regression, hospitalists on nonteaching services ordered more consultations than those on teaching services.² These findings suggest a gap between hospitalist self-reported perceptions of consultation use and actual use, which is important to understand, and highlight the need for further characterization of factors driving the use of this valuable resource.

Disclosures

The authors have no conflicts of interest to disclose.

Suicide is the second most common cause of death among children, adolescents, and young adults in the United States. In 2016, over 6,000 children and youth 5 to 24 years of age succumbed to suicide, thus reflecting a mortality rate nearly three times higher than deaths from malignancies and 28 times higher than deaths from sepsis in this age group.¹ Suicidal ideation and suicide attempts are even more common, with 17% of high school students reporting seriously considering suicide and 8% reporting suicide attempts in the previous 12 months.² These tragic statistics are reflected in our health system use, emergency department (ED) utilization for suicide attempts and suicidal ideation is growing at a tremendous rate, and over 50% of the children seen in EDs are subsequently admitted to the hospital for ongoing care.^3,4

In this issue of Journal of Hospital Medicine, Doupnik and colleagues present an analysis of pediatric hospitalizations for suicide attempts and suicidal ideation at acute care hospitals contained within the 2013 and 2014 National Readmissions Dataset.⁵ This dataset reflects a nationally representative sample of pediatric hospitalizations, weighted to allow for national estimates. Although their focus was on hospital readmission, their analysis yielded additional valuable data about suicide attempts and suicidal ideation in American youth. The investigators identified 181,575 pediatric acute care hospitalizations for suicide attempts and suicidal ideation over the two-year study period, accounting for 9.5% of all acute care hospitalizations among children and adolescents 6 to 17 years of age nationally. This number exceeds the biennial number of pediatric hospitalizations for cellulitis, dehydration, and urinary tract infections, all of which are generally considered the “bread and butter” of pediatric hospital medicine.⁶

Doupnik and colleagues rightly pointed out that hospital readmission is not a nationally endorsed measure to evaluate the quality of pediatric mental health hospitalizations. At the same time, their work highlights that acute care hospitals need strategies to measure the quality of pediatric hospitalizations for suicide attempts and suicidal ideation. Beyond readmissions, how should the quality of these hospital stays be evaluated? A recent review of 15 national quality measure sets identified 257 unique measures to evaluate pediatric quality of care.⁷ Of these, only one focused on mental health hospitalization. This measure, which was endorsed by the National Quality Forum, determines the percentage of discharges for patients six years of age and older who were hospitalized for mental health diagnoses and who had a follow-up visit with a mental health practitioner within 7 and 30 days of hospital discharge.⁸ Given Doupnik et al.’s finding that one-third of all 30-day hospital readmissions occurred within seven days of hospital discharge, early follow-up visits with mental health practitioners is arguably essential.

Although evidence-based quality measures to evaluate hospital-based mental healthcare are limited, quality measure development is ongoing, facilitated by recent federal health policy and associated research efforts. Four newly developed measures focus on the quality of inpatient care for suicidality, including two evaluated using data from health records and two derived from caregiver surveys. The first medical records-based measure identifies whether caregivers of patients admitted to hospital for dangerous self-harm or suicidality have documentation that they were counseled on how to restrict their child’s or adolescent’s access to potentially lethal means of suicide before discharge. The second record-based measure evaluates documentation in the medical record of discussion between the hospital provider and the patient’s outpatient provider regarding the plan for follow-up.⁹ The two survey-based measures ask caregivers whether they were counseled on how to restrict access to potentially lethal means of suicide, and, for children and adolescents started on a new antidepressant medication or dose, whether they were counseled regarding the potential benefits and risks of the medication.¹⁰ All measures were field-tested at children’s hospitals to ensure feasibility in data collection. However, as shown by Doupnik et al., only 7.4% of acute care hospitalizations for suicide attempts and suicidal ideation occurred at freestanding children’s hospitals; most occurred at urban nonteaching centers. Evaluation of these new quality measures across structurally diverse hospitals is an important next step.

Beyond the healthcare constructs evaluated by these quality measures, many foundational questions about what constitutes high quality inpatient healthcare for suicide attempts and suicidal ideation remain. An American Academy of Child and Adolescent Psychiatry (AACAP) practice parameter, which was published in 2001, established minimal standards for the assessment and treatment of children and adolescents with suicidal behavior.¹¹ This guideline recommends inpatient treatment until the mental state or level of suicidality has stabilized, with discharge considered only when the clinician is satisfied that adequate supervision and support will be available and when a responsible adult has agreed to secure or dispose of potentially lethal medications and firearms. It further recommends that the clinician treating the child or adolescent during the days following a suicide attempt be available to the patient and family – for example, to receive and make telephone calls outside of regular clinic hours. Recognizing the growing prevalence of suicidality in American children and youth, coupled with critical shortages in pediatric psychiatrists and fragmentation of inpatient and outpatient care, these minimal standards may be difficult to implement across the many settings where children receive their mental healthcare.^4,12,13

The large number of children and adolescents being hospitalized for suicide attempts and suicidal ideation at acute care hospitals demands that we take stock of how we manage this vulnerable population. Although Doupnik and colleagues suggest that exclusion of specialty psychiatric hospitals from their dataset is a limitation, their presentation of suicide attempts and suicidal ideation epidemiology at acute care hospitals provides valuable data for pediatric hospitalists. Given the presence of pediatric hospitalists at many acute care hospitals, comanagement by hospital medicine and psychiatry services may prove both efficient and effective while breaking down the silos that traditionally separate these specialties. Alternatively, extending the role of collaborative care teams, which are increasingly embedded in pediatric primary care, into inpatient settings may enable continuity of care and improve healthcare quality.¹⁴ Finally, nearly 20 years have passed since the AACAP published its practice parameter for the assessment and treatment of children and adolescents with suicidal behavior. An update to reflect contemporary suicide attempts and suicidal ideation statistics and evidence-based practices is needed, and collaboration between professional pediatric and psychiatric organizations in the creation of this update would recognize the growing role of pediatricians, including hospitalists, in the provision of mental healthcare for children.

Updated guidelines must take into account the transitions of care experienced by children and adolescents throughout their hospital stay: at admission, at discharge, and during their hospitalization if they move from medical to psychiatric care. Research is needed to determine what proportion of children and adolescents receive evidence-based mental health therapies while in hospital and how many are connected with wraparound mental health services before hospital discharge.¹⁵ Doupnik et al. excluded children and adolescents who were transferred to other hospitals, which included over 18,000 youth. How long did these patients spend “boarding,” and did they receive any mental health assessment or treatment during this period? Although the Joint Commission recommends that holding times for patients awaiting bed placement should not exceed four4 hours, hospitals have described average pediatric inpatient boarding times of 2-3 days while awaiting inpatient psychiatric care.^16,17 In one study of children and adolescents awaiting transfer for inpatient psychiatric care, mental health counseling was received by only 6%, which reflects lost time that could have been spent treating this highly vulnerable population.¹⁶ Multidisciplinary collaboration is needed to address these issues and inform best practices.

Although mortality is a rare outcome for most conditions we treat in pediatric hospital medicine, mortality following suicide attempts is all too common. The data presented by Doupnik and colleagues provide a powerful call to improve healthcare quality across the diverse settings where children with suicidality receive their care.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Funding

Dr. Leyenaar was supported by grant number K08HS024133 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ.

Suicide is the second most common cause of death among children, adolescents, and young adults in the United States. In 2016, over 6,000 children and youth 5 to 24 years of age succumbed to suicide, thus reflecting a mortality rate nearly three times higher than deaths from malignancies and 28 times higher than deaths from sepsis in this age group.¹ Suicidal ideation and suicide attempts are even more common, with 17% of high school students reporting seriously considering suicide and 8% reporting suicide attempts in the previous 12 months.² These tragic statistics are reflected in our health system use, emergency department (ED) utilization for suicide attempts and suicidal ideation is growing at a tremendous rate, and over 50% of the children seen in EDs are subsequently admitted to the hospital for ongoing care.^3,4

In this issue of Journal of Hospital Medicine, Doupnik and colleagues present an analysis of pediatric hospitalizations for suicide attempts and suicidal ideation at acute care hospitals contained within the 2013 and 2014 National Readmissions Dataset.⁵ This dataset reflects a nationally representative sample of pediatric hospitalizations, weighted to allow for national estimates. Although their focus was on hospital readmission, their analysis yielded additional valuable data about suicide attempts and suicidal ideation in American youth. The investigators identified 181,575 pediatric acute care hospitalizations for suicide attempts and suicidal ideation over the two-year study period, accounting for 9.5% of all acute care hospitalizations among children and adolescents 6 to 17 years of age nationally. This number exceeds the biennial number of pediatric hospitalizations for cellulitis, dehydration, and urinary tract infections, all of which are generally considered the “bread and butter” of pediatric hospital medicine.⁶

Doupnik and colleagues rightly pointed out that hospital readmission is not a nationally endorsed measure to evaluate the quality of pediatric mental health hospitalizations. At the same time, their work highlights that acute care hospitals need strategies to measure the quality of pediatric hospitalizations for suicide attempts and suicidal ideation. Beyond readmissions, how should the quality of these hospital stays be evaluated? A recent review of 15 national quality measure sets identified 257 unique measures to evaluate pediatric quality of care.⁷ Of these, only one focused on mental health hospitalization. This measure, which was endorsed by the National Quality Forum, determines the percentage of discharges for patients six years of age and older who were hospitalized for mental health diagnoses and who had a follow-up visit with a mental health practitioner within 7 and 30 days of hospital discharge.⁸ Given Doupnik et al.’s finding that one-third of all 30-day hospital readmissions occurred within seven days of hospital discharge, early follow-up visits with mental health practitioners is arguably essential.

Although evidence-based quality measures to evaluate hospital-based mental healthcare are limited, quality measure development is ongoing, facilitated by recent federal health policy and associated research efforts. Four newly developed measures focus on the quality of inpatient care for suicidality, including two evaluated using data from health records and two derived from caregiver surveys. The first medical records-based measure identifies whether caregivers of patients admitted to hospital for dangerous self-harm or suicidality have documentation that they were counseled on how to restrict their child’s or adolescent’s access to potentially lethal means of suicide before discharge. The second record-based measure evaluates documentation in the medical record of discussion between the hospital provider and the patient’s outpatient provider regarding the plan for follow-up.⁹ The two survey-based measures ask caregivers whether they were counseled on how to restrict access to potentially lethal means of suicide, and, for children and adolescents started on a new antidepressant medication or dose, whether they were counseled regarding the potential benefits and risks of the medication.¹⁰ All measures were field-tested at children’s hospitals to ensure feasibility in data collection. However, as shown by Doupnik et al., only 7.4% of acute care hospitalizations for suicide attempts and suicidal ideation occurred at freestanding children’s hospitals; most occurred at urban nonteaching centers. Evaluation of these new quality measures across structurally diverse hospitals is an important next step.

Beyond the healthcare constructs evaluated by these quality measures, many foundational questions about what constitutes high quality inpatient healthcare for suicide attempts and suicidal ideation remain. An American Academy of Child and Adolescent Psychiatry (AACAP) practice parameter, which was published in 2001, established minimal standards for the assessment and treatment of children and adolescents with suicidal behavior.¹¹ This guideline recommends inpatient treatment until the mental state or level of suicidality has stabilized, with discharge considered only when the clinician is satisfied that adequate supervision and support will be available and when a responsible adult has agreed to secure or dispose of potentially lethal medications and firearms. It further recommends that the clinician treating the child or adolescent during the days following a suicide attempt be available to the patient and family – for example, to receive and make telephone calls outside of regular clinic hours. Recognizing the growing prevalence of suicidality in American children and youth, coupled with critical shortages in pediatric psychiatrists and fragmentation of inpatient and outpatient care, these minimal standards may be difficult to implement across the many settings where children receive their mental healthcare.^4,12,13

The large number of children and adolescents being hospitalized for suicide attempts and suicidal ideation at acute care hospitals demands that we take stock of how we manage this vulnerable population. Although Doupnik and colleagues suggest that exclusion of specialty psychiatric hospitals from their dataset is a limitation, their presentation of suicide attempts and suicidal ideation epidemiology at acute care hospitals provides valuable data for pediatric hospitalists. Given the presence of pediatric hospitalists at many acute care hospitals, comanagement by hospital medicine and psychiatry services may prove both efficient and effective while breaking down the silos that traditionally separate these specialties. Alternatively, extending the role of collaborative care teams, which are increasingly embedded in pediatric primary care, into inpatient settings may enable continuity of care and improve healthcare quality.¹⁴ Finally, nearly 20 years have passed since the AACAP published its practice parameter for the assessment and treatment of children and adolescents with suicidal behavior. An update to reflect contemporary suicide attempts and suicidal ideation statistics and evidence-based practices is needed, and collaboration between professional pediatric and psychiatric organizations in the creation of this update would recognize the growing role of pediatricians, including hospitalists, in the provision of mental healthcare for children.

Updated guidelines must take into account the transitions of care experienced by children and adolescents throughout their hospital stay: at admission, at discharge, and during their hospitalization if they move from medical to psychiatric care. Research is needed to determine what proportion of children and adolescents receive evidence-based mental health therapies while in hospital and how many are connected with wraparound mental health services before hospital discharge.¹⁵ Doupnik et al. excluded children and adolescents who were transferred to other hospitals, which included over 18,000 youth. How long did these patients spend “boarding,” and did they receive any mental health assessment or treatment during this period? Although the Joint Commission recommends that holding times for patients awaiting bed placement should not exceed four4 hours, hospitals have described average pediatric inpatient boarding times of 2-3 days while awaiting inpatient psychiatric care.^16,17 In one study of children and adolescents awaiting transfer for inpatient psychiatric care, mental health counseling was received by only 6%, which reflects lost time that could have been spent treating this highly vulnerable population.¹⁶ Multidisciplinary collaboration is needed to address these issues and inform best practices.

Although mortality is a rare outcome for most conditions we treat in pediatric hospital medicine, mortality following suicide attempts is all too common. The data presented by Doupnik and colleagues provide a powerful call to improve healthcare quality across the diverse settings where children with suicidality receive their care.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Funding

Dr. Leyenaar was supported by grant number K08HS024133 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ.

Suicide is the second most common cause of death among children, adolescents, and young adults in the United States. In 2016, over 6,000 children and youth 5 to 24 years of age succumbed to suicide, thus reflecting a mortality rate nearly three times higher than deaths from malignancies and 28 times higher than deaths from sepsis in this age group.¹ Suicidal ideation and suicide attempts are even more common, with 17% of high school students reporting seriously considering suicide and 8% reporting suicide attempts in the previous 12 months.² These tragic statistics are reflected in our health system use, emergency department (ED) utilization for suicide attempts and suicidal ideation is growing at a tremendous rate, and over 50% of the children seen in EDs are subsequently admitted to the hospital for ongoing care.^3,4

In this issue of Journal of Hospital Medicine, Doupnik and colleagues present an analysis of pediatric hospitalizations for suicide attempts and suicidal ideation at acute care hospitals contained within the 2013 and 2014 National Readmissions Dataset.⁵ This dataset reflects a nationally representative sample of pediatric hospitalizations, weighted to allow for national estimates. Although their focus was on hospital readmission, their analysis yielded additional valuable data about suicide attempts and suicidal ideation in American youth. The investigators identified 181,575 pediatric acute care hospitalizations for suicide attempts and suicidal ideation over the two-year study period, accounting for 9.5% of all acute care hospitalizations among children and adolescents 6 to 17 years of age nationally. This number exceeds the biennial number of pediatric hospitalizations for cellulitis, dehydration, and urinary tract infections, all of which are generally considered the “bread and butter” of pediatric hospital medicine.⁶

Doupnik and colleagues rightly pointed out that hospital readmission is not a nationally endorsed measure to evaluate the quality of pediatric mental health hospitalizations. At the same time, their work highlights that acute care hospitals need strategies to measure the quality of pediatric hospitalizations for suicide attempts and suicidal ideation. Beyond readmissions, how should the quality of these hospital stays be evaluated? A recent review of 15 national quality measure sets identified 257 unique measures to evaluate pediatric quality of care.⁷ Of these, only one focused on mental health hospitalization. This measure, which was endorsed by the National Quality Forum, determines the percentage of discharges for patients six years of age and older who were hospitalized for mental health diagnoses and who had a follow-up visit with a mental health practitioner within 7 and 30 days of hospital discharge.⁸ Given Doupnik et al.’s finding that one-third of all 30-day hospital readmissions occurred within seven days of hospital discharge, early follow-up visits with mental health practitioners is arguably essential.

Although evidence-based quality measures to evaluate hospital-based mental healthcare are limited, quality measure development is ongoing, facilitated by recent federal health policy and associated research efforts. Four newly developed measures focus on the quality of inpatient care for suicidality, including two evaluated using data from health records and two derived from caregiver surveys. The first medical records-based measure identifies whether caregivers of patients admitted to hospital for dangerous self-harm or suicidality have documentation that they were counseled on how to restrict their child’s or adolescent’s access to potentially lethal means of suicide before discharge. The second record-based measure evaluates documentation in the medical record of discussion between the hospital provider and the patient’s outpatient provider regarding the plan for follow-up.⁹ The two survey-based measures ask caregivers whether they were counseled on how to restrict access to potentially lethal means of suicide, and, for children and adolescents started on a new antidepressant medication or dose, whether they were counseled regarding the potential benefits and risks of the medication.¹⁰ All measures were field-tested at children’s hospitals to ensure feasibility in data collection. However, as shown by Doupnik et al., only 7.4% of acute care hospitalizations for suicide attempts and suicidal ideation occurred at freestanding children’s hospitals; most occurred at urban nonteaching centers. Evaluation of these new quality measures across structurally diverse hospitals is an important next step.

Beyond the healthcare constructs evaluated by these quality measures, many foundational questions about what constitutes high quality inpatient healthcare for suicide attempts and suicidal ideation remain. An American Academy of Child and Adolescent Psychiatry (AACAP) practice parameter, which was published in 2001, established minimal standards for the assessment and treatment of children and adolescents with suicidal behavior.¹¹ This guideline recommends inpatient treatment until the mental state or level of suicidality has stabilized, with discharge considered only when the clinician is satisfied that adequate supervision and support will be available and when a responsible adult has agreed to secure or dispose of potentially lethal medications and firearms. It further recommends that the clinician treating the child or adolescent during the days following a suicide attempt be available to the patient and family – for example, to receive and make telephone calls outside of regular clinic hours. Recognizing the growing prevalence of suicidality in American children and youth, coupled with critical shortages in pediatric psychiatrists and fragmentation of inpatient and outpatient care, these minimal standards may be difficult to implement across the many settings where children receive their mental healthcare.^4,12,13

The large number of children and adolescents being hospitalized for suicide attempts and suicidal ideation at acute care hospitals demands that we take stock of how we manage this vulnerable population. Although Doupnik and colleagues suggest that exclusion of specialty psychiatric hospitals from their dataset is a limitation, their presentation of suicide attempts and suicidal ideation epidemiology at acute care hospitals provides valuable data for pediatric hospitalists. Given the presence of pediatric hospitalists at many acute care hospitals, comanagement by hospital medicine and psychiatry services may prove both efficient and effective while breaking down the silos that traditionally separate these specialties. Alternatively, extending the role of collaborative care teams, which are increasingly embedded in pediatric primary care, into inpatient settings may enable continuity of care and improve healthcare quality.¹⁴ Finally, nearly 20 years have passed since the AACAP published its practice parameter for the assessment and treatment of children and adolescents with suicidal behavior. An update to reflect contemporary suicide attempts and suicidal ideation statistics and evidence-based practices is needed, and collaboration between professional pediatric and psychiatric organizations in the creation of this update would recognize the growing role of pediatricians, including hospitalists, in the provision of mental healthcare for children.

Updated guidelines must take into account the transitions of care experienced by children and adolescents throughout their hospital stay: at admission, at discharge, and during their hospitalization if they move from medical to psychiatric care. Research is needed to determine what proportion of children and adolescents receive evidence-based mental health therapies while in hospital and how many are connected with wraparound mental health services before hospital discharge.¹⁵ Doupnik et al. excluded children and adolescents who were transferred to other hospitals, which included over 18,000 youth. How long did these patients spend “boarding,” and did they receive any mental health assessment or treatment during this period? Although the Joint Commission recommends that holding times for patients awaiting bed placement should not exceed four4 hours, hospitals have described average pediatric inpatient boarding times of 2-3 days while awaiting inpatient psychiatric care.^16,17 In one study of children and adolescents awaiting transfer for inpatient psychiatric care, mental health counseling was received by only 6%, which reflects lost time that could have been spent treating this highly vulnerable population.¹⁶ Multidisciplinary collaboration is needed to address these issues and inform best practices.

Although mortality is a rare outcome for most conditions we treat in pediatric hospital medicine, mortality following suicide attempts is all too common. The data presented by Doupnik and colleagues provide a powerful call to improve healthcare quality across the diverse settings where children with suicidality receive their care.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Funding

Dr. Leyenaar was supported by grant number K08HS024133 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ.

Suicide is a leading cause of death among 10- to 34-year-olds in the United States.^1,2 During the past two decades, the youth suicide death rate has risen by 24%, and more than 5,000 young people die from suicide each year.³ Suicide ideation (SI) and suicide attempts (SAs) are well-established risk factors for suicide death and a source of morbidity for patients and families. One-third of youth with SI attempt suicide at some point in their lifetime.⁴ Approximately 11% of youth SAs result in suicide death, and 2% of youth who attempt suicide subsequently go on to die from suicide after recovering from a prior SA.⁵ More than 60,000 youth are hospitalized for SI or SA each year,⁶ and young people hospitalized for SA are at high short-term risk of repeat SA and suicide death.⁷ Hospitals need strategies for measuring the quality of SI and SA hospitalizations, monitoring postdischarge outcomes, and identifying the patients at the highest risk of poor outcomes. Readmissions are a useful hospital quality measure that can indicate re-emergence of SI, repeat SA, or inadequate community-based mental health treatment, and interventions designed for patients with readmissions can potentially avert morbidity or mortality.

The National Committee on Quality Assurance recommends measurement of quality metrics for 30-day mental health follow-up after psychiatric hospitalizations, 30-day readmissions after adult (but not pediatric) psychiatric hospitalizations, and 30-day readmissions in pediatric medical and surgical hospitalizations. Readmission measures are not consistently used to evaluate pediatric psychiatric hospitalizations, and psychiatric quality measures are not used to evaluate medical or surgical hospitalizations for SA. Recent research has investigated transfers to postacute care,⁸ readmission prevalence, variation in hospital readmission performance, and risk factors for readmissions after pediatric psychiatric hospitalizations.^9–11 However, no national study has investigated 30-day readmissions in youth hospitalized specifically for SI or SA.

To inform hospital quality measurement and improve hospital and postdischarge care for youth at risk of suicide, more information is needed about the characteristics of and the risk factors for readmissions after index SI/SA hospitalization. To address this knowledge gap, among SI/SA hospitalizations in 6- to 17-year-olds, we examined (1) unplanned 30-day readmissions and characteristics of hospitalizations by 30-day readmission status; (2) patient, hospital, and regional characteristics associated with 30-day readmissions; and (3) characteristics of 30-day readmissions.

Study Design and Data Source

We conducted a national, retrospective cohort study of hospitalizations for patients aged 6-17 years using the Agency for Healthcare Research and Quality (AHRQ) 2013 and 2014 Nationwide Readmissions Database (NRD). The combined 2013-2014 NRD includes administrative data from a nationally representative sample of 29 million hospitalizations in 22 states, accounting for 49.3% of all US hospitalizations, and is weighted for national projections. The NRD includes hospital information, patient demographic information, and International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis, procedure, and external cause of injury codes (E-codes). The database includes one primary diagnosis, up to 24 additional diagnoses, and up to 4 E-codes for each hospitalization. The NRD includes information about hospitalizations in acute-care general hospitals (including their psychiatric units) but not from specialty psychiatric hospitals. The database also includes de-identified, verified patient linkage numbers so that patients can be tracked across multiple hospitalizations at the same institution or different institutions within the same state. This study was considered to be exempt from review by the Children’s Hospital of Philadelphia Institutional Review Board.

Sample

We identified a sample of 181,575 hospitalizations for SI (n = 119,037) or SA (n = 62,538) among 6- to 17-year-olds between January 1, 2013, and December 31, 2014 (Figure). We included children as young as 6 years because validated methods exist to identify SI and SA in this age group,¹² and because suicide deaths have recently increased among younger children.³ We excluded patients aged 18 years and older from this study since delivery of mental health services differs for adults.¹³ To create the sample, we first identified all hospitalizations of patients aged 6-17 years. We then used a validated algorithm relying on ICD-9-CM diagnosis codes for poisonings and E-codes for self-injury (E950-959) to identify hospitalizations related to SA.¹² Because E-code completeness varies among states,¹⁴ we also used the combination of having both a diagnosis code for injury (800-999) and an ICD-9-CM code for SI (V62.84) as a proxy for SA. Among hospitalizations without SA, we identified hospitalizations with SI using the ICD-9-CM code for SI (V62.84) in any position.

We identified 133,516 index hospitalizations with complete data at risk for an unplanned readmission. Because NRD data cannot be linked between calendar years, we limited the study time period for each calendar year to 10 months. We excluded hospitalizations in January because the full 30-day time frame to determine whether a hospitalization had occurred in the preceding 30 days, a known risk factor for readmissions in other samples,¹⁵ was not available. We excluded index hospitalizations occurring in December, because the full 30-day time frame to ascertain readmissions was not available. We excluded hospitalizations resulting in death, since these are not at risk for readmission, and hospitalizations resulting in transfer, since the timing of discharge to the community was not known. Readmission hospitalizations were eligible to be included as index hospitalizations if they met sample inclusion criteria. The final sample for readmission analyses included 95,354 SI hospitalizations and 38,162 SA hospitalizations (Figure).

Primary Outcome

The primary outcome was any unplanned, all-cause readmission within 30 days of index hospitalization for SI or SA. Among 30-day readmissions, we examined readmission timing, whether the readmission was to the same hospital or a different hospital, length of stay, and indication for readmission (medical/surgical or psychiatric, and presence of SI or SA diagnoses). Planned readmissions were identified using measure specifications endorsed by the AHRQ and the National Quality Forum¹⁶ and excluded from measurement.

Among index hospitalizations for SI, we specifically examined 30-day readmissions for subsequent SA, since one objective of hospitalization for SI is to prevent progression to SA or death. We could not identify hospitalizations for repeat SA after index hospitalization for SA, because diagnosis codes did not differentiate between readmission for complications of index SA and readmission for repeat SA.

Independent Variables

We analyzed demographic, clinical, and hospital factors associated with readmissions in other samples.^17–20 Demographic characteristics included patient gender and age, urban or rural residence, payer, and median national income quartile for a patient’s ZIP code. Race and ethnicity data are not available in the NRD.

Clinical characteristics included hospitalization in the 30-days preceding the index hospitalization, index hospitalization length of stay, and admission via the emergency department (ED) versus direct admission. A patient’s chronic condition profile was determined using index hospitalization diagnosis codes. Complex chronic conditions (eg, cancer, cystic fibrosis) were identified using a classification system used in several prior studies of hospital administrative datasets,²¹ and other noncomplex chronic medical conditions (eg, asthma, obesity) were identified using the Healthcare Cost and Utilization Project (HCUP) chronic condition indicator system.²² Psychiatric conditions (eg, anxiety disorders, substance abuse, autism) were identified and categorized using a classification system used in studies of hospital administrative datasets.²³ The number of psychiatric conditions was determined by counting the number of psychiatric condition categories in which a patient had a diagnosis. SA was categorized as having lower risk of death (eg, medication overdose, injury from cutting or piercing) or higher risk of death (eg, hanging, suffocation, or firearm injury).²⁴

Because of known temporal trends in SI and SA,^25,26 the month and year of admission were included as covariates. Hospital characteristics included teaching hospital and children’s hospital designations.

Statistical Analysis

We compared descriptive, summary statistics for characteristics of index hospitalizations with and without a 30-day readmission using Rao-Scott chi-square tests. In multivariable analyses, we derived logistic regression models to measure the associations of patient, hospital, and temporal factors with 30-day hospital readmissions. Analyses were conducted in SAS PROC SURVEYLOGISTIC and were weighted to achieve national estimates, clustered by sample stratum and hospital to account for the complex survey design,²⁷ clustered by patients to account for multiple index visits per patient, and adjusted for clinical, demographic, and hospital characteristics. SAS version 9.4 (SAS Institute, Cary, North Carolina) was used for all analyses. All tests were two-sided, and a P value <.05 was considered as statistically significant.

Sample Characteristics

In the weighted analyses, we identified 133,516 hospitalizations in acute-care hospitals for SI or SA, and 8.5% (n = 11,375) of hospitalizations had at least one unplanned 30-day readmission to an acute-care hospital. Unweighted, the sample included 37,683 patients and 42,198 hospitalizations. Among all patients represented in the sample, 90.5% had only a single SI or SA hospitalization, 7.7% had two hospitalizations, and 1.8% had >2 hospitalizations in one year.

Table 1 summarizes the sample characteristics and displays the demographic, clinical, and hospital characteristics of index hospitalizations by the 30-day readmission status. Patients represented in the index hospitalizations were 64.9% female, 3.6% were aged 6-9 years, 40.1% were aged 10-14 years, and 56.3% were aged 15-17 years. Nearly half of the patients (49.1%) used public insurance. Nearly half (44.9%) lived in metropolitan areas with >1 million residents, 36.1% lived in metropolitan areas with 50,000 to 1 million residents, and 14.7% lived in rural areas.

Association of Patient and Hospital Characteristics with Readmissions

Table 2 displays the patient and hospital characteristics associated with readmissions. Among demographic characteristics, 10- to 14-year-old patients had higher odds of readmission (odds ratio [OR]: 1.18, 95% confidence interval [CI]: 1.07-1.29) than 15- to 17-year-old patients. Having public insurance was associated with higher odds of readmission (OR: 1.14, 95% CI: 1.04-1.25). We found no differences in readmission rates based on sex, urban or rural location, or patient’s ZIP code income quartile.

Among clinical characteristics, hospitalizations with an admission for SI or SA in the preceding 30 days, meaning that the index hospitalization itself was a readmission, had the strongest association with readmissions (OR: 3.14, 95% CI: 2.73-3.61). In addition, patients admitted via the ED for the index hospitalization had higher odds of readmission (OR: 1.25, 95% CI: 1.15-1.36). Chronic psychiatric conditions associated with higher odds of readmission included psychotic disorders (OR: 1.39, 95% CI: 1.16-1.67) and bipolar disorder (OR: 1.27, 95% CI: 1.13-1.44).

Characteristics of 30-day Readmissions

Table 3 displays the characteristics of readmissions after SI compared to that after SA. Among the combined sample of 11,375 30-day readmissions, 34.1% occurred within 7 days, and 65.9% in 8-30 days. Eleven percent of patients with any readmission had more than one readmission within 30 days. Among readmissions, 94.5% were for a psychiatric problem and 5.5% were for a medical or surgical problem. A total of 43.9% had a diagnosis of SI and 19.5% a diagnosis of SA. Readmissions were more likely to occur at a different hospital after SI than after SA (48.1% vs. 31.3%, P < .001). Medical and surgical indications for readmission were less common after SI than after SA (4.4% vs. 8.7%, P < .001). Only 1.2% of SI hospitalizations had a readmission for SA within 30 days. Of these cases, 55.6% were aged 15-17 years, 43.3% were aged 10-14 years, and 1.1% were aged 6-9 years; 73.1% of the patients were female, and 49.1% used public insurance.

SI and SA in children and adolescents are substantial public health problems associated with significant hospital resource utilization. In 2013 and 2014, there were 181,575 pediatric acute-care hospitalizations for SI or SA, accounting for 9.5% of all hospitalizations in 6- to 17-year-old patients nationally. Among acute-care SI and SA hospitalizations, 8.5% had a readmission to an acute-care hospital within 30 days. The study data source did not include psychiatric specialty hospitals, and the number of index hospitalizations is likely substantially higher when psychiatric specialty hospitalizations are included. Readmissions may also be higher if patients were readmitted to psychiatric specialty hospitals after discharge from acute-care hospitals. The strongest risk factor for unplanned 30-day readmissions was a previous hospitalization in the 30 days before the index admission, likely a marker for severity or complexity of psychiatric illness. Other characteristics associated with higher odds of readmission were bipolar disorder, psychotic disorders, and age 10-14 years. More than one-third of readmissions occurred within the first 7 days after hospital discharge. The prevalence of SI and SA hospitalizations and readmissions was similar to findings in previous analyses of mental health hospitalizations.^10,28

A patient’s psychiatric illness type and severity, as evidenced by the need for frequent repeat hospitalizations, was highly associated with the risk of 30-day readmission. Any hospitalization in the 30 days preceding the index hospitalization, whether for SI/SA or for another problem, was a strong risk factor for readmissions. We suspect that prior SI/SA hospitalizations reflect a patient’s chronic elevated risk for suicide. Prior hospitalizations not for SI or SA could be hospitalizations for mental illness exacerbations that increase the risk of SI or SA, eg, bipolar disorder with acute mania, or they could represent physical health problems. Chronic physical health problems are a known risk factor for SI and SA.²⁹

A knowledge of those characteristics that increase the readmission risk can inform future resource allocation, research, and policy in several ways. First, longer hospital stays could mitigate readmission risk in some patients with severe psychiatric illness. European studies in older adolescents and adults show that for severe psychiatric illness, a longer hospital stay is associated with a lower risk of hospital readmission.^15,30 Second, better access to intensive community-based mental health (MH) services, including evidence-based psychotherapy and medication management, improves symptoms in young people.³¹ Access to these services likely affects the risk of hospital readmission. We found that readmission risk was highest in 10- to 14-year-olds. Taken in the context of existing evidence that suicide rates are rising in younger patients,^1,3 our findings suggest that particular attention to community services for younger patients is needed. Third, care coordination could help patients access beneficial services to reduce readmissions and improve other outcomes. Enhanced discharge care coordination reduced suicide deaths in high-risk populations in Europe³² and Japan³³ and improved attendance at mental health follow-up after pediatric ED discharge in a small United States sample.³⁴ Given that one-third of readmissions occurred within seven days, care coordination designed to ensure access to ambulatory services in the immediate postdischarge period may be particularly beneficial.

We found that ZIP code income quartile was not associated with readmissions. We suspect that poverty is not as closely correlated with MH hospitalization outcomes as it is with physical health hospitalization outcomes for several reasons. Medicaid insurance historically has more robust coverage of mental health services than some private insurance plans, which might offset some of the risk of poor mental health outcomes associated with poverty. Low-income families are eligible to use social services, and families accessing social services might have more opportunities to become familiar with community mental health programs. Further, the expectation of high achievement found in some higher income families is associated with MH problems in children and adolescents.³⁵ Therefore, being in a higher income quartile might not be as protective against poor mental health outcomes as it is against poor physical health outcomes.

Although the NRD provides a rich source of readmission data across hospitals nationally, several limitations are inherent to this administrative dataset. First, data from specialty psychiatric hospitals were not included in the NRD. The study underestimates the total number of index hospitalizations and readmissions, since index SI/SA hospitalizations at psychiatric hospitals are not included, and readmissions are not included if they occurred at specialty psychiatric hospitals. Second, because data cannot be linked between calendar years, we excluded January and December hospitalizations, and findings might not generalize to hospitalizations in January and December. Seasonal trends in SI/SA hospitalizations are known to occur.³⁶ Third, race, ethnicity, primary language, gender identity, and sexual orientation are not available in the NRD, and we could not examine the association of these characteristics with the likelihood of readmissions. Fourth, we did not have information about pre- or posthospitalization insurance enrollment or outpatient services that could affect the risk of readmission. Nevertheless, this study offers information on the characteristics of readmissions after hospitalizations for SI and SA in a large nationally representative sample of youth, and the findings can inform resource planning to prevent suicides.

Hospital readmissions are common in patients with SI and SA, and patients with a recent previous hospitalization have the highest risk of readmission. More than one-third of readmissions after SI or SA occurred within the first seven days. Due to the dearth of mental health services in the community, hospitals offer an important safety net for youth experiencing acute suicidal crises. Strategies to improve the continuum of care for patients at risk of suicide that solely focus on reducing readmissions are not likely to benefit patients. However, readmissions can identify opportunities for improving hospital discharge processes and outpatient services. Future research and clinical innovation to investigate and improve hospital discharge planning and access to community mental health services is likely to benefit patients and could reduce 30-day hospital readmissions.

Acknowledgments

The authors thank John Lawlor for his assistance with the analysis.

Disclosures

The authors have no potential conflicts of interest to disclose.

Funding

Dr. Zima received funding from the Behavioral Health Centers of Excellence for California (SB852).

Suicide is a leading cause of death among 10- to 34-year-olds in the United States.^1,2 During the past two decades, the youth suicide death rate has risen by 24%, and more than 5,000 young people die from suicide each year.³ Suicide ideation (SI) and suicide attempts (SAs) are well-established risk factors for suicide death and a source of morbidity for patients and families. One-third of youth with SI attempt suicide at some point in their lifetime.⁴ Approximately 11% of youth SAs result in suicide death, and 2% of youth who attempt suicide subsequently go on to die from suicide after recovering from a prior SA.⁵ More than 60,000 youth are hospitalized for SI or SA each year,⁶ and young people hospitalized for SA are at high short-term risk of repeat SA and suicide death.⁷ Hospitals need strategies for measuring the quality of SI and SA hospitalizations, monitoring postdischarge outcomes, and identifying the patients at the highest risk of poor outcomes. Readmissions are a useful hospital quality measure that can indicate re-emergence of SI, repeat SA, or inadequate community-based mental health treatment, and interventions designed for patients with readmissions can potentially avert morbidity or mortality.

The National Committee on Quality Assurance recommends measurement of quality metrics for 30-day mental health follow-up after psychiatric hospitalizations, 30-day readmissions after adult (but not pediatric) psychiatric hospitalizations, and 30-day readmissions in pediatric medical and surgical hospitalizations. Readmission measures are not consistently used to evaluate pediatric psychiatric hospitalizations, and psychiatric quality measures are not used to evaluate medical or surgical hospitalizations for SA. Recent research has investigated transfers to postacute care,⁸ readmission prevalence, variation in hospital readmission performance, and risk factors for readmissions after pediatric psychiatric hospitalizations.^9–11 However, no national study has investigated 30-day readmissions in youth hospitalized specifically for SI or SA.

To inform hospital quality measurement and improve hospital and postdischarge care for youth at risk of suicide, more information is needed about the characteristics of and the risk factors for readmissions after index SI/SA hospitalization. To address this knowledge gap, among SI/SA hospitalizations in 6- to 17-year-olds, we examined (1) unplanned 30-day readmissions and characteristics of hospitalizations by 30-day readmission status; (2) patient, hospital, and regional characteristics associated with 30-day readmissions; and (3) characteristics of 30-day readmissions.

Study Design and Data Source

We conducted a national, retrospective cohort study of hospitalizations for patients aged 6-17 years using the Agency for Healthcare Research and Quality (AHRQ) 2013 and 2014 Nationwide Readmissions Database (NRD). The combined 2013-2014 NRD includes administrative data from a nationally representative sample of 29 million hospitalizations in 22 states, accounting for 49.3% of all US hospitalizations, and is weighted for national projections. The NRD includes hospital information, patient demographic information, and International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis, procedure, and external cause of injury codes (E-codes). The database includes one primary diagnosis, up to 24 additional diagnoses, and up to 4 E-codes for each hospitalization. The NRD includes information about hospitalizations in acute-care general hospitals (including their psychiatric units) but not from specialty psychiatric hospitals. The database also includes de-identified, verified patient linkage numbers so that patients can be tracked across multiple hospitalizations at the same institution or different institutions within the same state. This study was considered to be exempt from review by the Children’s Hospital of Philadelphia Institutional Review Board.

Sample

We identified a sample of 181,575 hospitalizations for SI (n = 119,037) or SA (n = 62,538) among 6- to 17-year-olds between January 1, 2013, and December 31, 2014 (Figure). We included children as young as 6 years because validated methods exist to identify SI and SA in this age group,¹² and because suicide deaths have recently increased among younger children.³ We excluded patients aged 18 years and older from this study since delivery of mental health services differs for adults.¹³ To create the sample, we first identified all hospitalizations of patients aged 6-17 years. We then used a validated algorithm relying on ICD-9-CM diagnosis codes for poisonings and E-codes for self-injury (E950-959) to identify hospitalizations related to SA.¹² Because E-code completeness varies among states,¹⁴ we also used the combination of having both a diagnosis code for injury (800-999) and an ICD-9-CM code for SI (V62.84) as a proxy for SA. Among hospitalizations without SA, we identified hospitalizations with SI using the ICD-9-CM code for SI (V62.84) in any position.

We identified 133,516 index hospitalizations with complete data at risk for an unplanned readmission. Because NRD data cannot be linked between calendar years, we limited the study time period for each calendar year to 10 months. We excluded hospitalizations in January because the full 30-day time frame to determine whether a hospitalization had occurred in the preceding 30 days, a known risk factor for readmissions in other samples,¹⁵ was not available. We excluded index hospitalizations occurring in December, because the full 30-day time frame to ascertain readmissions was not available. We excluded hospitalizations resulting in death, since these are not at risk for readmission, and hospitalizations resulting in transfer, since the timing of discharge to the community was not known. Readmission hospitalizations were eligible to be included as index hospitalizations if they met sample inclusion criteria. The final sample for readmission analyses included 95,354 SI hospitalizations and 38,162 SA hospitalizations (Figure).

Primary Outcome

The primary outcome was any unplanned, all-cause readmission within 30 days of index hospitalization for SI or SA. Among 30-day readmissions, we examined readmission timing, whether the readmission was to the same hospital or a different hospital, length of stay, and indication for readmission (medical/surgical or psychiatric, and presence of SI or SA diagnoses). Planned readmissions were identified using measure specifications endorsed by the AHRQ and the National Quality Forum¹⁶ and excluded from measurement.

Among index hospitalizations for SI, we specifically examined 30-day readmissions for subsequent SA, since one objective of hospitalization for SI is to prevent progression to SA or death. We could not identify hospitalizations for repeat SA after index hospitalization for SA, because diagnosis codes did not differentiate between readmission for complications of index SA and readmission for repeat SA.

Independent Variables

We analyzed demographic, clinical, and hospital factors associated with readmissions in other samples.^17–20 Demographic characteristics included patient gender and age, urban or rural residence, payer, and median national income quartile for a patient’s ZIP code. Race and ethnicity data are not available in the NRD.

Clinical characteristics included hospitalization in the 30-days preceding the index hospitalization, index hospitalization length of stay, and admission via the emergency department (ED) versus direct admission. A patient’s chronic condition profile was determined using index hospitalization diagnosis codes. Complex chronic conditions (eg, cancer, cystic fibrosis) were identified using a classification system used in several prior studies of hospital administrative datasets,²¹ and other noncomplex chronic medical conditions (eg, asthma, obesity) were identified using the Healthcare Cost and Utilization Project (HCUP) chronic condition indicator system.²² Psychiatric conditions (eg, anxiety disorders, substance abuse, autism) were identified and categorized using a classification system used in studies of hospital administrative datasets.²³ The number of psychiatric conditions was determined by counting the number of psychiatric condition categories in which a patient had a diagnosis. SA was categorized as having lower risk of death (eg, medication overdose, injury from cutting or piercing) or higher risk of death (eg, hanging, suffocation, or firearm injury).²⁴

Because of known temporal trends in SI and SA,^25,26 the month and year of admission were included as covariates. Hospital characteristics included teaching hospital and children’s hospital designations.

Statistical Analysis

We compared descriptive, summary statistics for characteristics of index hospitalizations with and without a 30-day readmission using Rao-Scott chi-square tests. In multivariable analyses, we derived logistic regression models to measure the associations of patient, hospital, and temporal factors with 30-day hospital readmissions. Analyses were conducted in SAS PROC SURVEYLOGISTIC and were weighted to achieve national estimates, clustered by sample stratum and hospital to account for the complex survey design,²⁷ clustered by patients to account for multiple index visits per patient, and adjusted for clinical, demographic, and hospital characteristics. SAS version 9.4 (SAS Institute, Cary, North Carolina) was used for all analyses. All tests were two-sided, and a P value <.05 was considered as statistically significant.

Sample Characteristics

In the weighted analyses, we identified 133,516 hospitalizations in acute-care hospitals for SI or SA, and 8.5% (n = 11,375) of hospitalizations had at least one unplanned 30-day readmission to an acute-care hospital. Unweighted, the sample included 37,683 patients and 42,198 hospitalizations. Among all patients represented in the sample, 90.5% had only a single SI or SA hospitalization, 7.7% had two hospitalizations, and 1.8% had >2 hospitalizations in one year.

Table 1 summarizes the sample characteristics and displays the demographic, clinical, and hospital characteristics of index hospitalizations by the 30-day readmission status. Patients represented in the index hospitalizations were 64.9% female, 3.6% were aged 6-9 years, 40.1% were aged 10-14 years, and 56.3% were aged 15-17 years. Nearly half of the patients (49.1%) used public insurance. Nearly half (44.9%) lived in metropolitan areas with >1 million residents, 36.1% lived in metropolitan areas with 50,000 to 1 million residents, and 14.7% lived in rural areas.

Association of Patient and Hospital Characteristics with Readmissions

Table 2 displays the patient and hospital characteristics associated with readmissions. Among demographic characteristics, 10- to 14-year-old patients had higher odds of readmission (odds ratio [OR]: 1.18, 95% confidence interval [CI]: 1.07-1.29) than 15- to 17-year-old patients. Having public insurance was associated with higher odds of readmission (OR: 1.14, 95% CI: 1.04-1.25). We found no differences in readmission rates based on sex, urban or rural location, or patient’s ZIP code income quartile.

Among clinical characteristics, hospitalizations with an admission for SI or SA in the preceding 30 days, meaning that the index hospitalization itself was a readmission, had the strongest association with readmissions (OR: 3.14, 95% CI: 2.73-3.61). In addition, patients admitted via the ED for the index hospitalization had higher odds of readmission (OR: 1.25, 95% CI: 1.15-1.36). Chronic psychiatric conditions associated with higher odds of readmission included psychotic disorders (OR: 1.39, 95% CI: 1.16-1.67) and bipolar disorder (OR: 1.27, 95% CI: 1.13-1.44).

Characteristics of 30-day Readmissions

Table 3 displays the characteristics of readmissions after SI compared to that after SA. Among the combined sample of 11,375 30-day readmissions, 34.1% occurred within 7 days, and 65.9% in 8-30 days. Eleven percent of patients with any readmission had more than one readmission within 30 days. Among readmissions, 94.5% were for a psychiatric problem and 5.5% were for a medical or surgical problem. A total of 43.9% had a diagnosis of SI and 19.5% a diagnosis of SA. Readmissions were more likely to occur at a different hospital after SI than after SA (48.1% vs. 31.3%, P < .001). Medical and surgical indications for readmission were less common after SI than after SA (4.4% vs. 8.7%, P < .001). Only 1.2% of SI hospitalizations had a readmission for SA within 30 days. Of these cases, 55.6% were aged 15-17 years, 43.3% were aged 10-14 years, and 1.1% were aged 6-9 years; 73.1% of the patients were female, and 49.1% used public insurance.

SI and SA in children and adolescents are substantial public health problems associated with significant hospital resource utilization. In 2013 and 2014, there were 181,575 pediatric acute-care hospitalizations for SI or SA, accounting for 9.5% of all hospitalizations in 6- to 17-year-old patients nationally. Among acute-care SI and SA hospitalizations, 8.5% had a readmission to an acute-care hospital within 30 days. The study data source did not include psychiatric specialty hospitals, and the number of index hospitalizations is likely substantially higher when psychiatric specialty hospitalizations are included. Readmissions may also be higher if patients were readmitted to psychiatric specialty hospitals after discharge from acute-care hospitals. The strongest risk factor for unplanned 30-day readmissions was a previous hospitalization in the 30 days before the index admission, likely a marker for severity or complexity of psychiatric illness. Other characteristics associated with higher odds of readmission were bipolar disorder, psychotic disorders, and age 10-14 years. More than one-third of readmissions occurred within the first 7 days after hospital discharge. The prevalence of SI and SA hospitalizations and readmissions was similar to findings in previous analyses of mental health hospitalizations.^10,28

A patient’s psychiatric illness type and severity, as evidenced by the need for frequent repeat hospitalizations, was highly associated with the risk of 30-day readmission. Any hospitalization in the 30 days preceding the index hospitalization, whether for SI/SA or for another problem, was a strong risk factor for readmissions. We suspect that prior SI/SA hospitalizations reflect a patient’s chronic elevated risk for suicide. Prior hospitalizations not for SI or SA could be hospitalizations for mental illness exacerbations that increase the risk of SI or SA, eg, bipolar disorder with acute mania, or they could represent physical health problems. Chronic physical health problems are a known risk factor for SI and SA.²⁹

A knowledge of those characteristics that increase the readmission risk can inform future resource allocation, research, and policy in several ways. First, longer hospital stays could mitigate readmission risk in some patients with severe psychiatric illness. European studies in older adolescents and adults show that for severe psychiatric illness, a longer hospital stay is associated with a lower risk of hospital readmission.^15,30 Second, better access to intensive community-based mental health (MH) services, including evidence-based psychotherapy and medication management, improves symptoms in young people.³¹ Access to these services likely affects the risk of hospital readmission. We found that readmission risk was highest in 10- to 14-year-olds. Taken in the context of existing evidence that suicide rates are rising in younger patients,^1,3 our findings suggest that particular attention to community services for younger patients is needed. Third, care coordination could help patients access beneficial services to reduce readmissions and improve other outcomes. Enhanced discharge care coordination reduced suicide deaths in high-risk populations in Europe³² and Japan³³ and improved attendance at mental health follow-up after pediatric ED discharge in a small United States sample.³⁴ Given that one-third of readmissions occurred within seven days, care coordination designed to ensure access to ambulatory services in the immediate postdischarge period may be particularly beneficial.

We found that ZIP code income quartile was not associated with readmissions. We suspect that poverty is not as closely correlated with MH hospitalization outcomes as it is with physical health hospitalization outcomes for several reasons. Medicaid insurance historically has more robust coverage of mental health services than some private insurance plans, which might offset some of the risk of poor mental health outcomes associated with poverty. Low-income families are eligible to use social services, and families accessing social services might have more opportunities to become familiar with community mental health programs. Further, the expectation of high achievement found in some higher income families is associated with MH problems in children and adolescents.³⁵ Therefore, being in a higher income quartile might not be as protective against poor mental health outcomes as it is against poor physical health outcomes.

Although the NRD provides a rich source of readmission data across hospitals nationally, several limitations are inherent to this administrative dataset. First, data from specialty psychiatric hospitals were not included in the NRD. The study underestimates the total number of index hospitalizations and readmissions, since index SI/SA hospitalizations at psychiatric hospitals are not included, and readmissions are not included if they occurred at specialty psychiatric hospitals. Second, because data cannot be linked between calendar years, we excluded January and December hospitalizations, and findings might not generalize to hospitalizations in January and December. Seasonal trends in SI/SA hospitalizations are known to occur.³⁶ Third, race, ethnicity, primary language, gender identity, and sexual orientation are not available in the NRD, and we could not examine the association of these characteristics with the likelihood of readmissions. Fourth, we did not have information about pre- or posthospitalization insurance enrollment or outpatient services that could affect the risk of readmission. Nevertheless, this study offers information on the characteristics of readmissions after hospitalizations for SI and SA in a large nationally representative sample of youth, and the findings can inform resource planning to prevent suicides.

Hospital readmissions are common in patients with SI and SA, and patients with a recent previous hospitalization have the highest risk of readmission. More than one-third of readmissions after SI or SA occurred within the first seven days. Due to the dearth of mental health services in the community, hospitals offer an important safety net for youth experiencing acute suicidal crises. Strategies to improve the continuum of care for patients at risk of suicide that solely focus on reducing readmissions are not likely to benefit patients. However, readmissions can identify opportunities for improving hospital discharge processes and outpatient services. Future research and clinical innovation to investigate and improve hospital discharge planning and access to community mental health services is likely to benefit patients and could reduce 30-day hospital readmissions.

Acknowledgments

The authors thank John Lawlor for his assistance with the analysis.

Disclosures

The authors have no potential conflicts of interest to disclose.

Funding

Dr. Zima received funding from the Behavioral Health Centers of Excellence for California (SB852).

Suicide is a leading cause of death among 10- to 34-year-olds in the United States.^1,2 During the past two decades, the youth suicide death rate has risen by 24%, and more than 5,000 young people die from suicide each year.³ Suicide ideation (SI) and suicide attempts (SAs) are well-established risk factors for suicide death and a source of morbidity for patients and families. One-third of youth with SI attempt suicide at some point in their lifetime.⁴ Approximately 11% of youth SAs result in suicide death, and 2% of youth who attempt suicide subsequently go on to die from suicide after recovering from a prior SA.⁵ More than 60,000 youth are hospitalized for SI or SA each year,⁶ and young people hospitalized for SA are at high short-term risk of repeat SA and suicide death.⁷ Hospitals need strategies for measuring the quality of SI and SA hospitalizations, monitoring postdischarge outcomes, and identifying the patients at the highest risk of poor outcomes. Readmissions are a useful hospital quality measure that can indicate re-emergence of SI, repeat SA, or inadequate community-based mental health treatment, and interventions designed for patients with readmissions can potentially avert morbidity or mortality.

The National Committee on Quality Assurance recommends measurement of quality metrics for 30-day mental health follow-up after psychiatric hospitalizations, 30-day readmissions after adult (but not pediatric) psychiatric hospitalizations, and 30-day readmissions in pediatric medical and surgical hospitalizations. Readmission measures are not consistently used to evaluate pediatric psychiatric hospitalizations, and psychiatric quality measures are not used to evaluate medical or surgical hospitalizations for SA. Recent research has investigated transfers to postacute care,⁸ readmission prevalence, variation in hospital readmission performance, and risk factors for readmissions after pediatric psychiatric hospitalizations.^9–11 However, no national study has investigated 30-day readmissions in youth hospitalized specifically for SI or SA.

To inform hospital quality measurement and improve hospital and postdischarge care for youth at risk of suicide, more information is needed about the characteristics of and the risk factors for readmissions after index SI/SA hospitalization. To address this knowledge gap, among SI/SA hospitalizations in 6- to 17-year-olds, we examined (1) unplanned 30-day readmissions and characteristics of hospitalizations by 30-day readmission status; (2) patient, hospital, and regional characteristics associated with 30-day readmissions; and (3) characteristics of 30-day readmissions.

Study Design and Data Source

We conducted a national, retrospective cohort study of hospitalizations for patients aged 6-17 years using the Agency for Healthcare Research and Quality (AHRQ) 2013 and 2014 Nationwide Readmissions Database (NRD). The combined 2013-2014 NRD includes administrative data from a nationally representative sample of 29 million hospitalizations in 22 states, accounting for 49.3% of all US hospitalizations, and is weighted for national projections. The NRD includes hospital information, patient demographic information, and International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis, procedure, and external cause of injury codes (E-codes). The database includes one primary diagnosis, up to 24 additional diagnoses, and up to 4 E-codes for each hospitalization. The NRD includes information about hospitalizations in acute-care general hospitals (including their psychiatric units) but not from specialty psychiatric hospitals. The database also includes de-identified, verified patient linkage numbers so that patients can be tracked across multiple hospitalizations at the same institution or different institutions within the same state. This study was considered to be exempt from review by the Children’s Hospital of Philadelphia Institutional Review Board.

Sample

We identified a sample of 181,575 hospitalizations for SI (n = 119,037) or SA (n = 62,538) among 6- to 17-year-olds between January 1, 2013, and December 31, 2014 (Figure). We included children as young as 6 years because validated methods exist to identify SI and SA in this age group,¹² and because suicide deaths have recently increased among younger children.³ We excluded patients aged 18 years and older from this study since delivery of mental health services differs for adults.¹³ To create the sample, we first identified all hospitalizations of patients aged 6-17 years. We then used a validated algorithm relying on ICD-9-CM diagnosis codes for poisonings and E-codes for self-injury (E950-959) to identify hospitalizations related to SA.¹² Because E-code completeness varies among states,¹⁴ we also used the combination of having both a diagnosis code for injury (800-999) and an ICD-9-CM code for SI (V62.84) as a proxy for SA. Among hospitalizations without SA, we identified hospitalizations with SI using the ICD-9-CM code for SI (V62.84) in any position.

We identified 133,516 index hospitalizations with complete data at risk for an unplanned readmission. Because NRD data cannot be linked between calendar years, we limited the study time period for each calendar year to 10 months. We excluded hospitalizations in January because the full 30-day time frame to determine whether a hospitalization had occurred in the preceding 30 days, a known risk factor for readmissions in other samples,¹⁵ was not available. We excluded index hospitalizations occurring in December, because the full 30-day time frame to ascertain readmissions was not available. We excluded hospitalizations resulting in death, since these are not at risk for readmission, and hospitalizations resulting in transfer, since the timing of discharge to the community was not known. Readmission hospitalizations were eligible to be included as index hospitalizations if they met sample inclusion criteria. The final sample for readmission analyses included 95,354 SI hospitalizations and 38,162 SA hospitalizations (Figure).

Primary Outcome

The primary outcome was any unplanned, all-cause readmission within 30 days of index hospitalization for SI or SA. Among 30-day readmissions, we examined readmission timing, whether the readmission was to the same hospital or a different hospital, length of stay, and indication for readmission (medical/surgical or psychiatric, and presence of SI or SA diagnoses). Planned readmissions were identified using measure specifications endorsed by the AHRQ and the National Quality Forum¹⁶ and excluded from measurement.

Among index hospitalizations for SI, we specifically examined 30-day readmissions for subsequent SA, since one objective of hospitalization for SI is to prevent progression to SA or death. We could not identify hospitalizations for repeat SA after index hospitalization for SA, because diagnosis codes did not differentiate between readmission for complications of index SA and readmission for repeat SA.

Independent Variables

We analyzed demographic, clinical, and hospital factors associated with readmissions in other samples.^17–20 Demographic characteristics included patient gender and age, urban or rural residence, payer, and median national income quartile for a patient’s ZIP code. Race and ethnicity data are not available in the NRD.

Clinical characteristics included hospitalization in the 30-days preceding the index hospitalization, index hospitalization length of stay, and admission via the emergency department (ED) versus direct admission. A patient’s chronic condition profile was determined using index hospitalization diagnosis codes. Complex chronic conditions (eg, cancer, cystic fibrosis) were identified using a classification system used in several prior studies of hospital administrative datasets,²¹ and other noncomplex chronic medical conditions (eg, asthma, obesity) were identified using the Healthcare Cost and Utilization Project (HCUP) chronic condition indicator system.²² Psychiatric conditions (eg, anxiety disorders, substance abuse, autism) were identified and categorized using a classification system used in studies of hospital administrative datasets.²³ The number of psychiatric conditions was determined by counting the number of psychiatric condition categories in which a patient had a diagnosis. SA was categorized as having lower risk of death (eg, medication overdose, injury from cutting or piercing) or higher risk of death (eg, hanging, suffocation, or firearm injury).²⁴

Because of known temporal trends in SI and SA,^25,26 the month and year of admission were included as covariates. Hospital characteristics included teaching hospital and children’s hospital designations.

Statistical Analysis

We compared descriptive, summary statistics for characteristics of index hospitalizations with and without a 30-day readmission using Rao-Scott chi-square tests. In multivariable analyses, we derived logistic regression models to measure the associations of patient, hospital, and temporal factors with 30-day hospital readmissions. Analyses were conducted in SAS PROC SURVEYLOGISTIC and were weighted to achieve national estimates, clustered by sample stratum and hospital to account for the complex survey design,²⁷ clustered by patients to account for multiple index visits per patient, and adjusted for clinical, demographic, and hospital characteristics. SAS version 9.4 (SAS Institute, Cary, North Carolina) was used for all analyses. All tests were two-sided, and a P value <.05 was considered as statistically significant.

Sample Characteristics

In the weighted analyses, we identified 133,516 hospitalizations in acute-care hospitals for SI or SA, and 8.5% (n = 11,375) of hospitalizations had at least one unplanned 30-day readmission to an acute-care hospital. Unweighted, the sample included 37,683 patients and 42,198 hospitalizations. Among all patients represented in the sample, 90.5% had only a single SI or SA hospitalization, 7.7% had two hospitalizations, and 1.8% had >2 hospitalizations in one year.

Table 1 summarizes the sample characteristics and displays the demographic, clinical, and hospital characteristics of index hospitalizations by the 30-day readmission status. Patients represented in the index hospitalizations were 64.9% female, 3.6% were aged 6-9 years, 40.1% were aged 10-14 years, and 56.3% were aged 15-17 years. Nearly half of the patients (49.1%) used public insurance. Nearly half (44.9%) lived in metropolitan areas with >1 million residents, 36.1% lived in metropolitan areas with 50,000 to 1 million residents, and 14.7% lived in rural areas.

Association of Patient and Hospital Characteristics with Readmissions

Table 2 displays the patient and hospital characteristics associated with readmissions. Among demographic characteristics, 10- to 14-year-old patients had higher odds of readmission (odds ratio [OR]: 1.18, 95% confidence interval [CI]: 1.07-1.29) than 15- to 17-year-old patients. Having public insurance was associated with higher odds of readmission (OR: 1.14, 95% CI: 1.04-1.25). We found no differences in readmission rates based on sex, urban or rural location, or patient’s ZIP code income quartile.

Among clinical characteristics, hospitalizations with an admission for SI or SA in the preceding 30 days, meaning that the index hospitalization itself was a readmission, had the strongest association with readmissions (OR: 3.14, 95% CI: 2.73-3.61). In addition, patients admitted via the ED for the index hospitalization had higher odds of readmission (OR: 1.25, 95% CI: 1.15-1.36). Chronic psychiatric conditions associated with higher odds of readmission included psychotic disorders (OR: 1.39, 95% CI: 1.16-1.67) and bipolar disorder (OR: 1.27, 95% CI: 1.13-1.44).

Characteristics of 30-day Readmissions

Table 3 displays the characteristics of readmissions after SI compared to that after SA. Among the combined sample of 11,375 30-day readmissions, 34.1% occurred within 7 days, and 65.9% in 8-30 days. Eleven percent of patients with any readmission had more than one readmission within 30 days. Among readmissions, 94.5% were for a psychiatric problem and 5.5% were for a medical or surgical problem. A total of 43.9% had a diagnosis of SI and 19.5% a diagnosis of SA. Readmissions were more likely to occur at a different hospital after SI than after SA (48.1% vs. 31.3%, P < .001). Medical and surgical indications for readmission were less common after SI than after SA (4.4% vs. 8.7%, P < .001). Only 1.2% of SI hospitalizations had a readmission for SA within 30 days. Of these cases, 55.6% were aged 15-17 years, 43.3% were aged 10-14 years, and 1.1% were aged 6-9 years; 73.1% of the patients were female, and 49.1% used public insurance.

SI and SA in children and adolescents are substantial public health problems associated with significant hospital resource utilization. In 2013 and 2014, there were 181,575 pediatric acute-care hospitalizations for SI or SA, accounting for 9.5% of all hospitalizations in 6- to 17-year-old patients nationally. Among acute-care SI and SA hospitalizations, 8.5% had a readmission to an acute-care hospital within 30 days. The study data source did not include psychiatric specialty hospitals, and the number of index hospitalizations is likely substantially higher when psychiatric specialty hospitalizations are included. Readmissions may also be higher if patients were readmitted to psychiatric specialty hospitals after discharge from acute-care hospitals. The strongest risk factor for unplanned 30-day readmissions was a previous hospitalization in the 30 days before the index admission, likely a marker for severity or complexity of psychiatric illness. Other characteristics associated with higher odds of readmission were bipolar disorder, psychotic disorders, and age 10-14 years. More than one-third of readmissions occurred within the first 7 days after hospital discharge. The prevalence of SI and SA hospitalizations and readmissions was similar to findings in previous analyses of mental health hospitalizations.^10,28

A patient’s psychiatric illness type and severity, as evidenced by the need for frequent repeat hospitalizations, was highly associated with the risk of 30-day readmission. Any hospitalization in the 30 days preceding the index hospitalization, whether for SI/SA or for another problem, was a strong risk factor for readmissions. We suspect that prior SI/SA hospitalizations reflect a patient’s chronic elevated risk for suicide. Prior hospitalizations not for SI or SA could be hospitalizations for mental illness exacerbations that increase the risk of SI or SA, eg, bipolar disorder with acute mania, or they could represent physical health problems. Chronic physical health problems are a known risk factor for SI and SA.²⁹

A knowledge of those characteristics that increase the readmission risk can inform future resource allocation, research, and policy in several ways. First, longer hospital stays could mitigate readmission risk in some patients with severe psychiatric illness. European studies in older adolescents and adults show that for severe psychiatric illness, a longer hospital stay is associated with a lower risk of hospital readmission.^15,30 Second, better access to intensive community-based mental health (MH) services, including evidence-based psychotherapy and medication management, improves symptoms in young people.³¹ Access to these services likely affects the risk of hospital readmission. We found that readmission risk was highest in 10- to 14-year-olds. Taken in the context of existing evidence that suicide rates are rising in younger patients,^1,3 our findings suggest that particular attention to community services for younger patients is needed. Third, care coordination could help patients access beneficial services to reduce readmissions and improve other outcomes. Enhanced discharge care coordination reduced suicide deaths in high-risk populations in Europe³² and Japan³³ and improved attendance at mental health follow-up after pediatric ED discharge in a small United States sample.³⁴ Given that one-third of readmissions occurred within seven days, care coordination designed to ensure access to ambulatory services in the immediate postdischarge period may be particularly beneficial.

We found that ZIP code income quartile was not associated with readmissions. We suspect that poverty is not as closely correlated with MH hospitalization outcomes as it is with physical health hospitalization outcomes for several reasons. Medicaid insurance historically has more robust coverage of mental health services than some private insurance plans, which might offset some of the risk of poor mental health outcomes associated with poverty. Low-income families are eligible to use social services, and families accessing social services might have more opportunities to become familiar with community mental health programs. Further, the expectation of high achievement found in some higher income families is associated with MH problems in children and adolescents.³⁵ Therefore, being in a higher income quartile might not be as protective against poor mental health outcomes as it is against poor physical health outcomes.

Although the NRD provides a rich source of readmission data across hospitals nationally, several limitations are inherent to this administrative dataset. First, data from specialty psychiatric hospitals were not included in the NRD. The study underestimates the total number of index hospitalizations and readmissions, since index SI/SA hospitalizations at psychiatric hospitals are not included, and readmissions are not included if they occurred at specialty psychiatric hospitals. Second, because data cannot be linked between calendar years, we excluded January and December hospitalizations, and findings might not generalize to hospitalizations in January and December. Seasonal trends in SI/SA hospitalizations are known to occur.³⁶ Third, race, ethnicity, primary language, gender identity, and sexual orientation are not available in the NRD, and we could not examine the association of these characteristics with the likelihood of readmissions. Fourth, we did not have information about pre- or posthospitalization insurance enrollment or outpatient services that could affect the risk of readmission. Nevertheless, this study offers information on the characteristics of readmissions after hospitalizations for SI and SA in a large nationally representative sample of youth, and the findings can inform resource planning to prevent suicides.

Hospital readmissions are common in patients with SI and SA, and patients with a recent previous hospitalization have the highest risk of readmission. More than one-third of readmissions after SI or SA occurred within the first seven days. Due to the dearth of mental health services in the community, hospitals offer an important safety net for youth experiencing acute suicidal crises. Strategies to improve the continuum of care for patients at risk of suicide that solely focus on reducing readmissions are not likely to benefit patients. However, readmissions can identify opportunities for improving hospital discharge processes and outpatient services. Future research and clinical innovation to investigate and improve hospital discharge planning and access to community mental health services is likely to benefit patients and could reduce 30-day hospital readmissions.

Acknowledgments

The authors thank John Lawlor for his assistance with the analysis.

Disclosures

The authors have no potential conflicts of interest to disclose.

Funding

Dr. Zima received funding from the Behavioral Health Centers of Excellence for California (SB852).

Compared with other industries, medicine has been slow to embrace the digital age. Electronic health records have only recently become ubiquitous, and that was only realized after governmental prodding through Meaningful Use legislation. Other digital tools, such as video or remote sensor technologies, have been available for decades but had not been introduced into routine medical care until recently for various reasons, ranging from costs to security to reimbursement rules. However, we are currently in the midst of a paradigm shift in medicine toward virtual care, as exemplified by the Kaiser Permanente CEO’s proclamation in 2017 that this capitated care system had moved over half of its 100 million annual patient encounters to the virtual environment.¹

Regulation – both at the state and federal levels – has been the largest barrier to the adoption of virtual care. State licensure regulations for practicing medicine hamper virtual visits, which can otherwise be easily achieved without regard to geography. Although the Centers for Medicare & Medicaid Services (CMS) has had provisions for telehealth billing, these have been largely limited to rural areas. However, regulations are constantly evolving as the Interstate Medical Licensure Compact list is not CMS. The Interstate Medical Licensure Compact (www.imlcc.org) is an agreement involving 24 states that permits licensed physicians to practice medicine across state lines. CMS has recently proposed to add payments for virtual check-in visits, which will not be subject to the prior limitations on Medicare telehealth services.² These and future changes in regulation will likely spur the rapid adoption and evolution of virtual services.

Disclosures

Dr. Ong and Dr. Brotman have nothing to disclose.

Compared with other industries, medicine has been slow to embrace the digital age. Electronic health records have only recently become ubiquitous, and that was only realized after governmental prodding through Meaningful Use legislation. Other digital tools, such as video or remote sensor technologies, have been available for decades but had not been introduced into routine medical care until recently for various reasons, ranging from costs to security to reimbursement rules. However, we are currently in the midst of a paradigm shift in medicine toward virtual care, as exemplified by the Kaiser Permanente CEO’s proclamation in 2017 that this capitated care system had moved over half of its 100 million annual patient encounters to the virtual environment.¹

Regulation – both at the state and federal levels – has been the largest barrier to the adoption of virtual care. State licensure regulations for practicing medicine hamper virtual visits, which can otherwise be easily achieved without regard to geography. Although the Centers for Medicare & Medicaid Services (CMS) has had provisions for telehealth billing, these have been largely limited to rural areas. However, regulations are constantly evolving as the Interstate Medical Licensure Compact list is not CMS. The Interstate Medical Licensure Compact (www.imlcc.org) is an agreement involving 24 states that permits licensed physicians to practice medicine across state lines. CMS has recently proposed to add payments for virtual check-in visits, which will not be subject to the prior limitations on Medicare telehealth services.² These and future changes in regulation will likely spur the rapid adoption and evolution of virtual services.

Disclosures

Dr. Ong and Dr. Brotman have nothing to disclose.

Compared with other industries, medicine has been slow to embrace the digital age. Electronic health records have only recently become ubiquitous, and that was only realized after governmental prodding through Meaningful Use legislation. Other digital tools, such as video or remote sensor technologies, have been available for decades but had not been introduced into routine medical care until recently for various reasons, ranging from costs to security to reimbursement rules. However, we are currently in the midst of a paradigm shift in medicine toward virtual care, as exemplified by the Kaiser Permanente CEO’s proclamation in 2017 that this capitated care system had moved over half of its 100 million annual patient encounters to the virtual environment.¹

Regulation – both at the state and federal levels – has been the largest barrier to the adoption of virtual care. State licensure regulations for practicing medicine hamper virtual visits, which can otherwise be easily achieved without regard to geography. Although the Centers for Medicare & Medicaid Services (CMS) has had provisions for telehealth billing, these have been largely limited to rural areas. However, regulations are constantly evolving as the Interstate Medical Licensure Compact list is not CMS. The Interstate Medical Licensure Compact (www.imlcc.org) is an agreement involving 24 states that permits licensed physicians to practice medicine across state lines. CMS has recently proposed to add payments for virtual check-in visits, which will not be subject to the prior limitations on Medicare telehealth services.² These and future changes in regulation will likely spur the rapid adoption and evolution of virtual services.

Disclosures

Dr. Ong and Dr. Brotman have nothing to disclose.

Through increased involvement with families and caregivers, community hospitals can deliver better healthcare to patients.^1,2 Furthermore, when patients bypass local hospitals and directly present to tertiary care, mortality for time-sensitive illnesses, such as sepsis, increases.³ Unfortunately, although critical access hospitals (CAHs) had an equivalent risk-adjusted mortality in 2002, they have failed to improve their performance at the same rate as that of larger hospitals and lag in quality metrics.^4,5

One potential contributor to the lagging performance may be the low uptake of the hospitalist model at these facilities.⁶ Although dedicated hospitalists have improved patient outcomes and decreased spending in large hospitals,^7-9 implementing the hospitalist medicine model on a smaller scale remains difficult. Approximately 1,300 CAHs provide necessary emergency room and inpatient services in the rural United States.¹⁰ Assuming 12-hour shifts and every-other-week assignments, providing continuous, on-location hospitalist coverage would require more than 10% of the total hospitalist workforce to cover less than 3% of all hospital admissions.^11-13

Telemedicine allows content experts, including hospitalists, to supervise patient care remotely. This provides a potential solution to the logistical challenges of supplying continuous hospitalist coverage to a remote facility with a low daily census. We hypothesized that providing continuous “virtual hospitalist” coverage through telemedicine could increase the ability of a CAH to care for patients locally, decreasing the number of transfers to tertiary care centers and improving patient and provider satisfaction. We aimed to create a 25% relative reduction in CAH Emergency Department (ED) patient encounters resulting in transfer to outside hospitals within 6 months.

This quality improvement project was exempt from Institutional Review Board review.

Setting

The University of Iowa Hospitals and Clinics (UIHC) is a 750-bed teaching hospital based in a suburban community in Eastern Iowa and the only tertiary care hospital in the state of Iowa. The UIHC Hospitalist group contains 44 staff physicians and covers more than 12 service lines (both faculty-only and resident-covered) at this facility.

Van Buren County Hospital (VBCH) is a 24-bed CAH offering emergency, internal medicine, and obstetrical services and located 80 miles southwest of UIHC. X-ray and CT scan services are available continuously, but ultrasound and magnetic resonance imaging services are available only 2-3 times per week. While tertiary care patients were transferred to UIHC, patients requiring specialty care but with less complex illnesses (eg, stable myocardial infarction) were referred to closer facilities.

Prior to implementation, coverage of the acute inpatient ward and the emergency room at VBCH was simultaneously provided by a single physician or advanced practice providers (APPs). When APPs provided coverage, a physician was required to be notified of any new admissions and was immediately available for medical emergencies. The VBCH providers worked alone in 48- to 72-hour continuous shifts as the sole coverage for both ED and inpatient units. It was frequently necessary to bring in outside providers through locum tenens agencies to fill gaps in the provider schedule. Both VBCH and UIHC used a shared electronic medical record (EMR), which was a key consideration in choosing VBCH as our pilot site. Providers at both institutions had access to identical patient information through the EMR, including radiology images, laboratory results, and provider notes.

Intervention Development and Implementation

A site visit by clinical and administrative project leads to VBCH identified three deficits that we could address through telemedicine: (1) The extended duration of VBCH shifts was detrimental to provider experience and retention; (2) Lack of local expertise in hospital medicine led to limited comfort in caring for patients with stable but medically complex conditions (eg, drug-resistant urinary tract infection); and (3) Patient transitions between VBCH and UIHC during acute care transfer were frustrating and led to negative experiences with providers and patients.

We developed a model to address these deficits using the minimum number of specialties and employees to facilitate rapid implementation. Although local care ED and inpatient care was provided by 3 APPS and a single physician provider, we mandated the coverage of all acute inpatients by the virtual hospitalists. This coverage included daily videoconference patient rounds, continuous pager coverage for new acute issues, and listing the virtual hospitalists as the attending of record for patient admissions. We scheduled contact times in the morning and afternoon to accelerate familiarity and comfort with the technology. We used a secure, Health Insurance Portability and Accountability Act of 1996 (HIPAA)-compliant platform for videoconferencing, accessible through personal computers or portable smart devices (Vidyo, VidyoInc, Hackensack, New Jersey). At VBCH, two tablet computers were provided to serve as portable platforms to use either in provider conference rooms or to be taken into patient rooms. Twice a day, at 8:45 am and 4:30 pm, virtual hospitalists, local providers, and nursing staff would videoconference and review the status and care plan for all admitted patients. In addition, virtual hospitalists performed a videoconference interview using the tablet computers with all patients on the morning following admission and at other times on an as-needed basis. We asked the virtual hospitalists to cover a minimum of 72 consecutive hours to maintain provider continuity. Local APPs documented the history, examination, and medical decision-making for billing purposes, which were cosigned by the virtual hospitalists. The virtual hospitalists also created separate notes documenting their discussions with local staff, interview and limited direct physical examination findings (eg, appearance of rashes), and medical decision making. Due to limitations of the EMR, local APPs wrote patient orders. All virtual hospitalists were credentialed by proxy at VBCH. We consulted with the UIHC legal team to ensure that virtual hospitalists would be protected under their existing malpractice insurance.

Outcome Measures

Outcome measures were divided into three categories: (1) clinical and utilization outcomes; (2) virtual hospitalist outcomes; and (3) satisfaction outcomes. The primary clinical outcome was the percentage of ED encounters resulting in transfer to a different acute care hospital. We also monitored alternative ED dispositions, including local inpatient admission. Additional clinical and utilization outcomes after ED admission included the mean daily inpatient census at VBCH and the case mix index (CMI). We selected the mean length of stay, the percentage of inpatients transferred to other hospitals, and the inpatient mortality as balance measures due to concerns of increasing the acuity of the inpatient wards beyond the comfort and expertise of local staff. Virtual hospitalist outcomes included the mean daily time commitment and the mean time commitment per patient. Virtual hospitalists self-reported their time commitments as part of their daily documentation. We chose these measures in anticipation of expanding this program to other institutions in the future. Satisfaction outcomes included a weekly survey to all VBCH physicians and nursing staff (Appendix 1), weekly group discussions with virtual hospitalists and CAH staff, and 3 interviews with patients and family members after discharge (Appendix 2).

Statistical Analysis

Baseline data collected over a period of 24 weeks were used to measure pre-implementation performance and trends at VBCH. The virtual hospitalist service was started on November 15, 2016, and the two weeks before and two weeks after this date were excluded from analysis as a transition period. To account for weekend variation, we reported data in consecutive 28-day blocks. We used Chi-square tests to compare proportional outcomes and Student’s t-tests for continuous variables. Statistical Process Control charts were used to evaluate for temporal trends in quantitative data.

Funding

Development of this project was funded through the University of Iowa Hospitalist group and the Signal Center for Health Innovations at UI Health Ventures. Virtual hospitalist clinical time was paid for by the CAH on a fractional basis of a traditional hospitalist based on projected patient volumes through analysis of baseline data. Patients were not directly billed for virtual hospitalist service but were charged for the services provided by CAH providers.

Clinical and Utilization Outcomes

During the 24-week baseline period, VBCH had 947 ED encounters and 176 combined acute inpatient and observation admissions. For the 24 weeks following the transition, there were 930 ED visits and 186 admissions. We observed a 36% (157/947 to 98/930, P < .001) decrease in ED encounters ending in patient transfer to another hospital (Figure). In parallel, VBCH ED visits leading to local admission increased by 62% of baseline (39/947 to 62/930, P = .014). There was no significant change in the fraction of ED encounters resulting in an observation stay (104/947 to 99/930, P = .814). Daily ED visits did not change after virtual hospitalist coverage began (5.64 to 5.54 visits/day, P = .734), but the percentage of ED visits ending in discharge to a nonmedical setting increased from 79.0% to 82.7% (748/947 to 769/930, P = .042).

The implementation did not have a significant impact on ward census or patient complexity (Table 1). Both CMI and mean length of stay did not change after starting the service. The study was underpowered to detect differences in rare events, including inpatient mortality and transfer after admission. Despite the decrease in transfers, inpatient census was unchanged. This coincides with a 17% decrease (196/947 to 160/930, P = .054) in the proportion of ED patients referred for admission either locally or at an outside hospital.

Virtual Hospitalist Outcomes

Satisfaction Outcomes

The staff at VBCH identified several benefits to the virtual hospitalist service. Survey responses (N = 18) were positive, with staff expressing specific gratitude for the additional education and training provided by the virtual hospitalists. On a Likert scale ranging from 1 (very poor) to 5 (excellent), the respondents gave high mean scores to the overall service experience (4.8) and the effectiveness of care delivered (4.9) but were more critical of the ability to keep patients locally (4.5) and the experience with transferring patients (3.9). We also collected free-text feedback from both patients and staff at VBCH (Table 2).

The virtual hospitalist service allowed a higher percentage of acute inpatients to receive care in their local hospital and was positively perceived by providers and patients. The per-patient time commitment by virtual hospitalists was similar to traditional hospitalist coverage¹⁴ and could scale to multiple simultaneous institutions.

Despite the increase in the proportion of patients admitted locally, neither the mean inpatient census nor the complexity of patients (as measured by CMI) increased. The increase in patients admitted locally was offset by a parallel increase in the number of ED patients discharged home. Although virtual hospitalists were available to consult on ED patients, this consultation was not mandatory unless the CAH provider felt that admission was indicated. It remains unclear whether the changes in ED disposition were due to direct intervention by virtual hospitalists, increasing local expertise with inpatient medicine, or unrelated local factors.

Although outside transfers directly from the ED dropped, there was a potential increase in acute inpatients transferred after admission that failed to reach statistical significance. We anticipated increased transfers after admission as a potential consequence of accepting more complex patients for CAH admission. Reasons for transfer included emergent transfers for medically unstable patients and scheduled transfer for subspecialist evaluation or testing. Despite the possible increase in delayed transfers, there was no significant change in CAH inpatient mortality, and the total fraction of combined ED and inpatients transferred decreased after the intervention.

Despite the benefits of keeping patients within their communities, 20%-60% of rural patients bypass their local facilities when seeking emergent care.¹⁵ Despite publicity on local media,¹⁶ we did not observe an increase in daily ED visits after implementation. Although some investigators have found that increasing the services offered decreases in rural bypass,¹⁷ others have found no or mixed effects.^18,19 Further investigations into the local factors contributing to rural bypass may yield important insights, and future implementations should not rely on rapid increases in patient volume to establish economic viability.

Although telemedicine has been applied to a variety of previous settings, to our knowledge, this marks the first collaboration between an academic medical center and a CAH to provide continuous hospitalist coverage. A previous model for pediatric inpatients showed a similar decrease in patients transferred to tertiary centers.²⁰ Virtual hospitalists differ from other adult telemedicine projects, which focused on subspecialty care or overnight coverage.²¹ The advantages of our model include the ability to proactively address deficits, even when local providers are unaware of changes to the standards of care. We believe that mandatory scheduled interactions decreased the barriers to communication and increased provider reassurance in telemedicine management of their patients. The scheduled interactions also provided additional training and development for CAH personnel, were well received by local staff, and may contribute to local provider job satisfaction, retention, and recruitment.

Past efforts to integrate academic hospitalists into CAHs improved quality metrics and provider satisfaction but were economically infeasible due to low patient volumes.²² In contrast, virtual providers can distribute their efforts across multiple areas, including covering additional CAHs, providing local patient care at their home facility, or completing academic projects. By combining two or more CAHs into a single provider, sufficient patient volume can be generated to dedicated personnel.

There were several limitations to this initial investigation:

• As a pilot between two specific institutions, modifications will be required to replicate in other CAHs or academic centers.
• Generating sufficient revenue to cover a full hospitalist salary will require adding additional responsibilities, either covering multiple CAHs simultaneously or combining virtual coverage with in-person responsibilities.
• The accuracy of the self-report remains unmeasured, and the impact of combining two or more CAHs may not be strictly additive. Attempts to supplement the self-reported time spent with additional information from the EMR and cell phone logs were complicated by the use of multiple platforms in parallel, interruptions in provider workflow, and provider multitasking.
• Due to the need for reliable local physical examinations and regulations on telehealth reimbursement, local APPs were necessary for this implementation. Although most of the CAHs have an on-site provider to provide ED coverage, CAHs with sufficient volume to necessitate separating ED and inpatient ward coverage may have difficulty supporting both APP and virtual hospitalist coverage, even on a fractional basis.
• This study was underpowered to detect rare events with significant consequences, including inpatient mortality and inpatient transfer. Although CMI suggests similar complexity in CAH patients, we have insufficient data to draw further comparisons on patient characteristics before and after the intervention.
• The analysis may be vulnerable to secular trends in the CAH patient population, as only 24 weeks of data were used as a baseline for comparison (although no significant seasonal variation was detected during that time). Extending the baseline data to include an additional 30 weeks ED encounters did not significantly alter our conclusions.
• Virtual hospitalists were dependent on physical examinations performed independently by local APPs.
• Although virtual providers were obligated to be available for videoconferencing within 60 minutes, more urgent medical decisions were sometimes made based on phone conferences between VBCH and the virtual hospitalist without video or direct patient assessment.
• We selected a CAH utilizing an identical instance of our EMR. Although this increased the ability of virtual hospitalists to split their time between virtual and local patient encounters, this limits our ability to spread this intervention beyond institutions already partnering with the UIHC.

We succeeded in reducing outside transfers at a CAH by implementing a sustainable virtual hospitalist service. This model allows patients to receive more of their care within their local communities and provides an improved inpatient experience. Next steps include expanding this service to other CAHs within our region, both to understand if this model is applicable beyond our initial site and to monitor for complications induced by scaling. If successful, virtual hospitalist coverage can provide a sustainable solution to providing the latest innovations in hospital medicine even to the most rural communities.

The authors thank Ray Brownsworth, CEO of Van Buren County Hospital, as well as all the providers and staff who worked with them to implement and improve their services. The authors also thank Pat Brophy, founder of The Signal Center for Health Innovation, for providing leadership, support, and resources for innovation.

Disclosures 

None of the authors have identified a conflict of interest in relation to this manuscript.

Funding

This project was funded through the University of Iowa Health Care and the Signal Center for Health Innovations at UI Health Ventures.

Compliance With Ethical Standards

This quality improvement project was exempt from Institutional Review Board review

Through increased involvement with families and caregivers, community hospitals can deliver better healthcare to patients.^1,2 Furthermore, when patients bypass local hospitals and directly present to tertiary care, mortality for time-sensitive illnesses, such as sepsis, increases.³ Unfortunately, although critical access hospitals (CAHs) had an equivalent risk-adjusted mortality in 2002, they have failed to improve their performance at the same rate as that of larger hospitals and lag in quality metrics.^4,5

One potential contributor to the lagging performance may be the low uptake of the hospitalist model at these facilities.⁶ Although dedicated hospitalists have improved patient outcomes and decreased spending in large hospitals,^7-9 implementing the hospitalist medicine model on a smaller scale remains difficult. Approximately 1,300 CAHs provide necessary emergency room and inpatient services in the rural United States.¹⁰ Assuming 12-hour shifts and every-other-week assignments, providing continuous, on-location hospitalist coverage would require more than 10% of the total hospitalist workforce to cover less than 3% of all hospital admissions.^11-13

Telemedicine allows content experts, including hospitalists, to supervise patient care remotely. This provides a potential solution to the logistical challenges of supplying continuous hospitalist coverage to a remote facility with a low daily census. We hypothesized that providing continuous “virtual hospitalist” coverage through telemedicine could increase the ability of a CAH to care for patients locally, decreasing the number of transfers to tertiary care centers and improving patient and provider satisfaction. We aimed to create a 25% relative reduction in CAH Emergency Department (ED) patient encounters resulting in transfer to outside hospitals within 6 months.

This quality improvement project was exempt from Institutional Review Board review.

Setting

The University of Iowa Hospitals and Clinics (UIHC) is a 750-bed teaching hospital based in a suburban community in Eastern Iowa and the only tertiary care hospital in the state of Iowa. The UIHC Hospitalist group contains 44 staff physicians and covers more than 12 service lines (both faculty-only and resident-covered) at this facility.

Van Buren County Hospital (VBCH) is a 24-bed CAH offering emergency, internal medicine, and obstetrical services and located 80 miles southwest of UIHC. X-ray and CT scan services are available continuously, but ultrasound and magnetic resonance imaging services are available only 2-3 times per week. While tertiary care patients were transferred to UIHC, patients requiring specialty care but with less complex illnesses (eg, stable myocardial infarction) were referred to closer facilities.

Prior to implementation, coverage of the acute inpatient ward and the emergency room at VBCH was simultaneously provided by a single physician or advanced practice providers (APPs). When APPs provided coverage, a physician was required to be notified of any new admissions and was immediately available for medical emergencies. The VBCH providers worked alone in 48- to 72-hour continuous shifts as the sole coverage for both ED and inpatient units. It was frequently necessary to bring in outside providers through locum tenens agencies to fill gaps in the provider schedule. Both VBCH and UIHC used a shared electronic medical record (EMR), which was a key consideration in choosing VBCH as our pilot site. Providers at both institutions had access to identical patient information through the EMR, including radiology images, laboratory results, and provider notes.

Intervention Development and Implementation

A site visit by clinical and administrative project leads to VBCH identified three deficits that we could address through telemedicine: (1) The extended duration of VBCH shifts was detrimental to provider experience and retention; (2) Lack of local expertise in hospital medicine led to limited comfort in caring for patients with stable but medically complex conditions (eg, drug-resistant urinary tract infection); and (3) Patient transitions between VBCH and UIHC during acute care transfer were frustrating and led to negative experiences with providers and patients.

We developed a model to address these deficits using the minimum number of specialties and employees to facilitate rapid implementation. Although local care ED and inpatient care was provided by 3 APPS and a single physician provider, we mandated the coverage of all acute inpatients by the virtual hospitalists. This coverage included daily videoconference patient rounds, continuous pager coverage for new acute issues, and listing the virtual hospitalists as the attending of record for patient admissions. We scheduled contact times in the morning and afternoon to accelerate familiarity and comfort with the technology. We used a secure, Health Insurance Portability and Accountability Act of 1996 (HIPAA)-compliant platform for videoconferencing, accessible through personal computers or portable smart devices (Vidyo, VidyoInc, Hackensack, New Jersey). At VBCH, two tablet computers were provided to serve as portable platforms to use either in provider conference rooms or to be taken into patient rooms. Twice a day, at 8:45 am and 4:30 pm, virtual hospitalists, local providers, and nursing staff would videoconference and review the status and care plan for all admitted patients. In addition, virtual hospitalists performed a videoconference interview using the tablet computers with all patients on the morning following admission and at other times on an as-needed basis. We asked the virtual hospitalists to cover a minimum of 72 consecutive hours to maintain provider continuity. Local APPs documented the history, examination, and medical decision-making for billing purposes, which were cosigned by the virtual hospitalists. The virtual hospitalists also created separate notes documenting their discussions with local staff, interview and limited direct physical examination findings (eg, appearance of rashes), and medical decision making. Due to limitations of the EMR, local APPs wrote patient orders. All virtual hospitalists were credentialed by proxy at VBCH. We consulted with the UIHC legal team to ensure that virtual hospitalists would be protected under their existing malpractice insurance.

Outcome Measures

Outcome measures were divided into three categories: (1) clinical and utilization outcomes; (2) virtual hospitalist outcomes; and (3) satisfaction outcomes. The primary clinical outcome was the percentage of ED encounters resulting in transfer to a different acute care hospital. We also monitored alternative ED dispositions, including local inpatient admission. Additional clinical and utilization outcomes after ED admission included the mean daily inpatient census at VBCH and the case mix index (CMI). We selected the mean length of stay, the percentage of inpatients transferred to other hospitals, and the inpatient mortality as balance measures due to concerns of increasing the acuity of the inpatient wards beyond the comfort and expertise of local staff. Virtual hospitalist outcomes included the mean daily time commitment and the mean time commitment per patient. Virtual hospitalists self-reported their time commitments as part of their daily documentation. We chose these measures in anticipation of expanding this program to other institutions in the future. Satisfaction outcomes included a weekly survey to all VBCH physicians and nursing staff (Appendix 1), weekly group discussions with virtual hospitalists and CAH staff, and 3 interviews with patients and family members after discharge (Appendix 2).

Statistical Analysis

Baseline data collected over a period of 24 weeks were used to measure pre-implementation performance and trends at VBCH. The virtual hospitalist service was started on November 15, 2016, and the two weeks before and two weeks after this date were excluded from analysis as a transition period. To account for weekend variation, we reported data in consecutive 28-day blocks. We used Chi-square tests to compare proportional outcomes and Student’s t-tests for continuous variables. Statistical Process Control charts were used to evaluate for temporal trends in quantitative data.

Funding

Development of this project was funded through the University of Iowa Hospitalist group and the Signal Center for Health Innovations at UI Health Ventures. Virtual hospitalist clinical time was paid for by the CAH on a fractional basis of a traditional hospitalist based on projected patient volumes through analysis of baseline data. Patients were not directly billed for virtual hospitalist service but were charged for the services provided by CAH providers.

Clinical and Utilization Outcomes

During the 24-week baseline period, VBCH had 947 ED encounters and 176 combined acute inpatient and observation admissions. For the 24 weeks following the transition, there were 930 ED visits and 186 admissions. We observed a 36% (157/947 to 98/930, P < .001) decrease in ED encounters ending in patient transfer to another hospital (Figure). In parallel, VBCH ED visits leading to local admission increased by 62% of baseline (39/947 to 62/930, P = .014). There was no significant change in the fraction of ED encounters resulting in an observation stay (104/947 to 99/930, P = .814). Daily ED visits did not change after virtual hospitalist coverage began (5.64 to 5.54 visits/day, P = .734), but the percentage of ED visits ending in discharge to a nonmedical setting increased from 79.0% to 82.7% (748/947 to 769/930, P = .042).

The implementation did not have a significant impact on ward census or patient complexity (Table 1). Both CMI and mean length of stay did not change after starting the service. The study was underpowered to detect differences in rare events, including inpatient mortality and transfer after admission. Despite the decrease in transfers, inpatient census was unchanged. This coincides with a 17% decrease (196/947 to 160/930, P = .054) in the proportion of ED patients referred for admission either locally or at an outside hospital.

Virtual Hospitalist Outcomes

Satisfaction Outcomes

The staff at VBCH identified several benefits to the virtual hospitalist service. Survey responses (N = 18) were positive, with staff expressing specific gratitude for the additional education and training provided by the virtual hospitalists. On a Likert scale ranging from 1 (very poor) to 5 (excellent), the respondents gave high mean scores to the overall service experience (4.8) and the effectiveness of care delivered (4.9) but were more critical of the ability to keep patients locally (4.5) and the experience with transferring patients (3.9). We also collected free-text feedback from both patients and staff at VBCH (Table 2).

The virtual hospitalist service allowed a higher percentage of acute inpatients to receive care in their local hospital and was positively perceived by providers and patients. The per-patient time commitment by virtual hospitalists was similar to traditional hospitalist coverage¹⁴ and could scale to multiple simultaneous institutions.

Despite the increase in the proportion of patients admitted locally, neither the mean inpatient census nor the complexity of patients (as measured by CMI) increased. The increase in patients admitted locally was offset by a parallel increase in the number of ED patients discharged home. Although virtual hospitalists were available to consult on ED patients, this consultation was not mandatory unless the CAH provider felt that admission was indicated. It remains unclear whether the changes in ED disposition were due to direct intervention by virtual hospitalists, increasing local expertise with inpatient medicine, or unrelated local factors.

Although outside transfers directly from the ED dropped, there was a potential increase in acute inpatients transferred after admission that failed to reach statistical significance. We anticipated increased transfers after admission as a potential consequence of accepting more complex patients for CAH admission. Reasons for transfer included emergent transfers for medically unstable patients and scheduled transfer for subspecialist evaluation or testing. Despite the possible increase in delayed transfers, there was no significant change in CAH inpatient mortality, and the total fraction of combined ED and inpatients transferred decreased after the intervention.

Despite the benefits of keeping patients within their communities, 20%-60% of rural patients bypass their local facilities when seeking emergent care.¹⁵ Despite publicity on local media,¹⁶ we did not observe an increase in daily ED visits after implementation. Although some investigators have found that increasing the services offered decreases in rural bypass,¹⁷ others have found no or mixed effects.^18,19 Further investigations into the local factors contributing to rural bypass may yield important insights, and future implementations should not rely on rapid increases in patient volume to establish economic viability.

Although telemedicine has been applied to a variety of previous settings, to our knowledge, this marks the first collaboration between an academic medical center and a CAH to provide continuous hospitalist coverage. A previous model for pediatric inpatients showed a similar decrease in patients transferred to tertiary centers.²⁰ Virtual hospitalists differ from other adult telemedicine projects, which focused on subspecialty care or overnight coverage.²¹ The advantages of our model include the ability to proactively address deficits, even when local providers are unaware of changes to the standards of care. We believe that mandatory scheduled interactions decreased the barriers to communication and increased provider reassurance in telemedicine management of their patients. The scheduled interactions also provided additional training and development for CAH personnel, were well received by local staff, and may contribute to local provider job satisfaction, retention, and recruitment.

Past efforts to integrate academic hospitalists into CAHs improved quality metrics and provider satisfaction but were economically infeasible due to low patient volumes.²² In contrast, virtual providers can distribute their efforts across multiple areas, including covering additional CAHs, providing local patient care at their home facility, or completing academic projects. By combining two or more CAHs into a single provider, sufficient patient volume can be generated to dedicated personnel.

There were several limitations to this initial investigation:

• As a pilot between two specific institutions, modifications will be required to replicate in other CAHs or academic centers.
• Generating sufficient revenue to cover a full hospitalist salary will require adding additional responsibilities, either covering multiple CAHs simultaneously or combining virtual coverage with in-person responsibilities.
• The accuracy of the self-report remains unmeasured, and the impact of combining two or more CAHs may not be strictly additive. Attempts to supplement the self-reported time spent with additional information from the EMR and cell phone logs were complicated by the use of multiple platforms in parallel, interruptions in provider workflow, and provider multitasking.
• Due to the need for reliable local physical examinations and regulations on telehealth reimbursement, local APPs were necessary for this implementation. Although most of the CAHs have an on-site provider to provide ED coverage, CAHs with sufficient volume to necessitate separating ED and inpatient ward coverage may have difficulty supporting both APP and virtual hospitalist coverage, even on a fractional basis.
• This study was underpowered to detect rare events with significant consequences, including inpatient mortality and inpatient transfer. Although CMI suggests similar complexity in CAH patients, we have insufficient data to draw further comparisons on patient characteristics before and after the intervention.
• The analysis may be vulnerable to secular trends in the CAH patient population, as only 24 weeks of data were used as a baseline for comparison (although no significant seasonal variation was detected during that time). Extending the baseline data to include an additional 30 weeks ED encounters did not significantly alter our conclusions.
• Virtual hospitalists were dependent on physical examinations performed independently by local APPs.
• Although virtual providers were obligated to be available for videoconferencing within 60 minutes, more urgent medical decisions were sometimes made based on phone conferences between VBCH and the virtual hospitalist without video or direct patient assessment.
• We selected a CAH utilizing an identical instance of our EMR. Although this increased the ability of virtual hospitalists to split their time between virtual and local patient encounters, this limits our ability to spread this intervention beyond institutions already partnering with the UIHC.

We succeeded in reducing outside transfers at a CAH by implementing a sustainable virtual hospitalist service. This model allows patients to receive more of their care within their local communities and provides an improved inpatient experience. Next steps include expanding this service to other CAHs within our region, both to understand if this model is applicable beyond our initial site and to monitor for complications induced by scaling. If successful, virtual hospitalist coverage can provide a sustainable solution to providing the latest innovations in hospital medicine even to the most rural communities.

The authors thank Ray Brownsworth, CEO of Van Buren County Hospital, as well as all the providers and staff who worked with them to implement and improve their services. The authors also thank Pat Brophy, founder of The Signal Center for Health Innovation, for providing leadership, support, and resources for innovation.

Disclosures 

None of the authors have identified a conflict of interest in relation to this manuscript.

Funding

This project was funded through the University of Iowa Health Care and the Signal Center for Health Innovations at UI Health Ventures.

Compliance With Ethical Standards

This quality improvement project was exempt from Institutional Review Board review

Through increased involvement with families and caregivers, community hospitals can deliver better healthcare to patients.^1,2 Furthermore, when patients bypass local hospitals and directly present to tertiary care, mortality for time-sensitive illnesses, such as sepsis, increases.³ Unfortunately, although critical access hospitals (CAHs) had an equivalent risk-adjusted mortality in 2002, they have failed to improve their performance at the same rate as that of larger hospitals and lag in quality metrics.^4,5

One potential contributor to the lagging performance may be the low uptake of the hospitalist model at these facilities.⁶ Although dedicated hospitalists have improved patient outcomes and decreased spending in large hospitals,^7-9 implementing the hospitalist medicine model on a smaller scale remains difficult. Approximately 1,300 CAHs provide necessary emergency room and inpatient services in the rural United States.¹⁰ Assuming 12-hour shifts and every-other-week assignments, providing continuous, on-location hospitalist coverage would require more than 10% of the total hospitalist workforce to cover less than 3% of all hospital admissions.^11-13

Telemedicine allows content experts, including hospitalists, to supervise patient care remotely. This provides a potential solution to the logistical challenges of supplying continuous hospitalist coverage to a remote facility with a low daily census. We hypothesized that providing continuous “virtual hospitalist” coverage through telemedicine could increase the ability of a CAH to care for patients locally, decreasing the number of transfers to tertiary care centers and improving patient and provider satisfaction. We aimed to create a 25% relative reduction in CAH Emergency Department (ED) patient encounters resulting in transfer to outside hospitals within 6 months.

This quality improvement project was exempt from Institutional Review Board review.

Setting

The University of Iowa Hospitals and Clinics (UIHC) is a 750-bed teaching hospital based in a suburban community in Eastern Iowa and the only tertiary care hospital in the state of Iowa. The UIHC Hospitalist group contains 44 staff physicians and covers more than 12 service lines (both faculty-only and resident-covered) at this facility.

Van Buren County Hospital (VBCH) is a 24-bed CAH offering emergency, internal medicine, and obstetrical services and located 80 miles southwest of UIHC. X-ray and CT scan services are available continuously, but ultrasound and magnetic resonance imaging services are available only 2-3 times per week. While tertiary care patients were transferred to UIHC, patients requiring specialty care but with less complex illnesses (eg, stable myocardial infarction) were referred to closer facilities.

Prior to implementation, coverage of the acute inpatient ward and the emergency room at VBCH was simultaneously provided by a single physician or advanced practice providers (APPs). When APPs provided coverage, a physician was required to be notified of any new admissions and was immediately available for medical emergencies. The VBCH providers worked alone in 48- to 72-hour continuous shifts as the sole coverage for both ED and inpatient units. It was frequently necessary to bring in outside providers through locum tenens agencies to fill gaps in the provider schedule. Both VBCH and UIHC used a shared electronic medical record (EMR), which was a key consideration in choosing VBCH as our pilot site. Providers at both institutions had access to identical patient information through the EMR, including radiology images, laboratory results, and provider notes.

Intervention Development and Implementation

A site visit by clinical and administrative project leads to VBCH identified three deficits that we could address through telemedicine: (1) The extended duration of VBCH shifts was detrimental to provider experience and retention; (2) Lack of local expertise in hospital medicine led to limited comfort in caring for patients with stable but medically complex conditions (eg, drug-resistant urinary tract infection); and (3) Patient transitions between VBCH and UIHC during acute care transfer were frustrating and led to negative experiences with providers and patients.

We developed a model to address these deficits using the minimum number of specialties and employees to facilitate rapid implementation. Although local care ED and inpatient care was provided by 3 APPS and a single physician provider, we mandated the coverage of all acute inpatients by the virtual hospitalists. This coverage included daily videoconference patient rounds, continuous pager coverage for new acute issues, and listing the virtual hospitalists as the attending of record for patient admissions. We scheduled contact times in the morning and afternoon to accelerate familiarity and comfort with the technology. We used a secure, Health Insurance Portability and Accountability Act of 1996 (HIPAA)-compliant platform for videoconferencing, accessible through personal computers or portable smart devices (Vidyo, VidyoInc, Hackensack, New Jersey). At VBCH, two tablet computers were provided to serve as portable platforms to use either in provider conference rooms or to be taken into patient rooms. Twice a day, at 8:45 am and 4:30 pm, virtual hospitalists, local providers, and nursing staff would videoconference and review the status and care plan for all admitted patients. In addition, virtual hospitalists performed a videoconference interview using the tablet computers with all patients on the morning following admission and at other times on an as-needed basis. We asked the virtual hospitalists to cover a minimum of 72 consecutive hours to maintain provider continuity. Local APPs documented the history, examination, and medical decision-making for billing purposes, which were cosigned by the virtual hospitalists. The virtual hospitalists also created separate notes documenting their discussions with local staff, interview and limited direct physical examination findings (eg, appearance of rashes), and medical decision making. Due to limitations of the EMR, local APPs wrote patient orders. All virtual hospitalists were credentialed by proxy at VBCH. We consulted with the UIHC legal team to ensure that virtual hospitalists would be protected under their existing malpractice insurance.

Outcome Measures

Outcome measures were divided into three categories: (1) clinical and utilization outcomes; (2) virtual hospitalist outcomes; and (3) satisfaction outcomes. The primary clinical outcome was the percentage of ED encounters resulting in transfer to a different acute care hospital. We also monitored alternative ED dispositions, including local inpatient admission. Additional clinical and utilization outcomes after ED admission included the mean daily inpatient census at VBCH and the case mix index (CMI). We selected the mean length of stay, the percentage of inpatients transferred to other hospitals, and the inpatient mortality as balance measures due to concerns of increasing the acuity of the inpatient wards beyond the comfort and expertise of local staff. Virtual hospitalist outcomes included the mean daily time commitment and the mean time commitment per patient. Virtual hospitalists self-reported their time commitments as part of their daily documentation. We chose these measures in anticipation of expanding this program to other institutions in the future. Satisfaction outcomes included a weekly survey to all VBCH physicians and nursing staff (Appendix 1), weekly group discussions with virtual hospitalists and CAH staff, and 3 interviews with patients and family members after discharge (Appendix 2).

Statistical Analysis

Baseline data collected over a period of 24 weeks were used to measure pre-implementation performance and trends at VBCH. The virtual hospitalist service was started on November 15, 2016, and the two weeks before and two weeks after this date were excluded from analysis as a transition period. To account for weekend variation, we reported data in consecutive 28-day blocks. We used Chi-square tests to compare proportional outcomes and Student’s t-tests for continuous variables. Statistical Process Control charts were used to evaluate for temporal trends in quantitative data.

Funding

Development of this project was funded through the University of Iowa Hospitalist group and the Signal Center for Health Innovations at UI Health Ventures. Virtual hospitalist clinical time was paid for by the CAH on a fractional basis of a traditional hospitalist based on projected patient volumes through analysis of baseline data. Patients were not directly billed for virtual hospitalist service but were charged for the services provided by CAH providers.

Clinical and Utilization Outcomes

During the 24-week baseline period, VBCH had 947 ED encounters and 176 combined acute inpatient and observation admissions. For the 24 weeks following the transition, there were 930 ED visits and 186 admissions. We observed a 36% (157/947 to 98/930, P < .001) decrease in ED encounters ending in patient transfer to another hospital (Figure). In parallel, VBCH ED visits leading to local admission increased by 62% of baseline (39/947 to 62/930, P = .014). There was no significant change in the fraction of ED encounters resulting in an observation stay (104/947 to 99/930, P = .814). Daily ED visits did not change after virtual hospitalist coverage began (5.64 to 5.54 visits/day, P = .734), but the percentage of ED visits ending in discharge to a nonmedical setting increased from 79.0% to 82.7% (748/947 to 769/930, P = .042).

The implementation did not have a significant impact on ward census or patient complexity (Table 1). Both CMI and mean length of stay did not change after starting the service. The study was underpowered to detect differences in rare events, including inpatient mortality and transfer after admission. Despite the decrease in transfers, inpatient census was unchanged. This coincides with a 17% decrease (196/947 to 160/930, P = .054) in the proportion of ED patients referred for admission either locally or at an outside hospital.

Virtual Hospitalist Outcomes

Satisfaction Outcomes

The staff at VBCH identified several benefits to the virtual hospitalist service. Survey responses (N = 18) were positive, with staff expressing specific gratitude for the additional education and training provided by the virtual hospitalists. On a Likert scale ranging from 1 (very poor) to 5 (excellent), the respondents gave high mean scores to the overall service experience (4.8) and the effectiveness of care delivered (4.9) but were more critical of the ability to keep patients locally (4.5) and the experience with transferring patients (3.9). We also collected free-text feedback from both patients and staff at VBCH (Table 2).

The virtual hospitalist service allowed a higher percentage of acute inpatients to receive care in their local hospital and was positively perceived by providers and patients. The per-patient time commitment by virtual hospitalists was similar to traditional hospitalist coverage¹⁴ and could scale to multiple simultaneous institutions.

Despite the increase in the proportion of patients admitted locally, neither the mean inpatient census nor the complexity of patients (as measured by CMI) increased. The increase in patients admitted locally was offset by a parallel increase in the number of ED patients discharged home. Although virtual hospitalists were available to consult on ED patients, this consultation was not mandatory unless the CAH provider felt that admission was indicated. It remains unclear whether the changes in ED disposition were due to direct intervention by virtual hospitalists, increasing local expertise with inpatient medicine, or unrelated local factors.

Although outside transfers directly from the ED dropped, there was a potential increase in acute inpatients transferred after admission that failed to reach statistical significance. We anticipated increased transfers after admission as a potential consequence of accepting more complex patients for CAH admission. Reasons for transfer included emergent transfers for medically unstable patients and scheduled transfer for subspecialist evaluation or testing. Despite the possible increase in delayed transfers, there was no significant change in CAH inpatient mortality, and the total fraction of combined ED and inpatients transferred decreased after the intervention.

Despite the benefits of keeping patients within their communities, 20%-60% of rural patients bypass their local facilities when seeking emergent care.¹⁵ Despite publicity on local media,¹⁶ we did not observe an increase in daily ED visits after implementation. Although some investigators have found that increasing the services offered decreases in rural bypass,¹⁷ others have found no or mixed effects.^18,19 Further investigations into the local factors contributing to rural bypass may yield important insights, and future implementations should not rely on rapid increases in patient volume to establish economic viability.

Although telemedicine has been applied to a variety of previous settings, to our knowledge, this marks the first collaboration between an academic medical center and a CAH to provide continuous hospitalist coverage. A previous model for pediatric inpatients showed a similar decrease in patients transferred to tertiary centers.²⁰ Virtual hospitalists differ from other adult telemedicine projects, which focused on subspecialty care or overnight coverage.²¹ The advantages of our model include the ability to proactively address deficits, even when local providers are unaware of changes to the standards of care. We believe that mandatory scheduled interactions decreased the barriers to communication and increased provider reassurance in telemedicine management of their patients. The scheduled interactions also provided additional training and development for CAH personnel, were well received by local staff, and may contribute to local provider job satisfaction, retention, and recruitment.

Past efforts to integrate academic hospitalists into CAHs improved quality metrics and provider satisfaction but were economically infeasible due to low patient volumes.²² In contrast, virtual providers can distribute their efforts across multiple areas, including covering additional CAHs, providing local patient care at their home facility, or completing academic projects. By combining two or more CAHs into a single provider, sufficient patient volume can be generated to dedicated personnel.

There were several limitations to this initial investigation:

• As a pilot between two specific institutions, modifications will be required to replicate in other CAHs or academic centers.
• Generating sufficient revenue to cover a full hospitalist salary will require adding additional responsibilities, either covering multiple CAHs simultaneously or combining virtual coverage with in-person responsibilities.
• The accuracy of the self-report remains unmeasured, and the impact of combining two or more CAHs may not be strictly additive. Attempts to supplement the self-reported time spent with additional information from the EMR and cell phone logs were complicated by the use of multiple platforms in parallel, interruptions in provider workflow, and provider multitasking.
• Due to the need for reliable local physical examinations and regulations on telehealth reimbursement, local APPs were necessary for this implementation. Although most of the CAHs have an on-site provider to provide ED coverage, CAHs with sufficient volume to necessitate separating ED and inpatient ward coverage may have difficulty supporting both APP and virtual hospitalist coverage, even on a fractional basis.
• This study was underpowered to detect rare events with significant consequences, including inpatient mortality and inpatient transfer. Although CMI suggests similar complexity in CAH patients, we have insufficient data to draw further comparisons on patient characteristics before and after the intervention.
• The analysis may be vulnerable to secular trends in the CAH patient population, as only 24 weeks of data were used as a baseline for comparison (although no significant seasonal variation was detected during that time). Extending the baseline data to include an additional 30 weeks ED encounters did not significantly alter our conclusions.
• Virtual hospitalists were dependent on physical examinations performed independently by local APPs.
• Although virtual providers were obligated to be available for videoconferencing within 60 minutes, more urgent medical decisions were sometimes made based on phone conferences between VBCH and the virtual hospitalist without video or direct patient assessment.
• We selected a CAH utilizing an identical instance of our EMR. Although this increased the ability of virtual hospitalists to split their time between virtual and local patient encounters, this limits our ability to spread this intervention beyond institutions already partnering with the UIHC.

We succeeded in reducing outside transfers at a CAH by implementing a sustainable virtual hospitalist service. This model allows patients to receive more of their care within their local communities and provides an improved inpatient experience. Next steps include expanding this service to other CAHs within our region, both to understand if this model is applicable beyond our initial site and to monitor for complications induced by scaling. If successful, virtual hospitalist coverage can provide a sustainable solution to providing the latest innovations in hospital medicine even to the most rural communities.

The authors thank Ray Brownsworth, CEO of Van Buren County Hospital, as well as all the providers and staff who worked with them to implement and improve their services. The authors also thank Pat Brophy, founder of The Signal Center for Health Innovation, for providing leadership, support, and resources for innovation.

Disclosures 

None of the authors have identified a conflict of interest in relation to this manuscript.

Funding

This project was funded through the University of Iowa Health Care and the Signal Center for Health Innovations at UI Health Ventures.

Compliance With Ethical Standards

This quality improvement project was exempt from Institutional Review Board review

Cardiac complications account for at least one-third of perioperative deaths, and lead to substantial morbidity and cost.^1-4 Current guidelines recommend that patients undergo assessment of cardiac risk and functional status prior to noncardiac surgery.⁵ Preoperative cardiac stress testing is recommended for patients whose predicted cardiac risk exceeds 1%, whose functional status is limited, and for whom testing may change management.⁵

However, patients are not specifically selected according to risk of coronary artery disease (CAD) in current guidelines. The pretest probability of CAD may vary widely in this patient population, and the resultant accuracy of cardiac stress testing in making the diagnosis of CAD may vary as well.⁵ Meanwhile, CAD is a clear risk factor for perioperative cardiac events.^6-8

Because the pretest probability of CAD is heterogeneous, the optimal modality of cardiac stress testing in this population is unclear. False-positive results would likely lead to inflated estimates of operative risk, expensive and high-risk downstream testing, and potentially cancellation of otherwise beneficial surgeries. Meanwhile, false-negative results would lead to overly optimistic estimates of surgical risk and potentially to surgical intervention at higher levels of risk than would be desirable. Current guidelines leave the selection of either dobutamine stress echocardiography (DSE) or pharmacological stress myocardial perfusion imaging to the clinician.⁵ To inform decisions regarding the selection of cardiac stress testing modality prior to noncardiac surgery, we conducted this study to estimate the diagnostic accuracy of DSE and single-photon emission computed tomography (SPECT) among this patient population.

Surgical Cohort

The American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) samples patients undergoing surgery at participating hospitals and collects standardized clinical data on preoperative risk factors and postoperative complications.⁹ We acquired public use data from the 2009 NSQIP cohort, which included more than 336,000 surgical cases from 237 hospitals (principally in the United States). We excluded from our analysis patients undergoing cardiac surgery, patients with a prior diagnosis of CAD, and patients undergoing experimental surgeries. This left a sample of 300,462 for analysis.

Prediction of Dyslipidemia

The model we used to predict the presence of obstructive CAD required the presence or absence of dyslipidemia. A number of variables are common to both NSQIP and the National Health and Nutrition Examination Survey (NHANES), including age, weight, sex, tobacco use, diabetes, and prior stroke.¹⁰ Using those common variables, we developed a logistic regression to predict a diagnosis of dyslipidemia, applied that regression to the NSQIP cohort, and dichotomized. To assess the potential impact of misclassification, we performed separate sensitivity analyses in which either no patients or all patients had dyslipidemia.

Prediction of Obstructive CAD

To estimate the probability of obstructive CAD, we applied the risk prediction tool currently recommended by the European Society of Cardiology.¹¹ The clinical version of this tool relies on age; sex; diagnoses of diabetes, hypertension, and dyslipidemia; active tobacco use; and chest pain characteristics to predict the probability of obstructive CAD on coronary angiography. We assumed that all patients in our cohort had nonspecific chest pain, the referent in the calculator.

Prediction of Perioperative Event Risk

To predict the probability of a perioperative cardiac event, we used the Myocardial Infarction or Cardiac Arrest (MICA) calculator, which was derived from an earlier cohort of NSQIP.¹² All variables required for this prediction tool were included in the 2009 NSQIP cohort; our categorization of surgeries is included as an online appendix. MICA is one of three prediction tools included in the current American College of Cardiology/American Heart Association (ACC/AHA) guidelines.⁵

Prediction of Test Accuracy

We searched the MEDLINE database for estimates of the test characteristics of DSE and SPECT that adjusted for workup bias.¹³ (Also known as sequential-ordering bias, here we refer to the phenomenon whereby further workup is based on the results of diagnostic testing, resulting in underdiagnosis among patients with negative tests and falsely high estimates of sensitivity.¹⁴) Although other modalities of myocardial perfusion imaging exist, SPECT appears to be the most widely available, utilized, and studied modality of MPI.¹⁵ Our search strategy paired (“Sensitivity and Specificity” [MeSH Terms] AND “Coronary Disease/diagnostic imaging” [MAJR] AND “bias” [TIAB]) with (“Tomography, Emission-Computed, Single-Photon” [MAJR] OR “Echocardiography, Stress” [MAJR]). We reviewed the results for sensitivity and specificity estimates that corrected for workup bias. For each of SPECT and DSE, we drew the sensitivity and specificity from normal distributions based on literature estimates (see Table). We then calculated the expected accuracy of each modality for each patient in our dataset. All analyses were performed in Stata (version 14, College Station, Texas).

The median predicted probability of obstructive CAD was 5.1% (IQR: 1.8%-13.9%). Among patients with a predicted risk of a perioperative event of 1% or greater, the median probability of obstructive CAD was 18.1% (IQR: 9.6%-32.3%). The correlation between the predicted probabilities of CAD and a perioperative event was low (0.32), but highly statistically significant (P < .001).

Both accuracy and PPV were higher for DSE than for SPECT. The predicted accuracy of DSE was greater than that of SPECT in 73.5% of cases overall and in 60.5% of cases with a predicted operative cardiac risk greater than 1%. The mean PPV of DSE was 32.9% (median: 26.7%), while the equivalent PPVs for SPECT were 14.1% and 8.2%, respectively. Among cases with a predicted operative cardiac risk greater than 1%, the mean PPV of DSE was 57.5% (median: 60.2%), while the equivalent PPVs for SPECT were 29.8% and 26.7%, respectively.

DSE had a mean predicted accuracy of 93.0% (median: 96.2%), while SPECT had a mean accuracy of 92.6% (median: 95.6%). The predicted accuracies of DSE and SPECT are shown in the Figure, stratified by predicted perioperative risk across the 1% risk threshold currently used by ACC/AHA guidelines.

In this study, we demonstrated that the expected accuracy of DSE in the diagnosis of obstructive CAD among patients undergoing noncardiac surgery is higher than that of SPECT. This finding was true in both unselected patients and those selected by a perioperative risk of greater than 1%. The use of SPECT, compared with DSE, would likely result in greater numbers of false-positive tests in this patient population and less accurate results overall.

Cardiac stress testing, as with any diagnostic test, is most useful at intermediate probabilities. Insofar as stress testing offers diagnostic value, our analysis suggests that, in the range of the predicted risk of CAD found in patients undergoing noncardiac surgery, DSE is a more efficient testing strategy. To the extent that making a diagnosis of CAD informs the decision to proceed to surgery, a more accurate test would be preferable. The lower cost of DSE, the lack of ionizing radiation, and the information provided by echocardiography regarding diagnoses other than CAD, if considered, would further amplify that preference.

However, it is important to note that both modalities have limited positive predictive value. In the median patient who meets the currently recommended 1% perioperative event risk threshold, SPECT would lead to 2.74 false positive results for every true positive result. DSE would produce approximately two false positive results for every three true positive results. If lower rates of false-positive testing are desired, different patient selection criteria are required.

A few key limitations of this work warrant discussion. First, our results likely overestimate the probability of obstructive CAD in this population. We assumed that all patients have nonspecific chest pain at the time of the preoperative evaluation, though many patients do not, in fact, have chest pain. Tools to estimate the pretest probability of CAD (such as the ESC tool that we used or the older Diamond-Forrester prediction) are intended to stratify higher-risk patients than are seen in a preoperative setting. If asymptomatic patients seen in preoperative risk assessment clinics have lower risk of CAD than what we have predicted, we will have understated the case for DSE. Moreover, cases sampled from hospitals participating in NSQIP are not representative of the national surgical population. This likely further inflates our estimates of CAD risk compared with the “true” surgical population. Finally, we cannot comment on current practice from these data. Current guidelines recommend preoperative cardiac stress testing for patients whose risk of a perioperative cardiac event exceeds 1%, whose functional status is poor or unknown, and only if said testing will change management.⁵ Using these data, we cannot determine the pretest probability of patients referred for stress testing before noncardiac surgery.

Still, this analysis suggests that, among patients undergoing noncardiac surgery, selecting patients according to the risk of perioperative events would result in a population at an overall comparatively low risk of CAD, and that in this population, DSE would be more accurate than SPECT for making the diagnosis of CAD. If a diagnosis of CAD would change the decision to proceed to surgery, DSE is likely to be a more efficient test than SPECT.

Acknowledgements

The authors would like to thank Wael Jaber, MD, for his thoughtful comments on a draft of this manuscript.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

Cardiac complications account for at least one-third of perioperative deaths, and lead to substantial morbidity and cost.^1-4 Current guidelines recommend that patients undergo assessment of cardiac risk and functional status prior to noncardiac surgery.⁵ Preoperative cardiac stress testing is recommended for patients whose predicted cardiac risk exceeds 1%, whose functional status is limited, and for whom testing may change management.⁵

However, patients are not specifically selected according to risk of coronary artery disease (CAD) in current guidelines. The pretest probability of CAD may vary widely in this patient population, and the resultant accuracy of cardiac stress testing in making the diagnosis of CAD may vary as well.⁵ Meanwhile, CAD is a clear risk factor for perioperative cardiac events.^6-8

Because the pretest probability of CAD is heterogeneous, the optimal modality of cardiac stress testing in this population is unclear. False-positive results would likely lead to inflated estimates of operative risk, expensive and high-risk downstream testing, and potentially cancellation of otherwise beneficial surgeries. Meanwhile, false-negative results would lead to overly optimistic estimates of surgical risk and potentially to surgical intervention at higher levels of risk than would be desirable. Current guidelines leave the selection of either dobutamine stress echocardiography (DSE) or pharmacological stress myocardial perfusion imaging to the clinician.⁵ To inform decisions regarding the selection of cardiac stress testing modality prior to noncardiac surgery, we conducted this study to estimate the diagnostic accuracy of DSE and single-photon emission computed tomography (SPECT) among this patient population.

Surgical Cohort

The American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) samples patients undergoing surgery at participating hospitals and collects standardized clinical data on preoperative risk factors and postoperative complications.⁹ We acquired public use data from the 2009 NSQIP cohort, which included more than 336,000 surgical cases from 237 hospitals (principally in the United States). We excluded from our analysis patients undergoing cardiac surgery, patients with a prior diagnosis of CAD, and patients undergoing experimental surgeries. This left a sample of 300,462 for analysis.

Prediction of Dyslipidemia

The model we used to predict the presence of obstructive CAD required the presence or absence of dyslipidemia. A number of variables are common to both NSQIP and the National Health and Nutrition Examination Survey (NHANES), including age, weight, sex, tobacco use, diabetes, and prior stroke.¹⁰ Using those common variables, we developed a logistic regression to predict a diagnosis of dyslipidemia, applied that regression to the NSQIP cohort, and dichotomized. To assess the potential impact of misclassification, we performed separate sensitivity analyses in which either no patients or all patients had dyslipidemia.

Prediction of Obstructive CAD

To estimate the probability of obstructive CAD, we applied the risk prediction tool currently recommended by the European Society of Cardiology.¹¹ The clinical version of this tool relies on age; sex; diagnoses of diabetes, hypertension, and dyslipidemia; active tobacco use; and chest pain characteristics to predict the probability of obstructive CAD on coronary angiography. We assumed that all patients in our cohort had nonspecific chest pain, the referent in the calculator.

Prediction of Perioperative Event Risk

To predict the probability of a perioperative cardiac event, we used the Myocardial Infarction or Cardiac Arrest (MICA) calculator, which was derived from an earlier cohort of NSQIP.¹² All variables required for this prediction tool were included in the 2009 NSQIP cohort; our categorization of surgeries is included as an online appendix. MICA is one of three prediction tools included in the current American College of Cardiology/American Heart Association (ACC/AHA) guidelines.⁵

Prediction of Test Accuracy

We searched the MEDLINE database for estimates of the test characteristics of DSE and SPECT that adjusted for workup bias.¹³ (Also known as sequential-ordering bias, here we refer to the phenomenon whereby further workup is based on the results of diagnostic testing, resulting in underdiagnosis among patients with negative tests and falsely high estimates of sensitivity.¹⁴) Although other modalities of myocardial perfusion imaging exist, SPECT appears to be the most widely available, utilized, and studied modality of MPI.¹⁵ Our search strategy paired (“Sensitivity and Specificity” [MeSH Terms] AND “Coronary Disease/diagnostic imaging” [MAJR] AND “bias” [TIAB]) with (“Tomography, Emission-Computed, Single-Photon” [MAJR] OR “Echocardiography, Stress” [MAJR]). We reviewed the results for sensitivity and specificity estimates that corrected for workup bias. For each of SPECT and DSE, we drew the sensitivity and specificity from normal distributions based on literature estimates (see Table). We then calculated the expected accuracy of each modality for each patient in our dataset. All analyses were performed in Stata (version 14, College Station, Texas).

The median predicted probability of obstructive CAD was 5.1% (IQR: 1.8%-13.9%). Among patients with a predicted risk of a perioperative event of 1% or greater, the median probability of obstructive CAD was 18.1% (IQR: 9.6%-32.3%). The correlation between the predicted probabilities of CAD and a perioperative event was low (0.32), but highly statistically significant (P < .001).

Both accuracy and PPV were higher for DSE than for SPECT. The predicted accuracy of DSE was greater than that of SPECT in 73.5% of cases overall and in 60.5% of cases with a predicted operative cardiac risk greater than 1%. The mean PPV of DSE was 32.9% (median: 26.7%), while the equivalent PPVs for SPECT were 14.1% and 8.2%, respectively. Among cases with a predicted operative cardiac risk greater than 1%, the mean PPV of DSE was 57.5% (median: 60.2%), while the equivalent PPVs for SPECT were 29.8% and 26.7%, respectively.

DSE had a mean predicted accuracy of 93.0% (median: 96.2%), while SPECT had a mean accuracy of 92.6% (median: 95.6%). The predicted accuracies of DSE and SPECT are shown in the Figure, stratified by predicted perioperative risk across the 1% risk threshold currently used by ACC/AHA guidelines.

In this study, we demonstrated that the expected accuracy of DSE in the diagnosis of obstructive CAD among patients undergoing noncardiac surgery is higher than that of SPECT. This finding was true in both unselected patients and those selected by a perioperative risk of greater than 1%. The use of SPECT, compared with DSE, would likely result in greater numbers of false-positive tests in this patient population and less accurate results overall.

Cardiac stress testing, as with any diagnostic test, is most useful at intermediate probabilities. Insofar as stress testing offers diagnostic value, our analysis suggests that, in the range of the predicted risk of CAD found in patients undergoing noncardiac surgery, DSE is a more efficient testing strategy. To the extent that making a diagnosis of CAD informs the decision to proceed to surgery, a more accurate test would be preferable. The lower cost of DSE, the lack of ionizing radiation, and the information provided by echocardiography regarding diagnoses other than CAD, if considered, would further amplify that preference.

However, it is important to note that both modalities have limited positive predictive value. In the median patient who meets the currently recommended 1% perioperative event risk threshold, SPECT would lead to 2.74 false positive results for every true positive result. DSE would produce approximately two false positive results for every three true positive results. If lower rates of false-positive testing are desired, different patient selection criteria are required.

A few key limitations of this work warrant discussion. First, our results likely overestimate the probability of obstructive CAD in this population. We assumed that all patients have nonspecific chest pain at the time of the preoperative evaluation, though many patients do not, in fact, have chest pain. Tools to estimate the pretest probability of CAD (such as the ESC tool that we used or the older Diamond-Forrester prediction) are intended to stratify higher-risk patients than are seen in a preoperative setting. If asymptomatic patients seen in preoperative risk assessment clinics have lower risk of CAD than what we have predicted, we will have understated the case for DSE. Moreover, cases sampled from hospitals participating in NSQIP are not representative of the national surgical population. This likely further inflates our estimates of CAD risk compared with the “true” surgical population. Finally, we cannot comment on current practice from these data. Current guidelines recommend preoperative cardiac stress testing for patients whose risk of a perioperative cardiac event exceeds 1%, whose functional status is poor or unknown, and only if said testing will change management.⁵ Using these data, we cannot determine the pretest probability of patients referred for stress testing before noncardiac surgery.

Still, this analysis suggests that, among patients undergoing noncardiac surgery, selecting patients according to the risk of perioperative events would result in a population at an overall comparatively low risk of CAD, and that in this population, DSE would be more accurate than SPECT for making the diagnosis of CAD. If a diagnosis of CAD would change the decision to proceed to surgery, DSE is likely to be a more efficient test than SPECT.

Acknowledgements

The authors would like to thank Wael Jaber, MD, for his thoughtful comments on a draft of this manuscript.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

Cardiac complications account for at least one-third of perioperative deaths, and lead to substantial morbidity and cost.^1-4 Current guidelines recommend that patients undergo assessment of cardiac risk and functional status prior to noncardiac surgery.⁵ Preoperative cardiac stress testing is recommended for patients whose predicted cardiac risk exceeds 1%, whose functional status is limited, and for whom testing may change management.⁵

However, patients are not specifically selected according to risk of coronary artery disease (CAD) in current guidelines. The pretest probability of CAD may vary widely in this patient population, and the resultant accuracy of cardiac stress testing in making the diagnosis of CAD may vary as well.⁵ Meanwhile, CAD is a clear risk factor for perioperative cardiac events.^6-8

Because the pretest probability of CAD is heterogeneous, the optimal modality of cardiac stress testing in this population is unclear. False-positive results would likely lead to inflated estimates of operative risk, expensive and high-risk downstream testing, and potentially cancellation of otherwise beneficial surgeries. Meanwhile, false-negative results would lead to overly optimistic estimates of surgical risk and potentially to surgical intervention at higher levels of risk than would be desirable. Current guidelines leave the selection of either dobutamine stress echocardiography (DSE) or pharmacological stress myocardial perfusion imaging to the clinician.⁵ To inform decisions regarding the selection of cardiac stress testing modality prior to noncardiac surgery, we conducted this study to estimate the diagnostic accuracy of DSE and single-photon emission computed tomography (SPECT) among this patient population.

Surgical Cohort

The American College of Surgeons’ National Surgical Quality Improvement Program (NSQIP) samples patients undergoing surgery at participating hospitals and collects standardized clinical data on preoperative risk factors and postoperative complications.⁹ We acquired public use data from the 2009 NSQIP cohort, which included more than 336,000 surgical cases from 237 hospitals (principally in the United States). We excluded from our analysis patients undergoing cardiac surgery, patients with a prior diagnosis of CAD, and patients undergoing experimental surgeries. This left a sample of 300,462 for analysis.

Prediction of Dyslipidemia

The model we used to predict the presence of obstructive CAD required the presence or absence of dyslipidemia. A number of variables are common to both NSQIP and the National Health and Nutrition Examination Survey (NHANES), including age, weight, sex, tobacco use, diabetes, and prior stroke.¹⁰ Using those common variables, we developed a logistic regression to predict a diagnosis of dyslipidemia, applied that regression to the NSQIP cohort, and dichotomized. To assess the potential impact of misclassification, we performed separate sensitivity analyses in which either no patients or all patients had dyslipidemia.

Prediction of Obstructive CAD

To estimate the probability of obstructive CAD, we applied the risk prediction tool currently recommended by the European Society of Cardiology.¹¹ The clinical version of this tool relies on age; sex; diagnoses of diabetes, hypertension, and dyslipidemia; active tobacco use; and chest pain characteristics to predict the probability of obstructive CAD on coronary angiography. We assumed that all patients in our cohort had nonspecific chest pain, the referent in the calculator.

Prediction of Perioperative Event Risk

To predict the probability of a perioperative cardiac event, we used the Myocardial Infarction or Cardiac Arrest (MICA) calculator, which was derived from an earlier cohort of NSQIP.¹² All variables required for this prediction tool were included in the 2009 NSQIP cohort; our categorization of surgeries is included as an online appendix. MICA is one of three prediction tools included in the current American College of Cardiology/American Heart Association (ACC/AHA) guidelines.⁵

Prediction of Test Accuracy

We searched the MEDLINE database for estimates of the test characteristics of DSE and SPECT that adjusted for workup bias.¹³ (Also known as sequential-ordering bias, here we refer to the phenomenon whereby further workup is based on the results of diagnostic testing, resulting in underdiagnosis among patients with negative tests and falsely high estimates of sensitivity.¹⁴) Although other modalities of myocardial perfusion imaging exist, SPECT appears to be the most widely available, utilized, and studied modality of MPI.¹⁵ Our search strategy paired (“Sensitivity and Specificity” [MeSH Terms] AND “Coronary Disease/diagnostic imaging” [MAJR] AND “bias” [TIAB]) with (“Tomography, Emission-Computed, Single-Photon” [MAJR] OR “Echocardiography, Stress” [MAJR]). We reviewed the results for sensitivity and specificity estimates that corrected for workup bias. For each of SPECT and DSE, we drew the sensitivity and specificity from normal distributions based on literature estimates (see Table). We then calculated the expected accuracy of each modality for each patient in our dataset. All analyses were performed in Stata (version 14, College Station, Texas).

The median predicted probability of obstructive CAD was 5.1% (IQR: 1.8%-13.9%). Among patients with a predicted risk of a perioperative event of 1% or greater, the median probability of obstructive CAD was 18.1% (IQR: 9.6%-32.3%). The correlation between the predicted probabilities of CAD and a perioperative event was low (0.32), but highly statistically significant (P < .001).

Both accuracy and PPV were higher for DSE than for SPECT. The predicted accuracy of DSE was greater than that of SPECT in 73.5% of cases overall and in 60.5% of cases with a predicted operative cardiac risk greater than 1%. The mean PPV of DSE was 32.9% (median: 26.7%), while the equivalent PPVs for SPECT were 14.1% and 8.2%, respectively. Among cases with a predicted operative cardiac risk greater than 1%, the mean PPV of DSE was 57.5% (median: 60.2%), while the equivalent PPVs for SPECT were 29.8% and 26.7%, respectively.

DSE had a mean predicted accuracy of 93.0% (median: 96.2%), while SPECT had a mean accuracy of 92.6% (median: 95.6%). The predicted accuracies of DSE and SPECT are shown in the Figure, stratified by predicted perioperative risk across the 1% risk threshold currently used by ACC/AHA guidelines.

In this study, we demonstrated that the expected accuracy of DSE in the diagnosis of obstructive CAD among patients undergoing noncardiac surgery is higher than that of SPECT. This finding was true in both unselected patients and those selected by a perioperative risk of greater than 1%. The use of SPECT, compared with DSE, would likely result in greater numbers of false-positive tests in this patient population and less accurate results overall.

Cardiac stress testing, as with any diagnostic test, is most useful at intermediate probabilities. Insofar as stress testing offers diagnostic value, our analysis suggests that, in the range of the predicted risk of CAD found in patients undergoing noncardiac surgery, DSE is a more efficient testing strategy. To the extent that making a diagnosis of CAD informs the decision to proceed to surgery, a more accurate test would be preferable. The lower cost of DSE, the lack of ionizing radiation, and the information provided by echocardiography regarding diagnoses other than CAD, if considered, would further amplify that preference.

However, it is important to note that both modalities have limited positive predictive value. In the median patient who meets the currently recommended 1% perioperative event risk threshold, SPECT would lead to 2.74 false positive results for every true positive result. DSE would produce approximately two false positive results for every three true positive results. If lower rates of false-positive testing are desired, different patient selection criteria are required.

A few key limitations of this work warrant discussion. First, our results likely overestimate the probability of obstructive CAD in this population. We assumed that all patients have nonspecific chest pain at the time of the preoperative evaluation, though many patients do not, in fact, have chest pain. Tools to estimate the pretest probability of CAD (such as the ESC tool that we used or the older Diamond-Forrester prediction) are intended to stratify higher-risk patients than are seen in a preoperative setting. If asymptomatic patients seen in preoperative risk assessment clinics have lower risk of CAD than what we have predicted, we will have understated the case for DSE. Moreover, cases sampled from hospitals participating in NSQIP are not representative of the national surgical population. This likely further inflates our estimates of CAD risk compared with the “true” surgical population. Finally, we cannot comment on current practice from these data. Current guidelines recommend preoperative cardiac stress testing for patients whose risk of a perioperative cardiac event exceeds 1%, whose functional status is poor or unknown, and only if said testing will change management.⁵ Using these data, we cannot determine the pretest probability of patients referred for stress testing before noncardiac surgery.

Still, this analysis suggests that, among patients undergoing noncardiac surgery, selecting patients according to the risk of perioperative events would result in a population at an overall comparatively low risk of CAD, and that in this population, DSE would be more accurate than SPECT for making the diagnosis of CAD. If a diagnosis of CAD would change the decision to proceed to surgery, DSE is likely to be a more efficient test than SPECT.

Acknowledgements

The authors would like to thank Wael Jaber, MD, for his thoughtful comments on a draft of this manuscript.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no specific funding for this work.

While the use of pediatric readmission rates as a quality metric remains controversial, pediatric hospital-to-home transitions need improvement.¹ As many as a third of pediatric readmissions are preventable,² but the multifactorial and complex nature of factors that contribute to pediatric readmissions presents a challenge in tackling readmission. Several factors are associated with increased risk of readmission; these factors include both clinical and sociodemographic characteristics;³ however, we still have much to learn. Further, the only large trial of an intervention to prevent pediatric readmissions across all comers (nontargeted) was unsuccessful in decreasing reutilization.⁴ By contrast, various studies have succeeded in reducing readmission and/or emergency department revisit rates associated with inpatient interventions in select populations.⁵ Currently, however, no standardized or validated pediatric risk prediction tool can reliably identify the high-risk patients who may benefit from interventions. In the Journal of Hospital Medicine, Brittan and colleagues add to the literature base exploring the factors associated with an increased 30-day readmission risk by trialing an electronic health record (EHR)-based tool composed of three components: presence of home health, polypharmacy in the form of ≥6 medications, and presence of a caregiver who prefers a language other than English.⁶

This brief report contributes significantly to the literature. First, the presence of a tool embedded within the pediatric EHR and readily accessible at the point of clinical care is novel. Study authors purposefully chose components easily extractable from the EHR which update automatically. This infrastructure generates an automated score that is easily accessible to clinicians in real-time. Second, the transparency of the tool is notable given its display via the EHR’s “Discharge Readiness Report,” where a clinician can view not only the total composite score (1 point for each component) but also the specific components for which a point was allocated. Although a composite score in and of itself is potentially helpful, understanding specific factors that contribute to a patient’s increased risk of readmission allows for better targeting of interventions. For example, in Brittan’s simple, three-component model, the presence of polypharmacy might trigger a pharmacist to meet with the family prior to discharge to discuss indications for and how to properly administer medications. Finally, a multidisciplinary team composed of clinicians, nurse-family educators, case managers, social workers, and informatics experts developed and implemented this tool. Although the roll-out and longitudinal use of this tool is not described, the engagement of these multiple provider-types is likely to increase successful roll-out and utilization of the tool.

Unfortunately, the utility of this tool in predicting readmission is limited as evidenced by its low c-statistic. This limitation may be due to several reasons. The tool was not originally built as a tool to predict readmissions but rather an instrument to identify complex discharge care as part of a quality improvement initiative to improve discharge processes. Given the questions about readmission risk prediction, the authors explored the potential for the tool to predict readmission risk. The authors acknowledge that the tool excluded many known readmission risk factors based upon inconsistent documentation within the EHR and the desire to emphasize only modifiable factors. Thus, variables, including prior hospitalization which is a well-documented risk factor for readmissions (but not modifiable) and social determinants of health (which are not consistently documented), were excluded. Additionally, the included variable of “language preference” may have been a considerably broad characteristic. Limited English proficiency has been increasingly recognized as a construct placing patients at higher risk for adverse outcomes. However, caregivers with high English proficiency also exhibit varying degrees of health literacy. The inclusion of health literacy may be additive to a readmission risk prediction tool. Finally, the outcome is not well-described with regard to identification of “unplanned” events. Thus, their outcome measure may have included planned admissions for which the readmission risk prediction tool would be irrelevant.

In summary, Brittan and colleagues engaged a multidisciplinary group of providers to address discharge planning processes and leveraged the EHR to support their efforts in the form of a brief screening tool. Although this tool was not predictive of hospital readmissions, it highlights the opportunity to better utilize the EHR to gather meaningful, real-time data and subsequently use this information to positively impact our clinical care and allocation of resources. The tool should serve as a stepping stone to building a more extensive tool with inclusion of other known and potential readmission risk factors, thus resulting in a clinically relevant readmission risk prediction tool.

Disclosures

The authors have nothing to disclose.

While the use of pediatric readmission rates as a quality metric remains controversial, pediatric hospital-to-home transitions need improvement.¹ As many as a third of pediatric readmissions are preventable,² but the multifactorial and complex nature of factors that contribute to pediatric readmissions presents a challenge in tackling readmission. Several factors are associated with increased risk of readmission; these factors include both clinical and sociodemographic characteristics;³ however, we still have much to learn. Further, the only large trial of an intervention to prevent pediatric readmissions across all comers (nontargeted) was unsuccessful in decreasing reutilization.⁴ By contrast, various studies have succeeded in reducing readmission and/or emergency department revisit rates associated with inpatient interventions in select populations.⁵ Currently, however, no standardized or validated pediatric risk prediction tool can reliably identify the high-risk patients who may benefit from interventions. In the Journal of Hospital Medicine, Brittan and colleagues add to the literature base exploring the factors associated with an increased 30-day readmission risk by trialing an electronic health record (EHR)-based tool composed of three components: presence of home health, polypharmacy in the form of ≥6 medications, and presence of a caregiver who prefers a language other than English.⁶

This brief report contributes significantly to the literature. First, the presence of a tool embedded within the pediatric EHR and readily accessible at the point of clinical care is novel. Study authors purposefully chose components easily extractable from the EHR which update automatically. This infrastructure generates an automated score that is easily accessible to clinicians in real-time. Second, the transparency of the tool is notable given its display via the EHR’s “Discharge Readiness Report,” where a clinician can view not only the total composite score (1 point for each component) but also the specific components for which a point was allocated. Although a composite score in and of itself is potentially helpful, understanding specific factors that contribute to a patient’s increased risk of readmission allows for better targeting of interventions. For example, in Brittan’s simple, three-component model, the presence of polypharmacy might trigger a pharmacist to meet with the family prior to discharge to discuss indications for and how to properly administer medications. Finally, a multidisciplinary team composed of clinicians, nurse-family educators, case managers, social workers, and informatics experts developed and implemented this tool. Although the roll-out and longitudinal use of this tool is not described, the engagement of these multiple provider-types is likely to increase successful roll-out and utilization of the tool.

Unfortunately, the utility of this tool in predicting readmission is limited as evidenced by its low c-statistic. This limitation may be due to several reasons. The tool was not originally built as a tool to predict readmissions but rather an instrument to identify complex discharge care as part of a quality improvement initiative to improve discharge processes. Given the questions about readmission risk prediction, the authors explored the potential for the tool to predict readmission risk. The authors acknowledge that the tool excluded many known readmission risk factors based upon inconsistent documentation within the EHR and the desire to emphasize only modifiable factors. Thus, variables, including prior hospitalization which is a well-documented risk factor for readmissions (but not modifiable) and social determinants of health (which are not consistently documented), were excluded. Additionally, the included variable of “language preference” may have been a considerably broad characteristic. Limited English proficiency has been increasingly recognized as a construct placing patients at higher risk for adverse outcomes. However, caregivers with high English proficiency also exhibit varying degrees of health literacy. The inclusion of health literacy may be additive to a readmission risk prediction tool. Finally, the outcome is not well-described with regard to identification of “unplanned” events. Thus, their outcome measure may have included planned admissions for which the readmission risk prediction tool would be irrelevant.

In summary, Brittan and colleagues engaged a multidisciplinary group of providers to address discharge planning processes and leveraged the EHR to support their efforts in the form of a brief screening tool. Although this tool was not predictive of hospital readmissions, it highlights the opportunity to better utilize the EHR to gather meaningful, real-time data and subsequently use this information to positively impact our clinical care and allocation of resources. The tool should serve as a stepping stone to building a more extensive tool with inclusion of other known and potential readmission risk factors, thus resulting in a clinically relevant readmission risk prediction tool.

Disclosures

The authors have nothing to disclose.

While the use of pediatric readmission rates as a quality metric remains controversial, pediatric hospital-to-home transitions need improvement.¹ As many as a third of pediatric readmissions are preventable,² but the multifactorial and complex nature of factors that contribute to pediatric readmissions presents a challenge in tackling readmission. Several factors are associated with increased risk of readmission; these factors include both clinical and sociodemographic characteristics;³ however, we still have much to learn. Further, the only large trial of an intervention to prevent pediatric readmissions across all comers (nontargeted) was unsuccessful in decreasing reutilization.⁴ By contrast, various studies have succeeded in reducing readmission and/or emergency department revisit rates associated with inpatient interventions in select populations.⁵ Currently, however, no standardized or validated pediatric risk prediction tool can reliably identify the high-risk patients who may benefit from interventions. In the Journal of Hospital Medicine, Brittan and colleagues add to the literature base exploring the factors associated with an increased 30-day readmission risk by trialing an electronic health record (EHR)-based tool composed of three components: presence of home health, polypharmacy in the form of ≥6 medications, and presence of a caregiver who prefers a language other than English.⁶

This brief report contributes significantly to the literature. First, the presence of a tool embedded within the pediatric EHR and readily accessible at the point of clinical care is novel. Study authors purposefully chose components easily extractable from the EHR which update automatically. This infrastructure generates an automated score that is easily accessible to clinicians in real-time. Second, the transparency of the tool is notable given its display via the EHR’s “Discharge Readiness Report,” where a clinician can view not only the total composite score (1 point for each component) but also the specific components for which a point was allocated. Although a composite score in and of itself is potentially helpful, understanding specific factors that contribute to a patient’s increased risk of readmission allows for better targeting of interventions. For example, in Brittan’s simple, three-component model, the presence of polypharmacy might trigger a pharmacist to meet with the family prior to discharge to discuss indications for and how to properly administer medications. Finally, a multidisciplinary team composed of clinicians, nurse-family educators, case managers, social workers, and informatics experts developed and implemented this tool. Although the roll-out and longitudinal use of this tool is not described, the engagement of these multiple provider-types is likely to increase successful roll-out and utilization of the tool.

Unfortunately, the utility of this tool in predicting readmission is limited as evidenced by its low c-statistic. This limitation may be due to several reasons. The tool was not originally built as a tool to predict readmissions but rather an instrument to identify complex discharge care as part of a quality improvement initiative to improve discharge processes. Given the questions about readmission risk prediction, the authors explored the potential for the tool to predict readmission risk. The authors acknowledge that the tool excluded many known readmission risk factors based upon inconsistent documentation within the EHR and the desire to emphasize only modifiable factors. Thus, variables, including prior hospitalization which is a well-documented risk factor for readmissions (but not modifiable) and social determinants of health (which are not consistently documented), were excluded. Additionally, the included variable of “language preference” may have been a considerably broad characteristic. Limited English proficiency has been increasingly recognized as a construct placing patients at higher risk for adverse outcomes. However, caregivers with high English proficiency also exhibit varying degrees of health literacy. The inclusion of health literacy may be additive to a readmission risk prediction tool. Finally, the outcome is not well-described with regard to identification of “unplanned” events. Thus, their outcome measure may have included planned admissions for which the readmission risk prediction tool would be irrelevant.

In summary, Brittan and colleagues engaged a multidisciplinary group of providers to address discharge planning processes and leveraged the EHR to support their efforts in the form of a brief screening tool. Although this tool was not predictive of hospital readmissions, it highlights the opportunity to better utilize the EHR to gather meaningful, real-time data and subsequently use this information to positively impact our clinical care and allocation of resources. The tool should serve as a stepping stone to building a more extensive tool with inclusion of other known and potential readmission risk factors, thus resulting in a clinically relevant readmission risk prediction tool.

Disclosures

The authors have nothing to disclose.