Vineet M. Arora, MD, MAPP

Clinicians, educators, and medical journals are increasingly using the social media outlet, Twitter, as a medium to connect and engage with their colleagues. In particular, online journal clubs have created a space for the timely discussion of research, creation of online communities, and dissemination of research.

Social media-based journal clubs are thought to be one way in which journals can leverage the power of social networks so that researchers can engage with a diverse range of end users⁴ (including bedside clinicians, administrators, and patients). Several examples of these models exist. For example, #GeriMedJC acts as a complimentary, synchronous chat that takes place at the same time as a live, in-person journal club. #NephJC offers multiple 1-hour chats per month and provides an in-depth summary and analysis of each article, while #UroJC is an asynchronous discussion that takes place over 48 hours. Few data exist to describe whether any of these programs produce measurable improvements in indicators of engagement or dissemination of results.

In 2015, the Journal of Hospital Medicine (JHM) began producing a Twitter-based journal club as a means to connect and engage the Hospital Medicine community and allow for discussion and rapid exchange of information and opinions around a specific clinical topic. This study aims to describe the implementation of the first Journal-sponsored, Twitter-based online journal club and ascertain its impact on both Twitter and journal metrics.

#JHMChat was launched in October 2015, and was initially held every 2-3 months until January 2017, when chats began to take place monthly. Each 1-hour chat focused on a recently published article in JHM, was moderated by a JHM social media editor (C.M.W., V.M.A.), and included at least 1 study author or guest expert. Articles were chosen by the social media editors based on the following criteria: (1) attractiveness to possible participants, (2) providing topic variety within the journal club series, and (3) sustainability and topic conduciveness to the online chat model. Chats were held at 9 PM EST in order to engage hospitalists across all US time zones and on different days to accommodate authors’ availability. All sessions were framed by 3-4 questions intended to encourage discussion and presented to chat participants at spaced intervals so as to stimulate a current of responses.

Chats were promoted by way of the JHM (@JHospMedicine, 3400 followers) and Society of Hospital Medicine (SHM; @SHMLive, 5800 followers) Twitter feeds beginning 1 month prior to each session. Visual Abstracts^5,6 were used to publicize the sessions, also via Twitter, starting in February 2017.

Continuing Medical Education (CME) credits were offered through the SHM to registered participants, starting in July 2016.⁷ All sessions were cosponsored by the American Board of Internal Medicine (ABIM) Foundation and the Costs of Care Organization, a non-profit organization aimed at improving healthcare value.

Twitter Metrics

After each session, the following Twitter-based engagement metrics were obtained using the Symplur^© Healthcare Hashtag project;⁸ total number of participants and tweets, tweets/participant, and total impressions (calculated as the number of tweets from each participant multiplied by the number of followers that participant currently had then summed up for all participants). Simply put, impressions can also be thought of as the number of times a single Tweet makes it into someone else’s Twitter feed. So as to avoid artificially inflated metrics, all were obtained 2 hours after the end of the journal club. Participants were defined as anyone who posted an original tweet or retweeted during the session and were encouraged to tag their tweets with the hashtag #JHMChat for post-discussion indexing and measurement. Because authors’ or guests’ popularity on Twitter may influence participation rates, we also assessed the number of followers for each participating author. Spearman’s rank correlation was calculated (Microsoft ExcelTM) where appropriate.

Altmetrics and Page Views

As a means to measure exposure and dissemination external to Twitter, we assessed the change (“Delta”) in the each article’s Altmetric score⁹, a digital-based metric that quantifies the attention received by a scientific publication on various online platforms including news, blogs, and social media. Delta Altmetric scores were calculated as the difference between the scores on the day of the session and 2 weeks after the respective session, with higher scores indicating greater global online discussion. By measuring the Altmetric score on the day of the discussion, we established a baseline score for comparison purposes. Additionally, this allowed us to better attribute any changes that may have occurred to the discussion itself.

Additionally, using information provided by the journal publisher (John Wiley & Sons Publishing) in 2016, we assessed the effect of #JHMChat on the number of article page views on the JHM website relative to the release of the electronic Table of Contents (eTOC). The eTOC release was chosen as it is historically associated with a high number of page views. In order to isolate the effect of #JHMChat, we only reviewed months in which #JHMChat was not held within 3 days of the eTOC release. Because JHM changed publishers in January 2017, we only assessed page view data on 2016 sessions, as the new publisher lacked enhanced search optimization to obtain these data.

Thematic Analysis

In addition to the above measurements, a thematic analysis of each article was conducted to assess any common themes that would influence our chosen metrics. Themes were assessed and ascribed by one author (C.M.W.) and verified by another (V.M.A.).

Participant and Author Experience

To assess the participant experience, responses to a post-session CME questionnaire that assessed (1) overall quality, (2) comprehensiveness of the discussion, (3) whether the participant would recommend the chat to a colleague, and (4) whether participation would lead to practice-changing measures were reviewed. Registration of each session for CME was also quantified. Finally, each participating author was asked to fill out an electronic post-chat survey (SurveyMonkey^®) meant to assess the authors’ experience with the journal club (Appendix).

Between October 2015 and November 2017, a total of 15 sessions were held with a mean of 2.17 (±0.583) million impressions/session, 499 (±129) total tweets/session, and 73 (±24) participants/session (compared to a range of 21-58 participants/session from other online journal clubs, where reported) with 7.2 (±2.0) tweets/participant (Table 1). The total number of participants for all sessions was 1096. Participating authors had on average 1389 (±2714) followers, ranging from a low of 37 to a high of 10,376 (Appendix). No correlation between author following and number of participants (r = 0.19), impressions (r = 0.05), or change in Altmetric score (r = 0.17) was seen.

Thematic analysis revealed 3 predominant themes among the chosen articles: Value-based care (VBC), Quality and Patient Safety (QPS), and Medical Education (ME). Articles focused on VBC had the greatest number of impressions (mean ±SD: 2.61 ± 0.55 million) and participants (mean ±SD: 90 ± 12), while QPS articles had the fewest impressions (mean ±SD: 1.71 ± 0.59 million) and number of participants (mean ±SD: 47 ± 16). The mean increase in the Altmetric score among all discussed articles was 14 (±12), from an average baseline of 30 (±37). Medical Education-themed articles appeared to garner the greatest increase in Altmetric scores, averaging an increase of 32 points, compared with an average baseline score of 31 (±32). In contrast, VBC and QPS articles averaged an increase of 8.6 and 8.4 points, from average baselines of 55 (±53) and 17 (±13), respectively. A 2-month analysis of JHM articles not included in these discussions, in which Altmetric scores were measured in the same way as those from the discussion, revealed a baseline Altmetric score of 27 (±24) with an average increase of 8 (±6) 2 weeks following the chat.

Four articles met the inclusion criteria for page view analysis and suggested that article page views increased to similar levels as the eTOC release (mean: 2668 vs. 2998, respectively; P = .35) (Figure). These increases equate to a 33% and 50% increase in average daily page views (2002) for the chat and eTOC release, respectively.

The Twitter-based journal club #JHMChat appears to increase social media awareness and dissemination of journal articles and was considered a useful engagement platform by both authors and participants.

Articles with a focus on VBC and ME had the greatest impact on dissemination metrics, particularly, total impressions and Altmetric scores, respectively. Given the strong presence and interest in these topics within Twitter and social media, these findings are not surprising.^10,11 For example, over the past several years, the VBC movement has taken shape and grown alongside the expansion of social media, thus giving a space for this community to grow and engage. Of note, the cosponsorship relationship with the ABIM Foundation (which works closely with the Choosing Wisely™ campaign) and the Costs of Care Organization could have influenced the participation and dissemination rates of VBC articles. Medical education articles were also popular and appeared to have increased uptake after chats, based on their Altmetric scores. This may be explained by the fact that medical educators have long utilized social media as a means to connect and engage within their community.^12–14 It is also possible that the use of Twitter by trainees (residents, students) may have driven some of the dissemination of ME articles, as this group may not be regular subscribers to JHM.

Online journal clubs offer distinct advantages over traditional in-person journal clubs. First, online journal clubs allow for increased connectivity among online communities, bringing together participants from different geographic areas with diverse training and clinical experiences. Subsequently, this allows for the rapid exchange of both personal and organizational approaches to the topic of discussion.^15–17 Second, online journal clubs allow for continual access to the discussion material. For example, while the metrics used in this study only assessed active, synchronous participation, anecdotal evidence and feedback to the authors suggests that many individuals passively engaged by following along or reviewed the chat feed post hoc at their convenience. This asynchronous access is a quality not found in more traditional journal club formats. Finally, because online journal clubs commonly operate with a flattened hierarchy,¹⁸ they can break down access barriers to both the researchers who performed the study and thought leaders who commonly participate.¹⁷

Several insightful lessons were gleaned in the production and management of this online journal club. On the implementation side, promotion, preparation, and continued organization of an online journal club requires a fair amount of work. In this case, the required time and resources were provided by 2 social media editors in addition to administrative assistance from the SHM. The high attrition rate of online journal clubs over the years attests to these difficulties.²⁴ Additionally, finding incentives to attract and sustain participation can be difficult, as we noted that neither CME nor author popularity (based on their Twitter following) appeared to influence engagement metrics (number of participants, total tweets, and tweets/participant). We also found that partnering with other journal club communities, in particular #NephJC, lead to greater participation rates and impressions. Thus, leveraging connections and topics that span clinical domains may be one way to improve and broaden engagement within these forums. Finally, feedback from participants revealed that the timing of the journal club and the inability to have in-depth discussions, a characteristic commonly associated with traditional journal clubs, were problematic.

This study has several limitations. First, the metrics used to assess social media engagement and dissemination can be easily skewed. For instance, the activity of 1 or 2 individuals with large followings can dramatically influence the number of impressions, giving a falsely elevated sense of broad dissemination. Conversely, there may have been some participants who did not use the #JHMChat hashtag, thus leading to an underestimation in these metrics. Second, while we report total impressions as a measure of dissemination, this metric represents possible interactions and does not guarantee interaction or visualization of that tweet. Additionally, we were unable to characterize our participants and their participation rates over time, as this information is not made available through Symplur© analytics. Third, our page view assessment was limited to 2016 sessions only; therefore, these data may not be an accurate reflection of the impact of #JHMChat on this metric. Fourth, given the marginal response rate to our CME questionnaire, a selection bias could have occurred. Finally, whether social media discussions such as online journal clubs act as leading indicators for future citations remains unclear, as some research has shown an association between increased Altmetric scores and increased citation rates,^19-21 while others have not.^22,23 Our study was not equipped to assess this correlation.

Online journal clubs create new opportunities to connect, engage, and disseminate medical research. These developing forums provide journal editors, researchers, patients, and clinicians with a means to connect and discuss research in ways that were not previously possible. In order to continue to evolve and grow, future research in online journal clubs should explore the downstream effects on citation rates, clinical uptake, and participant knowledge after the sessions.

Acknowledgments

The authors would like to thank Felicia Steele for her assistance in organizing and promoting the chats. Additionally, the authors would like to thank all the authors, guests and participants who took time from their families, work, and daily lives to participate in these activities. Your time and presence were truly appreciated.

Disclosures

The authors of this article operate as the Social Media Editors (C.M.W., V.M.A.) and the Editor-in-Chief (A.A.) for the Journal of Hospital Medicine. Dr. Wray had full access to all the data in the project, takes responsibility for the integrity of the data, and the accuracy of the data analysis.

Clinicians, educators, and medical journals are increasingly using the social media outlet, Twitter, as a medium to connect and engage with their colleagues. In particular, online journal clubs have created a space for the timely discussion of research, creation of online communities, and dissemination of research.

Social media-based journal clubs are thought to be one way in which journals can leverage the power of social networks so that researchers can engage with a diverse range of end users⁴ (including bedside clinicians, administrators, and patients). Several examples of these models exist. For example, #GeriMedJC acts as a complimentary, synchronous chat that takes place at the same time as a live, in-person journal club. #NephJC offers multiple 1-hour chats per month and provides an in-depth summary and analysis of each article, while #UroJC is an asynchronous discussion that takes place over 48 hours. Few data exist to describe whether any of these programs produce measurable improvements in indicators of engagement or dissemination of results.

In 2015, the Journal of Hospital Medicine (JHM) began producing a Twitter-based journal club as a means to connect and engage the Hospital Medicine community and allow for discussion and rapid exchange of information and opinions around a specific clinical topic. This study aims to describe the implementation of the first Journal-sponsored, Twitter-based online journal club and ascertain its impact on both Twitter and journal metrics.

#JHMChat was launched in October 2015, and was initially held every 2-3 months until January 2017, when chats began to take place monthly. Each 1-hour chat focused on a recently published article in JHM, was moderated by a JHM social media editor (C.M.W., V.M.A.), and included at least 1 study author or guest expert. Articles were chosen by the social media editors based on the following criteria: (1) attractiveness to possible participants, (2) providing topic variety within the journal club series, and (3) sustainability and topic conduciveness to the online chat model. Chats were held at 9 PM EST in order to engage hospitalists across all US time zones and on different days to accommodate authors’ availability. All sessions were framed by 3-4 questions intended to encourage discussion and presented to chat participants at spaced intervals so as to stimulate a current of responses.

Chats were promoted by way of the JHM (@JHospMedicine, 3400 followers) and Society of Hospital Medicine (SHM; @SHMLive, 5800 followers) Twitter feeds beginning 1 month prior to each session. Visual Abstracts^5,6 were used to publicize the sessions, also via Twitter, starting in February 2017.

Continuing Medical Education (CME) credits were offered through the SHM to registered participants, starting in July 2016.⁷ All sessions were cosponsored by the American Board of Internal Medicine (ABIM) Foundation and the Costs of Care Organization, a non-profit organization aimed at improving healthcare value.

Twitter Metrics

After each session, the following Twitter-based engagement metrics were obtained using the Symplur^© Healthcare Hashtag project;⁸ total number of participants and tweets, tweets/participant, and total impressions (calculated as the number of tweets from each participant multiplied by the number of followers that participant currently had then summed up for all participants). Simply put, impressions can also be thought of as the number of times a single Tweet makes it into someone else’s Twitter feed. So as to avoid artificially inflated metrics, all were obtained 2 hours after the end of the journal club. Participants were defined as anyone who posted an original tweet or retweeted during the session and were encouraged to tag their tweets with the hashtag #JHMChat for post-discussion indexing and measurement. Because authors’ or guests’ popularity on Twitter may influence participation rates, we also assessed the number of followers for each participating author. Spearman’s rank correlation was calculated (Microsoft ExcelTM) where appropriate.

Altmetrics and Page Views

As a means to measure exposure and dissemination external to Twitter, we assessed the change (“Delta”) in the each article’s Altmetric score⁹, a digital-based metric that quantifies the attention received by a scientific publication on various online platforms including news, blogs, and social media. Delta Altmetric scores were calculated as the difference between the scores on the day of the session and 2 weeks after the respective session, with higher scores indicating greater global online discussion. By measuring the Altmetric score on the day of the discussion, we established a baseline score for comparison purposes. Additionally, this allowed us to better attribute any changes that may have occurred to the discussion itself.

Additionally, using information provided by the journal publisher (John Wiley & Sons Publishing) in 2016, we assessed the effect of #JHMChat on the number of article page views on the JHM website relative to the release of the electronic Table of Contents (eTOC). The eTOC release was chosen as it is historically associated with a high number of page views. In order to isolate the effect of #JHMChat, we only reviewed months in which #JHMChat was not held within 3 days of the eTOC release. Because JHM changed publishers in January 2017, we only assessed page view data on 2016 sessions, as the new publisher lacked enhanced search optimization to obtain these data.

Thematic Analysis

In addition to the above measurements, a thematic analysis of each article was conducted to assess any common themes that would influence our chosen metrics. Themes were assessed and ascribed by one author (C.M.W.) and verified by another (V.M.A.).

Participant and Author Experience

To assess the participant experience, responses to a post-session CME questionnaire that assessed (1) overall quality, (2) comprehensiveness of the discussion, (3) whether the participant would recommend the chat to a colleague, and (4) whether participation would lead to practice-changing measures were reviewed. Registration of each session for CME was also quantified. Finally, each participating author was asked to fill out an electronic post-chat survey (SurveyMonkey^®) meant to assess the authors’ experience with the journal club (Appendix).

Between October 2015 and November 2017, a total of 15 sessions were held with a mean of 2.17 (±0.583) million impressions/session, 499 (±129) total tweets/session, and 73 (±24) participants/session (compared to a range of 21-58 participants/session from other online journal clubs, where reported) with 7.2 (±2.0) tweets/participant (Table 1). The total number of participants for all sessions was 1096. Participating authors had on average 1389 (±2714) followers, ranging from a low of 37 to a high of 10,376 (Appendix). No correlation between author following and number of participants (r = 0.19), impressions (r = 0.05), or change in Altmetric score (r = 0.17) was seen.

Thematic analysis revealed 3 predominant themes among the chosen articles: Value-based care (VBC), Quality and Patient Safety (QPS), and Medical Education (ME). Articles focused on VBC had the greatest number of impressions (mean ±SD: 2.61 ± 0.55 million) and participants (mean ±SD: 90 ± 12), while QPS articles had the fewest impressions (mean ±SD: 1.71 ± 0.59 million) and number of participants (mean ±SD: 47 ± 16). The mean increase in the Altmetric score among all discussed articles was 14 (±12), from an average baseline of 30 (±37). Medical Education-themed articles appeared to garner the greatest increase in Altmetric scores, averaging an increase of 32 points, compared with an average baseline score of 31 (±32). In contrast, VBC and QPS articles averaged an increase of 8.6 and 8.4 points, from average baselines of 55 (±53) and 17 (±13), respectively. A 2-month analysis of JHM articles not included in these discussions, in which Altmetric scores were measured in the same way as those from the discussion, revealed a baseline Altmetric score of 27 (±24) with an average increase of 8 (±6) 2 weeks following the chat.

Four articles met the inclusion criteria for page view analysis and suggested that article page views increased to similar levels as the eTOC release (mean: 2668 vs. 2998, respectively; P = .35) (Figure). These increases equate to a 33% and 50% increase in average daily page views (2002) for the chat and eTOC release, respectively.

The Twitter-based journal club #JHMChat appears to increase social media awareness and dissemination of journal articles and was considered a useful engagement platform by both authors and participants.

Articles with a focus on VBC and ME had the greatest impact on dissemination metrics, particularly, total impressions and Altmetric scores, respectively. Given the strong presence and interest in these topics within Twitter and social media, these findings are not surprising.^10,11 For example, over the past several years, the VBC movement has taken shape and grown alongside the expansion of social media, thus giving a space for this community to grow and engage. Of note, the cosponsorship relationship with the ABIM Foundation (which works closely with the Choosing Wisely™ campaign) and the Costs of Care Organization could have influenced the participation and dissemination rates of VBC articles. Medical education articles were also popular and appeared to have increased uptake after chats, based on their Altmetric scores. This may be explained by the fact that medical educators have long utilized social media as a means to connect and engage within their community.^12–14 It is also possible that the use of Twitter by trainees (residents, students) may have driven some of the dissemination of ME articles, as this group may not be regular subscribers to JHM.

Online journal clubs offer distinct advantages over traditional in-person journal clubs. First, online journal clubs allow for increased connectivity among online communities, bringing together participants from different geographic areas with diverse training and clinical experiences. Subsequently, this allows for the rapid exchange of both personal and organizational approaches to the topic of discussion.^15–17 Second, online journal clubs allow for continual access to the discussion material. For example, while the metrics used in this study only assessed active, synchronous participation, anecdotal evidence and feedback to the authors suggests that many individuals passively engaged by following along or reviewed the chat feed post hoc at their convenience. This asynchronous access is a quality not found in more traditional journal club formats. Finally, because online journal clubs commonly operate with a flattened hierarchy,¹⁸ they can break down access barriers to both the researchers who performed the study and thought leaders who commonly participate.¹⁷

Several insightful lessons were gleaned in the production and management of this online journal club. On the implementation side, promotion, preparation, and continued organization of an online journal club requires a fair amount of work. In this case, the required time and resources were provided by 2 social media editors in addition to administrative assistance from the SHM. The high attrition rate of online journal clubs over the years attests to these difficulties.²⁴ Additionally, finding incentives to attract and sustain participation can be difficult, as we noted that neither CME nor author popularity (based on their Twitter following) appeared to influence engagement metrics (number of participants, total tweets, and tweets/participant). We also found that partnering with other journal club communities, in particular #NephJC, lead to greater participation rates and impressions. Thus, leveraging connections and topics that span clinical domains may be one way to improve and broaden engagement within these forums. Finally, feedback from participants revealed that the timing of the journal club and the inability to have in-depth discussions, a characteristic commonly associated with traditional journal clubs, were problematic.

This study has several limitations. First, the metrics used to assess social media engagement and dissemination can be easily skewed. For instance, the activity of 1 or 2 individuals with large followings can dramatically influence the number of impressions, giving a falsely elevated sense of broad dissemination. Conversely, there may have been some participants who did not use the #JHMChat hashtag, thus leading to an underestimation in these metrics. Second, while we report total impressions as a measure of dissemination, this metric represents possible interactions and does not guarantee interaction or visualization of that tweet. Additionally, we were unable to characterize our participants and their participation rates over time, as this information is not made available through Symplur© analytics. Third, our page view assessment was limited to 2016 sessions only; therefore, these data may not be an accurate reflection of the impact of #JHMChat on this metric. Fourth, given the marginal response rate to our CME questionnaire, a selection bias could have occurred. Finally, whether social media discussions such as online journal clubs act as leading indicators for future citations remains unclear, as some research has shown an association between increased Altmetric scores and increased citation rates,^19-21 while others have not.^22,23 Our study was not equipped to assess this correlation.

Online journal clubs create new opportunities to connect, engage, and disseminate medical research. These developing forums provide journal editors, researchers, patients, and clinicians with a means to connect and discuss research in ways that were not previously possible. In order to continue to evolve and grow, future research in online journal clubs should explore the downstream effects on citation rates, clinical uptake, and participant knowledge after the sessions.

Acknowledgments

The authors would like to thank Felicia Steele for her assistance in organizing and promoting the chats. Additionally, the authors would like to thank all the authors, guests and participants who took time from their families, work, and daily lives to participate in these activities. Your time and presence were truly appreciated.

Disclosures

The authors of this article operate as the Social Media Editors (C.M.W., V.M.A.) and the Editor-in-Chief (A.A.) for the Journal of Hospital Medicine. Dr. Wray had full access to all the data in the project, takes responsibility for the integrity of the data, and the accuracy of the data analysis.

Clinicians, educators, and medical journals are increasingly using the social media outlet, Twitter, as a medium to connect and engage with their colleagues. In particular, online journal clubs have created a space for the timely discussion of research, creation of online communities, and dissemination of research.

Social media-based journal clubs are thought to be one way in which journals can leverage the power of social networks so that researchers can engage with a diverse range of end users⁴ (including bedside clinicians, administrators, and patients). Several examples of these models exist. For example, #GeriMedJC acts as a complimentary, synchronous chat that takes place at the same time as a live, in-person journal club. #NephJC offers multiple 1-hour chats per month and provides an in-depth summary and analysis of each article, while #UroJC is an asynchronous discussion that takes place over 48 hours. Few data exist to describe whether any of these programs produce measurable improvements in indicators of engagement or dissemination of results.

In 2015, the Journal of Hospital Medicine (JHM) began producing a Twitter-based journal club as a means to connect and engage the Hospital Medicine community and allow for discussion and rapid exchange of information and opinions around a specific clinical topic. This study aims to describe the implementation of the first Journal-sponsored, Twitter-based online journal club and ascertain its impact on both Twitter and journal metrics.

#JHMChat was launched in October 2015, and was initially held every 2-3 months until January 2017, when chats began to take place monthly. Each 1-hour chat focused on a recently published article in JHM, was moderated by a JHM social media editor (C.M.W., V.M.A.), and included at least 1 study author or guest expert. Articles were chosen by the social media editors based on the following criteria: (1) attractiveness to possible participants, (2) providing topic variety within the journal club series, and (3) sustainability and topic conduciveness to the online chat model. Chats were held at 9 PM EST in order to engage hospitalists across all US time zones and on different days to accommodate authors’ availability. All sessions were framed by 3-4 questions intended to encourage discussion and presented to chat participants at spaced intervals so as to stimulate a current of responses.

Chats were promoted by way of the JHM (@JHospMedicine, 3400 followers) and Society of Hospital Medicine (SHM; @SHMLive, 5800 followers) Twitter feeds beginning 1 month prior to each session. Visual Abstracts^5,6 were used to publicize the sessions, also via Twitter, starting in February 2017.

Continuing Medical Education (CME) credits were offered through the SHM to registered participants, starting in July 2016.⁷ All sessions were cosponsored by the American Board of Internal Medicine (ABIM) Foundation and the Costs of Care Organization, a non-profit organization aimed at improving healthcare value.

Twitter Metrics

After each session, the following Twitter-based engagement metrics were obtained using the Symplur^© Healthcare Hashtag project;⁸ total number of participants and tweets, tweets/participant, and total impressions (calculated as the number of tweets from each participant multiplied by the number of followers that participant currently had then summed up for all participants). Simply put, impressions can also be thought of as the number of times a single Tweet makes it into someone else’s Twitter feed. So as to avoid artificially inflated metrics, all were obtained 2 hours after the end of the journal club. Participants were defined as anyone who posted an original tweet or retweeted during the session and were encouraged to tag their tweets with the hashtag #JHMChat for post-discussion indexing and measurement. Because authors’ or guests’ popularity on Twitter may influence participation rates, we also assessed the number of followers for each participating author. Spearman’s rank correlation was calculated (Microsoft ExcelTM) where appropriate.

Altmetrics and Page Views

As a means to measure exposure and dissemination external to Twitter, we assessed the change (“Delta”) in the each article’s Altmetric score⁹, a digital-based metric that quantifies the attention received by a scientific publication on various online platforms including news, blogs, and social media. Delta Altmetric scores were calculated as the difference between the scores on the day of the session and 2 weeks after the respective session, with higher scores indicating greater global online discussion. By measuring the Altmetric score on the day of the discussion, we established a baseline score for comparison purposes. Additionally, this allowed us to better attribute any changes that may have occurred to the discussion itself.

Additionally, using information provided by the journal publisher (John Wiley & Sons Publishing) in 2016, we assessed the effect of #JHMChat on the number of article page views on the JHM website relative to the release of the electronic Table of Contents (eTOC). The eTOC release was chosen as it is historically associated with a high number of page views. In order to isolate the effect of #JHMChat, we only reviewed months in which #JHMChat was not held within 3 days of the eTOC release. Because JHM changed publishers in January 2017, we only assessed page view data on 2016 sessions, as the new publisher lacked enhanced search optimization to obtain these data.

Thematic Analysis

In addition to the above measurements, a thematic analysis of each article was conducted to assess any common themes that would influence our chosen metrics. Themes were assessed and ascribed by one author (C.M.W.) and verified by another (V.M.A.).

Participant and Author Experience

To assess the participant experience, responses to a post-session CME questionnaire that assessed (1) overall quality, (2) comprehensiveness of the discussion, (3) whether the participant would recommend the chat to a colleague, and (4) whether participation would lead to practice-changing measures were reviewed. Registration of each session for CME was also quantified. Finally, each participating author was asked to fill out an electronic post-chat survey (SurveyMonkey^®) meant to assess the authors’ experience with the journal club (Appendix).

Between October 2015 and November 2017, a total of 15 sessions were held with a mean of 2.17 (±0.583) million impressions/session, 499 (±129) total tweets/session, and 73 (±24) participants/session (compared to a range of 21-58 participants/session from other online journal clubs, where reported) with 7.2 (±2.0) tweets/participant (Table 1). The total number of participants for all sessions was 1096. Participating authors had on average 1389 (±2714) followers, ranging from a low of 37 to a high of 10,376 (Appendix). No correlation between author following and number of participants (r = 0.19), impressions (r = 0.05), or change in Altmetric score (r = 0.17) was seen.

Thematic analysis revealed 3 predominant themes among the chosen articles: Value-based care (VBC), Quality and Patient Safety (QPS), and Medical Education (ME). Articles focused on VBC had the greatest number of impressions (mean ±SD: 2.61 ± 0.55 million) and participants (mean ±SD: 90 ± 12), while QPS articles had the fewest impressions (mean ±SD: 1.71 ± 0.59 million) and number of participants (mean ±SD: 47 ± 16). The mean increase in the Altmetric score among all discussed articles was 14 (±12), from an average baseline of 30 (±37). Medical Education-themed articles appeared to garner the greatest increase in Altmetric scores, averaging an increase of 32 points, compared with an average baseline score of 31 (±32). In contrast, VBC and QPS articles averaged an increase of 8.6 and 8.4 points, from average baselines of 55 (±53) and 17 (±13), respectively. A 2-month analysis of JHM articles not included in these discussions, in which Altmetric scores were measured in the same way as those from the discussion, revealed a baseline Altmetric score of 27 (±24) with an average increase of 8 (±6) 2 weeks following the chat.

Four articles met the inclusion criteria for page view analysis and suggested that article page views increased to similar levels as the eTOC release (mean: 2668 vs. 2998, respectively; P = .35) (Figure). These increases equate to a 33% and 50% increase in average daily page views (2002) for the chat and eTOC release, respectively.

The Twitter-based journal club #JHMChat appears to increase social media awareness and dissemination of journal articles and was considered a useful engagement platform by both authors and participants.

Articles with a focus on VBC and ME had the greatest impact on dissemination metrics, particularly, total impressions and Altmetric scores, respectively. Given the strong presence and interest in these topics within Twitter and social media, these findings are not surprising.^10,11 For example, over the past several years, the VBC movement has taken shape and grown alongside the expansion of social media, thus giving a space for this community to grow and engage. Of note, the cosponsorship relationship with the ABIM Foundation (which works closely with the Choosing Wisely™ campaign) and the Costs of Care Organization could have influenced the participation and dissemination rates of VBC articles. Medical education articles were also popular and appeared to have increased uptake after chats, based on their Altmetric scores. This may be explained by the fact that medical educators have long utilized social media as a means to connect and engage within their community.^12–14 It is also possible that the use of Twitter by trainees (residents, students) may have driven some of the dissemination of ME articles, as this group may not be regular subscribers to JHM.

Online journal clubs offer distinct advantages over traditional in-person journal clubs. First, online journal clubs allow for increased connectivity among online communities, bringing together participants from different geographic areas with diverse training and clinical experiences. Subsequently, this allows for the rapid exchange of both personal and organizational approaches to the topic of discussion.^15–17 Second, online journal clubs allow for continual access to the discussion material. For example, while the metrics used in this study only assessed active, synchronous participation, anecdotal evidence and feedback to the authors suggests that many individuals passively engaged by following along or reviewed the chat feed post hoc at their convenience. This asynchronous access is a quality not found in more traditional journal club formats. Finally, because online journal clubs commonly operate with a flattened hierarchy,¹⁸ they can break down access barriers to both the researchers who performed the study and thought leaders who commonly participate.¹⁷

Several insightful lessons were gleaned in the production and management of this online journal club. On the implementation side, promotion, preparation, and continued organization of an online journal club requires a fair amount of work. In this case, the required time and resources were provided by 2 social media editors in addition to administrative assistance from the SHM. The high attrition rate of online journal clubs over the years attests to these difficulties.²⁴ Additionally, finding incentives to attract and sustain participation can be difficult, as we noted that neither CME nor author popularity (based on their Twitter following) appeared to influence engagement metrics (number of participants, total tweets, and tweets/participant). We also found that partnering with other journal club communities, in particular #NephJC, lead to greater participation rates and impressions. Thus, leveraging connections and topics that span clinical domains may be one way to improve and broaden engagement within these forums. Finally, feedback from participants revealed that the timing of the journal club and the inability to have in-depth discussions, a characteristic commonly associated with traditional journal clubs, were problematic.

This study has several limitations. First, the metrics used to assess social media engagement and dissemination can be easily skewed. For instance, the activity of 1 or 2 individuals with large followings can dramatically influence the number of impressions, giving a falsely elevated sense of broad dissemination. Conversely, there may have been some participants who did not use the #JHMChat hashtag, thus leading to an underestimation in these metrics. Second, while we report total impressions as a measure of dissemination, this metric represents possible interactions and does not guarantee interaction or visualization of that tweet. Additionally, we were unable to characterize our participants and their participation rates over time, as this information is not made available through Symplur© analytics. Third, our page view assessment was limited to 2016 sessions only; therefore, these data may not be an accurate reflection of the impact of #JHMChat on this metric. Fourth, given the marginal response rate to our CME questionnaire, a selection bias could have occurred. Finally, whether social media discussions such as online journal clubs act as leading indicators for future citations remains unclear, as some research has shown an association between increased Altmetric scores and increased citation rates,^19-21 while others have not.^22,23 Our study was not equipped to assess this correlation.

Online journal clubs create new opportunities to connect, engage, and disseminate medical research. These developing forums provide journal editors, researchers, patients, and clinicians with a means to connect and discuss research in ways that were not previously possible. In order to continue to evolve and grow, future research in online journal clubs should explore the downstream effects on citation rates, clinical uptake, and participant knowledge after the sessions.

Acknowledgments

The authors would like to thank Felicia Steele for her assistance in organizing and promoting the chats. Additionally, the authors would like to thank all the authors, guests and participants who took time from their families, work, and daily lives to participate in these activities. Your time and presence were truly appreciated.

Disclosures

The authors of this article operate as the Social Media Editors (C.M.W., V.M.A.) and the Editor-in-Chief (A.A.) for the Journal of Hospital Medicine. Dr. Wray had full access to all the data in the project, takes responsibility for the integrity of the data, and the accuracy of the data analysis.

Fifteen years ago, beepers with 5-digit call-back numbers were the norm. Pushing a call light button outside the patient’s room to flag the desk clerk that a new order had been hand-written was all part of the lived experience of residency. Using that as our baseline, we have clearly come a long way in the way that we communicate with other clinicians in hospitals. Communication among the patient care team in the digital age predominantly involves bidirectional messaging using mobile devices. The approach is both immediate and convenient. Mobile devices can improve work efficiency, patient safety, and quality of care, but their main advantage may be real-time bedside decision support.^1,2 However, the widespread use of mobile devices for communication in healthcare is not without its concerns. First and foremost, there has been abundant literature around short message service (SMS) use in the healthcare setting, and there are concerns surrounding both threats to privacy and the prevalence and impact of interruptions in clinical care.

The first SMS was sent in 1992.³ Text messaging since then has become ubiquitous, even in healthcare, raising concerns around the protection of patient health information under the Health Insurance Portability and Accountability Act (HIPAA). Interestingly, the United States Department of Health and Human Services Office for Civil Rights, enforcer of HIPAA, is tech neutral on the subject.³ Multiple studies have assessed physician stances on SMS communication in the healthcare setting using routine, non-HIPAA-compliant mobile phones. Overall, 60%-80% of respondents admitted to using SMS in patient care, while in another study, 72% and 80% of Internal Medicine residents surveyed found SMS to be the most efficient form of communication and overall preferred method of communication, respectively.^3,4 Interestingly, 82.5% of those same residents preferred the hospital-based alphanumeric paging system for security purposes, even though Freundlich et al. make a compelling argument that unidirectional alphanumeric paging systems are most certainly less HIPAA compliant, lacking encryption and password protection.⁵ Newer platforms that enable HIPAA-compliant messaging are promising, although they may not be fully adopted by clinical teams without full-scale implementation in hospitals.⁶In addition to privacy concerns with SMS applications on mobile phones, interruptions in healthcare – be it from phone calls, emails, text messages, or in-person conversations – are common. In fact, famed communication researcher Enrico Coeira has notoriously described healthcare communication as ”interrupt-driven.”⁷ Prior work has shown that frequent interruptions in the healthcare setting can lead to medication prescription errors, errors in computerized physician order entry, and even surgical procedural errors.^8-10

While studies have focused on interruptions in clinical care in the healthcare setting, little is known about how education may be compromised by interruptions due to mobile devices. Text messaging during dedicated conference time can lead to inadequate learning and a sense of frustration among residents. In this issue of the Journal of Hospital Medicine, Mendel et al. performed a quality improvement study involving eight academic inpatient clinical training units with the aim of reducing nonurgent text messages during education rounds.¹¹ Their unique interventions included learning sessions, posters, adding alerts to the digital communication platform, and alternative messaging options. Of four sequential interventions, a message alerting the sender that they will be interrupting educational rounds and suggesting a “delayed send” or “send as an FYI” showed the greatest impact, reducing the number of text interruptions per team per educational hour from 0.81 to 0.59 (95% CI 0.51-0.67). When comparing a four-week pre-intervention sample with a four-week end-intervention sample, the percentage of nonurgent messages decreased from 82% to 68% (P < .01).

While these results are promising, challenges to large-scale implementation of such a program exist. Buy-in from the ancillary healthcare team is critical for such interventions to succeed and be sustained. It also places a burden of “point triage” on the healthcare team members, who must assess the patient situation and determine the level of urgency and whether to immediately interrupt, delay interrupt or send an FYI message. For example, in the study by Mendel et al.,¹¹ it is noteworthy that urgent patient care issues were mislabeled as “FYI” in 2% of patients. While this is a seemingly low rate, even one of these mislabeled messages could result in significant adverse patient outcomes and should be considered a “never event.” Finally, the study used a messaging platform with programming flexibility and IT personnel to assist. This could be cost prohibitive for some programs, especially if rolled out to an entire institution.

Communication is critical for effective patient care and unfortunately, the timing of such communication is often not orderly but rather, chaotic. Text message communication can introduce interruptions into all aspects of patient care and education, not only dedicated learning conferences. If the goal is for all residents to attend all conferences, it seems impossible (and likely dangerous) to eliminate all messaging interruptions during conference hours. Nevertheless, it is worth noting that Mandel et al. have moved us creatively toward that goal with a multifaceted approach.¹¹ Future work should address more downstream outcomes, such as objective resident learning retention and adverse patient events relative to the number of interruptions per educational hour. If such studies showed improved learning outcomes and fewer adverse patient events, the next step would be to further strengthen and refine their protocol with real-time and scheduled feedback sessions between providers and other patient care team members in addition to the continued search for additional innovative approaches. In addition, combining artificial intelligence or predictive modeling may help us delineate when an interruption is warranted, for example, when a patient is at high clinical risk without intervention. Likewise, human factors research may help us understand the best way to time and execute an interruption to minimize the risk to clinical care or education. After all, the ideal system would not eliminate interruptions entirely but allow clinicians to know when someone should be interrupted and when they do not need to be interrupted.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Fifteen years ago, beepers with 5-digit call-back numbers were the norm. Pushing a call light button outside the patient’s room to flag the desk clerk that a new order had been hand-written was all part of the lived experience of residency. Using that as our baseline, we have clearly come a long way in the way that we communicate with other clinicians in hospitals. Communication among the patient care team in the digital age predominantly involves bidirectional messaging using mobile devices. The approach is both immediate and convenient. Mobile devices can improve work efficiency, patient safety, and quality of care, but their main advantage may be real-time bedside decision support.^1,2 However, the widespread use of mobile devices for communication in healthcare is not without its concerns. First and foremost, there has been abundant literature around short message service (SMS) use in the healthcare setting, and there are concerns surrounding both threats to privacy and the prevalence and impact of interruptions in clinical care.

The first SMS was sent in 1992.³ Text messaging since then has become ubiquitous, even in healthcare, raising concerns around the protection of patient health information under the Health Insurance Portability and Accountability Act (HIPAA). Interestingly, the United States Department of Health and Human Services Office for Civil Rights, enforcer of HIPAA, is tech neutral on the subject.³ Multiple studies have assessed physician stances on SMS communication in the healthcare setting using routine, non-HIPAA-compliant mobile phones. Overall, 60%-80% of respondents admitted to using SMS in patient care, while in another study, 72% and 80% of Internal Medicine residents surveyed found SMS to be the most efficient form of communication and overall preferred method of communication, respectively.^3,4 Interestingly, 82.5% of those same residents preferred the hospital-based alphanumeric paging system for security purposes, even though Freundlich et al. make a compelling argument that unidirectional alphanumeric paging systems are most certainly less HIPAA compliant, lacking encryption and password protection.⁵ Newer platforms that enable HIPAA-compliant messaging are promising, although they may not be fully adopted by clinical teams without full-scale implementation in hospitals.⁶In addition to privacy concerns with SMS applications on mobile phones, interruptions in healthcare – be it from phone calls, emails, text messages, or in-person conversations – are common. In fact, famed communication researcher Enrico Coeira has notoriously described healthcare communication as ”interrupt-driven.”⁷ Prior work has shown that frequent interruptions in the healthcare setting can lead to medication prescription errors, errors in computerized physician order entry, and even surgical procedural errors.^8-10

While studies have focused on interruptions in clinical care in the healthcare setting, little is known about how education may be compromised by interruptions due to mobile devices. Text messaging during dedicated conference time can lead to inadequate learning and a sense of frustration among residents. In this issue of the Journal of Hospital Medicine, Mendel et al. performed a quality improvement study involving eight academic inpatient clinical training units with the aim of reducing nonurgent text messages during education rounds.¹¹ Their unique interventions included learning sessions, posters, adding alerts to the digital communication platform, and alternative messaging options. Of four sequential interventions, a message alerting the sender that they will be interrupting educational rounds and suggesting a “delayed send” or “send as an FYI” showed the greatest impact, reducing the number of text interruptions per team per educational hour from 0.81 to 0.59 (95% CI 0.51-0.67). When comparing a four-week pre-intervention sample with a four-week end-intervention sample, the percentage of nonurgent messages decreased from 82% to 68% (P < .01).

While these results are promising, challenges to large-scale implementation of such a program exist. Buy-in from the ancillary healthcare team is critical for such interventions to succeed and be sustained. It also places a burden of “point triage” on the healthcare team members, who must assess the patient situation and determine the level of urgency and whether to immediately interrupt, delay interrupt or send an FYI message. For example, in the study by Mendel et al.,¹¹ it is noteworthy that urgent patient care issues were mislabeled as “FYI” in 2% of patients. While this is a seemingly low rate, even one of these mislabeled messages could result in significant adverse patient outcomes and should be considered a “never event.” Finally, the study used a messaging platform with programming flexibility and IT personnel to assist. This could be cost prohibitive for some programs, especially if rolled out to an entire institution.

Communication is critical for effective patient care and unfortunately, the timing of such communication is often not orderly but rather, chaotic. Text message communication can introduce interruptions into all aspects of patient care and education, not only dedicated learning conferences. If the goal is for all residents to attend all conferences, it seems impossible (and likely dangerous) to eliminate all messaging interruptions during conference hours. Nevertheless, it is worth noting that Mandel et al. have moved us creatively toward that goal with a multifaceted approach.¹¹ Future work should address more downstream outcomes, such as objective resident learning retention and adverse patient events relative to the number of interruptions per educational hour. If such studies showed improved learning outcomes and fewer adverse patient events, the next step would be to further strengthen and refine their protocol with real-time and scheduled feedback sessions between providers and other patient care team members in addition to the continued search for additional innovative approaches. In addition, combining artificial intelligence or predictive modeling may help us delineate when an interruption is warranted, for example, when a patient is at high clinical risk without intervention. Likewise, human factors research may help us understand the best way to time and execute an interruption to minimize the risk to clinical care or education. After all, the ideal system would not eliminate interruptions entirely but allow clinicians to know when someone should be interrupted and when they do not need to be interrupted.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Fifteen years ago, beepers with 5-digit call-back numbers were the norm. Pushing a call light button outside the patient’s room to flag the desk clerk that a new order had been hand-written was all part of the lived experience of residency. Using that as our baseline, we have clearly come a long way in the way that we communicate with other clinicians in hospitals. Communication among the patient care team in the digital age predominantly involves bidirectional messaging using mobile devices. The approach is both immediate and convenient. Mobile devices can improve work efficiency, patient safety, and quality of care, but their main advantage may be real-time bedside decision support.^1,2 However, the widespread use of mobile devices for communication in healthcare is not without its concerns. First and foremost, there has been abundant literature around short message service (SMS) use in the healthcare setting, and there are concerns surrounding both threats to privacy and the prevalence and impact of interruptions in clinical care.

The first SMS was sent in 1992.³ Text messaging since then has become ubiquitous, even in healthcare, raising concerns around the protection of patient health information under the Health Insurance Portability and Accountability Act (HIPAA). Interestingly, the United States Department of Health and Human Services Office for Civil Rights, enforcer of HIPAA, is tech neutral on the subject.³ Multiple studies have assessed physician stances on SMS communication in the healthcare setting using routine, non-HIPAA-compliant mobile phones. Overall, 60%-80% of respondents admitted to using SMS in patient care, while in another study, 72% and 80% of Internal Medicine residents surveyed found SMS to be the most efficient form of communication and overall preferred method of communication, respectively.^3,4 Interestingly, 82.5% of those same residents preferred the hospital-based alphanumeric paging system for security purposes, even though Freundlich et al. make a compelling argument that unidirectional alphanumeric paging systems are most certainly less HIPAA compliant, lacking encryption and password protection.⁵ Newer platforms that enable HIPAA-compliant messaging are promising, although they may not be fully adopted by clinical teams without full-scale implementation in hospitals.⁶In addition to privacy concerns with SMS applications on mobile phones, interruptions in healthcare – be it from phone calls, emails, text messages, or in-person conversations – are common. In fact, famed communication researcher Enrico Coeira has notoriously described healthcare communication as ”interrupt-driven.”⁷ Prior work has shown that frequent interruptions in the healthcare setting can lead to medication prescription errors, errors in computerized physician order entry, and even surgical procedural errors.^8-10

While studies have focused on interruptions in clinical care in the healthcare setting, little is known about how education may be compromised by interruptions due to mobile devices. Text messaging during dedicated conference time can lead to inadequate learning and a sense of frustration among residents. In this issue of the Journal of Hospital Medicine, Mendel et al. performed a quality improvement study involving eight academic inpatient clinical training units with the aim of reducing nonurgent text messages during education rounds.¹¹ Their unique interventions included learning sessions, posters, adding alerts to the digital communication platform, and alternative messaging options. Of four sequential interventions, a message alerting the sender that they will be interrupting educational rounds and suggesting a “delayed send” or “send as an FYI” showed the greatest impact, reducing the number of text interruptions per team per educational hour from 0.81 to 0.59 (95% CI 0.51-0.67). When comparing a four-week pre-intervention sample with a four-week end-intervention sample, the percentage of nonurgent messages decreased from 82% to 68% (P < .01).

While these results are promising, challenges to large-scale implementation of such a program exist. Buy-in from the ancillary healthcare team is critical for such interventions to succeed and be sustained. It also places a burden of “point triage” on the healthcare team members, who must assess the patient situation and determine the level of urgency and whether to immediately interrupt, delay interrupt or send an FYI message. For example, in the study by Mendel et al.,¹¹ it is noteworthy that urgent patient care issues were mislabeled as “FYI” in 2% of patients. While this is a seemingly low rate, even one of these mislabeled messages could result in significant adverse patient outcomes and should be considered a “never event.” Finally, the study used a messaging platform with programming flexibility and IT personnel to assist. This could be cost prohibitive for some programs, especially if rolled out to an entire institution.

Communication is critical for effective patient care and unfortunately, the timing of such communication is often not orderly but rather, chaotic. Text message communication can introduce interruptions into all aspects of patient care and education, not only dedicated learning conferences. If the goal is for all residents to attend all conferences, it seems impossible (and likely dangerous) to eliminate all messaging interruptions during conference hours. Nevertheless, it is worth noting that Mandel et al. have moved us creatively toward that goal with a multifaceted approach.¹¹ Future work should address more downstream outcomes, such as objective resident learning retention and adverse patient events relative to the number of interruptions per educational hour. If such studies showed improved learning outcomes and fewer adverse patient events, the next step would be to further strengthen and refine their protocol with real-time and scheduled feedback sessions between providers and other patient care team members in addition to the continued search for additional innovative approaches. In addition, combining artificial intelligence or predictive modeling may help us delineate when an interruption is warranted, for example, when a patient is at high clinical risk without intervention. Likewise, human factors research may help us understand the best way to time and execute an interruption to minimize the risk to clinical care or education. After all, the ideal system would not eliminate interruptions entirely but allow clinicians to know when someone should be interrupted and when they do not need to be interrupted.

Disclosures

The authors have no financial relationships relevant to this article to disclose.

Health literacy (HL), defined as patients’ ability to understand health information and make health decisions,¹ is a prevalent problem in the outpatient and inpatient settings.^2,3 In both settings, low HL has adverse implications for self-care including interpreting health labels⁴ and taking medications correctly.⁵ Among outpatient cohorts, HL has been associated with worse outcomes and acute care utilization.⁶ Associations with low HL include increased hospitalizations,⁷ rehospitalizations,^8,9 emergency department visits,¹⁰ and decreased preventative care use.¹¹ Among the elderly, low HL is associated with increased mortality¹² and decreased self-perception of health.¹³

A systematic review revealed that most high-quality HL outcome studies were conducted in the outpatient setting.⁶ There have been very few studies assessing effects of low HL in an acute-care setting.^7,14 These studies have evaluated postdischarge outcomes, including admissions or readmissions,^7-9 and medication knowledge.¹⁴ To the best of our knowledge, there are no studies evaluating associations between HL and hospital length of stay (LOS).

LOS has received much attention as providers and payers focus more on resource utilization and eliminating adverse effects of prolonged hospitalization.¹⁵ LOS is multifactorial, depending on clinical characteristics like disease severity, as well as on sociocultural, demographic, and geographic factors.¹⁶ Despite evidence that LOS reductions translate into improved resource allocation and potentially fewer complications, there remains a tension between the appropriate LOS and one that is too short for a given condition.¹⁷

Because low HL is associated with inefficient resource utilization, we hypothesized that low HL would be associated with increased LOS after controlling for illness severity. Our objectives were to evaluate the association between low HL and LOS and whether such an association was modified by illness severity and sociodemographics.

Study Design, Setting, Participants

An in-hospital, cohort study design of patients who were admitted or transferred to the general medicine service at the University of Chicago between October 2012 and November 2015 and screened for inclusion as part of a large, ongoing study of inpatient care quality was conducted.¹⁸ Exclusion criteria included observation status, age under 18 years, non-English speaking, and repeat participants. Those who died during hospitalization or whose discharge status was missing were excluded because the primary goal was to examine the association of HL and time to discharge, which could not be evaluated among those who died. We excluded participants with LOS >30 days to limit overly influential effects of extreme outliers (1% of the population).

Variables

HL was screened using the Brief Health Literacy Screen (BHLS), a validated, 3-question verbal survey not requiring adequate visual acuity to assess HL.^19,20 The 3 questions are as follows: (1) “How confident are you filling out medical forms by yourself?”, (2) “How often do you have someone help you read hospital materials?”, and (3) “How often do you have problems learning about your medical condition because of difficulty understanding written information?” Responses to the questions were scored on a 5-point Likert scale in which higher scores corresponded to higher HL.^21,22 The scores for each of the 3 questions were summed to yield a range between 3 and 15. On the individual questions, prior work has demonstrated improved test performance with a cutoff of ≤3, which corresponds to a response of “some of the time” or “somewhat”; therefore, when the 3 questions were summed together, scores of ≤9 were considered indicative of low HL.^21,23

For severity of illness adjustment, we used relative weights derived from the 3M (3M, Maplewood, MN) All Patient Refined Diagnosis Related Groups (APR-DRG) classification system, which uses administrative data to classify the severity. The APR-DRG system assigns each admission to a DRG based on principal diagnosis; for each DRG, patients are then subdivided into 4 severity classes based on age, comorbidity, and interactions between these variables and the admitting diagnosis.²⁴ Using the base DRG and severity score, the system assigns relative weights that reflect differences in expected hospital resource utilization.

LOS was derived from hospital administrative data and counted from the date of admission to the hospital. Participants who were discharged on the day of admission were counted as having an LOS of 1. Insurance status (Medicare, Medicaid, no payer, private) also was obtained from administrative data. Age, sex (male or female), education (junior high or less, some high school, high school graduate, some college, college graduate, postgraduate), and race (black/African American, white, Asian or Pacific Islander [including Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, other Asian, Native Hawaiian, Guam/Chamorro, Samoan, other Pacific], American Indian or Alaskan Native, multiple race) were obtained from administrative data based on information provided by the patient. Participants with missing data on any of the sociodemographic variables or on the APR-DRG score were excluded from the analysis.

Statistical Analysis

χ² and 2-tailed t tests were used to compare categorical and continuous variables, respectively. Multivariate linear regressions were employed to measure associations between the independent variables (HL, illness severity, race, gender, education, and insurance status) and the dependent variable, LOS. Independent variables were chosen for clinical significance and retained in the model regardless of statistical significance. The adjusted R² values of models with and without the HL variable included were reported to provide information on the contribution of HL to the overall model.

Because LOS was observed to be right skewed and residuals of the untransformed regression were observed to be non-normally distributed, the decision was made to natural log transform LOS, which is consistent with previous hospital LOS studies.¹⁶ Regression coefficients and confidence intervals were then transformed into percentage estimates using the following equation: 100(e^β–1). Adjusted R² was reported for the transformed regression.

The APR-DRG relative weight was treated as a continuous variable. Sociodemographic variables were dichotomized as follows: female vs male; high school graduates vs not; African American vs not; Medicaid/no payer vs Medicare/private payer. Age was not included in the multivariate model because it has been incorporated into the weighted APR-DRG illness severity scores.

Each of the sociodemographic variables and the APR-DRG score were examined for effect modification via the same multivariate linear equation described above, with the addition of an interaction term. A separate regression was performed with an interaction term between age (dichotomized at ≥65) and HL to investigate whether age modified the association between HL and LOS. Finally, we explored whether effects were isolated to long vs short LOS by dividing the sample based on the mean LOS (≥6 days) and performing separate multivariate comparisons.

Sensitivity analyses were performed to exclude those with LOS greater than the 90th percentile and those with APR-DRG score greater than the 90th percentile; age was added to the model as a continuous variable to evaluate whether the illness severity score fully adjusted for the effects of age on LOS. Furthermore, we compared the participants with missing data to those with complete data across both dependent and independent variables. Alpha was set at 0.05; analyses were performed using Stata Version 14 (Stata, College Station, TX).

A total of 5983 participants met inclusion criteria and completed the HL assessment; of these participants, 75 (1%) died during hospitalization, 9 (0.2%) had missing discharge status, and 79 (1%) had LOS >30 days. Two hundred eighty (5%) were missing data on sociodemographic variables or APR-DRG score. Of the remaining (n = 5540), the mean age was 57 years (standard deviation [SD] = 19 years), over half of participants were female (57%), and the majority were African American (73%) and had graduated from high school (81%). The sample was divided into those with private insurance (25%), those with Medicare (46%), and those with Medicaid (26%); 2% had no payer. The mean APR-DRG score was 1.3 (SD = 1.2), and the scores ranged from 0.3 to 15.8.

On the BHLS screen for HL, 20% (1104/5540) had inadequate HL. Participants with low HL had higher weighted illness severity scores (average 1.4 vs 1.3; P = 0.003). Participants with low HL were also more likely to be 65 or older (55% vs 33%; P < 0.001), non-high school graduates (35% vs 15%; P < 0.001), and African American (78% vs 72%; P < 0.001), and to have Medicare or private insurance (75% vs 71%; P = 0.02). There was no significant difference with respect to gender (54% male vs 57% female; P = 0.1)

To our knowledge, this study is the first to evaluate the association between low HL and an important in-hospital outcome measure, hospital LOS. We found that low HL was associated with a longer hospital LOS, a result which remained significant when controlling for severity of illness and sociodemographic variables and when testing the model for sensitivity to the highest values of LOS and illness severity. Additionally, the association of HL with LOS appeared concentrated among participants with shorter LOS. Relative to other predictors, the contribution of HL to the overall LOS model was small, as evidenced by the change in adjusted R² values with HL excluded.

Among the covariates, only gender modified the association between HL and LOS; the findings suggested that men were more susceptible to the effect of low HL on increased LOS. Illness severity and other sociodemographics, including age ≥65, did not appear to modify the association. We also found that being African American and having Medicaid or no insurance were associated with a significantly shorter LOS in multivariate analysis.

Previous work suggested that the adverse health effects of low HL may be mediated through several pathways, including health knowledge, self-efficacy, health skills, and illness stigma.^25-27 The finding of a small but significant relationship between HL and LOS was not surprising given these known associations; nevertheless, there may be an additional patient-dependent effect of low HL on LOS not discovered here. For instance, patients with poor health knowledge and self-efficacy might stay in the hospital longer if they or their providers do not feel comfortable with their self-care ability.

This finding may be useful in developing hospital-based interventions. HL-specific interventions, several of which have been tested in the inpatient setting,^14,28,29 have shown promise toward improving health knowledge,³⁰ disease severity,³¹ and health resource utilization.³²

Those with low HL may lack the self-efficacy to participate in discharge planning; in fact, previous work has related low HL to posthospital readmissions.^8,9 Conversely, patients with low HL might struggle to engage in the inpatient milieu, advocating for shorter LOS if they feel alienated by the inpatient experience.

These possibilities show that LOS is a complex measure shown to depend on patient-level characteristics and on provider-based, geographical, and sociocultural factors.^16,33 With these forces at play, additional effects of lower levels of HL may be lost without phenotyping patients by both level of HL and related characteristics, such as self-efficacy, health skills, and stigma. By gathering these additional data, future work should explore whether subpopulations of patients with low HL may be at risk for too-short vs too-long hospital admissions.

For instance, in this study, both race and Medicaid insurance were associated with shorter LOS. Being African American was associated with shorter LOS in our study but has been found to be associated with longer LOS in another study specifically focused on diabetes.³⁴ Prior findings found uninsured patients have shorter LOS.³⁵ Therefore, these findings in our study are difficult to explain without further work to understand whether there are health disparities in the way patients are cared for during hospitalization that may shorten or lengthen their LOS because of factors outside of their clinical need.

The finding that gender modified the effect of low HL on LOS was unexpected. There were similar proportions of men and women with low HL. There is evidence to support that women make the majority of health decisions for themselves and their familes³⁶; therefore, there may be unmeasured aspects of HL that provide an advantage for female vs male inpatients. Furthermore, omitted confounders, such as social support, may not fully capture potential gender-related differences. Future work is needed to understand the role of gender in relationship to HL and LOS.

Limitations of this study include its observational, single-centered design with information derived from administrative data; positive and negative confounding cannot be ruled out. For instance, we did not control for complex aspects affecting LOS, such as discharge disposition and goals of care (eg, aggressive care after discharge vs hospice). To address this limitation, multivariate analyses were performed, which were adjusted for illness severity scores and took into account both comorbidity and severity of the current illness. Additionally, although it is important to study such populations, our largely urban, minority sample is not representative of the U.S. population, and within our large sample, there were participants with missing data who had lower HL on average, although this group represented only 5% of the sample. Finally, different HL tools have noncomplete concordance, which has been seen when comparing the BHLS with more objective tools.^20,37 Furthermore, certain in-hospital clinical scenarios (eg, recent stroke or prolonged intensive care unit stay) may present unique challenges in establishing a baseline HL level. However, the BHLS was used in this study because of its greater feasibility.

In conclusion, this study is the first to evaluate the relationship between low HL and LOS. The findings suggest that HL may play a role in shaping outcomes in the inpatient setting and that targeting interventions toward screened patients may be a pathway toward mitigating adverse effects. Our findings need to be replicated in larger, more representative samples, and further work understanding subpopulations within the low HL population is needed. Future work should measure this association in diverse inpatient settings (eg, psychiatric, surgical, and specialty), in addition to assessing associations between HL and other important in-hospital outcome measures, including mortality and discharge disposition.

Acknowledgments

The authors thank the Hospitalist Project team for their assistance with data collection. The authors especially thank Chuanhong Liao and Ashley Snyder for assistance with statistical analyses; Andrea Flores, Ainoa Coltri, and Tom Best for their assistance with data management. The authors would also like to thank Nicole Twu for her help with preparing and editing the manuscript.

Disclosures

Dr. Jaffee was supported by a Calvin Fentress Research Fellowship and NIH R25MH094612. Dr. Press was supported by a career development award (NHLBI K23HL118151). This work was also supported by a seed grant from the Center for Health Administration Studies. All other authors declare no conflicts of interest.

Health literacy (HL), defined as patients’ ability to understand health information and make health decisions,¹ is a prevalent problem in the outpatient and inpatient settings.^2,3 In both settings, low HL has adverse implications for self-care including interpreting health labels⁴ and taking medications correctly.⁵ Among outpatient cohorts, HL has been associated with worse outcomes and acute care utilization.⁶ Associations with low HL include increased hospitalizations,⁷ rehospitalizations,^8,9 emergency department visits,¹⁰ and decreased preventative care use.¹¹ Among the elderly, low HL is associated with increased mortality¹² and decreased self-perception of health.¹³

A systematic review revealed that most high-quality HL outcome studies were conducted in the outpatient setting.⁶ There have been very few studies assessing effects of low HL in an acute-care setting.^7,14 These studies have evaluated postdischarge outcomes, including admissions or readmissions,^7-9 and medication knowledge.¹⁴ To the best of our knowledge, there are no studies evaluating associations between HL and hospital length of stay (LOS).

LOS has received much attention as providers and payers focus more on resource utilization and eliminating adverse effects of prolonged hospitalization.¹⁵ LOS is multifactorial, depending on clinical characteristics like disease severity, as well as on sociocultural, demographic, and geographic factors.¹⁶ Despite evidence that LOS reductions translate into improved resource allocation and potentially fewer complications, there remains a tension between the appropriate LOS and one that is too short for a given condition.¹⁷

Because low HL is associated with inefficient resource utilization, we hypothesized that low HL would be associated with increased LOS after controlling for illness severity. Our objectives were to evaluate the association between low HL and LOS and whether such an association was modified by illness severity and sociodemographics.

Study Design, Setting, Participants

An in-hospital, cohort study design of patients who were admitted or transferred to the general medicine service at the University of Chicago between October 2012 and November 2015 and screened for inclusion as part of a large, ongoing study of inpatient care quality was conducted.¹⁸ Exclusion criteria included observation status, age under 18 years, non-English speaking, and repeat participants. Those who died during hospitalization or whose discharge status was missing were excluded because the primary goal was to examine the association of HL and time to discharge, which could not be evaluated among those who died. We excluded participants with LOS >30 days to limit overly influential effects of extreme outliers (1% of the population).

Variables

HL was screened using the Brief Health Literacy Screen (BHLS), a validated, 3-question verbal survey not requiring adequate visual acuity to assess HL.^19,20 The 3 questions are as follows: (1) “How confident are you filling out medical forms by yourself?”, (2) “How often do you have someone help you read hospital materials?”, and (3) “How often do you have problems learning about your medical condition because of difficulty understanding written information?” Responses to the questions were scored on a 5-point Likert scale in which higher scores corresponded to higher HL.^21,22 The scores for each of the 3 questions were summed to yield a range between 3 and 15. On the individual questions, prior work has demonstrated improved test performance with a cutoff of ≤3, which corresponds to a response of “some of the time” or “somewhat”; therefore, when the 3 questions were summed together, scores of ≤9 were considered indicative of low HL.^21,23

For severity of illness adjustment, we used relative weights derived from the 3M (3M, Maplewood, MN) All Patient Refined Diagnosis Related Groups (APR-DRG) classification system, which uses administrative data to classify the severity. The APR-DRG system assigns each admission to a DRG based on principal diagnosis; for each DRG, patients are then subdivided into 4 severity classes based on age, comorbidity, and interactions between these variables and the admitting diagnosis.²⁴ Using the base DRG and severity score, the system assigns relative weights that reflect differences in expected hospital resource utilization.

LOS was derived from hospital administrative data and counted from the date of admission to the hospital. Participants who were discharged on the day of admission were counted as having an LOS of 1. Insurance status (Medicare, Medicaid, no payer, private) also was obtained from administrative data. Age, sex (male or female), education (junior high or less, some high school, high school graduate, some college, college graduate, postgraduate), and race (black/African American, white, Asian or Pacific Islander [including Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, other Asian, Native Hawaiian, Guam/Chamorro, Samoan, other Pacific], American Indian or Alaskan Native, multiple race) were obtained from administrative data based on information provided by the patient. Participants with missing data on any of the sociodemographic variables or on the APR-DRG score were excluded from the analysis.

Statistical Analysis

χ² and 2-tailed t tests were used to compare categorical and continuous variables, respectively. Multivariate linear regressions were employed to measure associations between the independent variables (HL, illness severity, race, gender, education, and insurance status) and the dependent variable, LOS. Independent variables were chosen for clinical significance and retained in the model regardless of statistical significance. The adjusted R² values of models with and without the HL variable included were reported to provide information on the contribution of HL to the overall model.

Because LOS was observed to be right skewed and residuals of the untransformed regression were observed to be non-normally distributed, the decision was made to natural log transform LOS, which is consistent with previous hospital LOS studies.¹⁶ Regression coefficients and confidence intervals were then transformed into percentage estimates using the following equation: 100(e^β–1). Adjusted R² was reported for the transformed regression.

The APR-DRG relative weight was treated as a continuous variable. Sociodemographic variables were dichotomized as follows: female vs male; high school graduates vs not; African American vs not; Medicaid/no payer vs Medicare/private payer. Age was not included in the multivariate model because it has been incorporated into the weighted APR-DRG illness severity scores.

Each of the sociodemographic variables and the APR-DRG score were examined for effect modification via the same multivariate linear equation described above, with the addition of an interaction term. A separate regression was performed with an interaction term between age (dichotomized at ≥65) and HL to investigate whether age modified the association between HL and LOS. Finally, we explored whether effects were isolated to long vs short LOS by dividing the sample based on the mean LOS (≥6 days) and performing separate multivariate comparisons.

Sensitivity analyses were performed to exclude those with LOS greater than the 90th percentile and those with APR-DRG score greater than the 90th percentile; age was added to the model as a continuous variable to evaluate whether the illness severity score fully adjusted for the effects of age on LOS. Furthermore, we compared the participants with missing data to those with complete data across both dependent and independent variables. Alpha was set at 0.05; analyses were performed using Stata Version 14 (Stata, College Station, TX).

A total of 5983 participants met inclusion criteria and completed the HL assessment; of these participants, 75 (1%) died during hospitalization, 9 (0.2%) had missing discharge status, and 79 (1%) had LOS >30 days. Two hundred eighty (5%) were missing data on sociodemographic variables or APR-DRG score. Of the remaining (n = 5540), the mean age was 57 years (standard deviation [SD] = 19 years), over half of participants were female (57%), and the majority were African American (73%) and had graduated from high school (81%). The sample was divided into those with private insurance (25%), those with Medicare (46%), and those with Medicaid (26%); 2% had no payer. The mean APR-DRG score was 1.3 (SD = 1.2), and the scores ranged from 0.3 to 15.8.

On the BHLS screen for HL, 20% (1104/5540) had inadequate HL. Participants with low HL had higher weighted illness severity scores (average 1.4 vs 1.3; P = 0.003). Participants with low HL were also more likely to be 65 or older (55% vs 33%; P < 0.001), non-high school graduates (35% vs 15%; P < 0.001), and African American (78% vs 72%; P < 0.001), and to have Medicare or private insurance (75% vs 71%; P = 0.02). There was no significant difference with respect to gender (54% male vs 57% female; P = 0.1)

To our knowledge, this study is the first to evaluate the association between low HL and an important in-hospital outcome measure, hospital LOS. We found that low HL was associated with a longer hospital LOS, a result which remained significant when controlling for severity of illness and sociodemographic variables and when testing the model for sensitivity to the highest values of LOS and illness severity. Additionally, the association of HL with LOS appeared concentrated among participants with shorter LOS. Relative to other predictors, the contribution of HL to the overall LOS model was small, as evidenced by the change in adjusted R² values with HL excluded.

Among the covariates, only gender modified the association between HL and LOS; the findings suggested that men were more susceptible to the effect of low HL on increased LOS. Illness severity and other sociodemographics, including age ≥65, did not appear to modify the association. We also found that being African American and having Medicaid or no insurance were associated with a significantly shorter LOS in multivariate analysis.

Previous work suggested that the adverse health effects of low HL may be mediated through several pathways, including health knowledge, self-efficacy, health skills, and illness stigma.^25-27 The finding of a small but significant relationship between HL and LOS was not surprising given these known associations; nevertheless, there may be an additional patient-dependent effect of low HL on LOS not discovered here. For instance, patients with poor health knowledge and self-efficacy might stay in the hospital longer if they or their providers do not feel comfortable with their self-care ability.

This finding may be useful in developing hospital-based interventions. HL-specific interventions, several of which have been tested in the inpatient setting,^14,28,29 have shown promise toward improving health knowledge,³⁰ disease severity,³¹ and health resource utilization.³²

Those with low HL may lack the self-efficacy to participate in discharge planning; in fact, previous work has related low HL to posthospital readmissions.^8,9 Conversely, patients with low HL might struggle to engage in the inpatient milieu, advocating for shorter LOS if they feel alienated by the inpatient experience.

These possibilities show that LOS is a complex measure shown to depend on patient-level characteristics and on provider-based, geographical, and sociocultural factors.^16,33 With these forces at play, additional effects of lower levels of HL may be lost without phenotyping patients by both level of HL and related characteristics, such as self-efficacy, health skills, and stigma. By gathering these additional data, future work should explore whether subpopulations of patients with low HL may be at risk for too-short vs too-long hospital admissions.

For instance, in this study, both race and Medicaid insurance were associated with shorter LOS. Being African American was associated with shorter LOS in our study but has been found to be associated with longer LOS in another study specifically focused on diabetes.³⁴ Prior findings found uninsured patients have shorter LOS.³⁵ Therefore, these findings in our study are difficult to explain without further work to understand whether there are health disparities in the way patients are cared for during hospitalization that may shorten or lengthen their LOS because of factors outside of their clinical need.

The finding that gender modified the effect of low HL on LOS was unexpected. There were similar proportions of men and women with low HL. There is evidence to support that women make the majority of health decisions for themselves and their familes³⁶; therefore, there may be unmeasured aspects of HL that provide an advantage for female vs male inpatients. Furthermore, omitted confounders, such as social support, may not fully capture potential gender-related differences. Future work is needed to understand the role of gender in relationship to HL and LOS.

Limitations of this study include its observational, single-centered design with information derived from administrative data; positive and negative confounding cannot be ruled out. For instance, we did not control for complex aspects affecting LOS, such as discharge disposition and goals of care (eg, aggressive care after discharge vs hospice). To address this limitation, multivariate analyses were performed, which were adjusted for illness severity scores and took into account both comorbidity and severity of the current illness. Additionally, although it is important to study such populations, our largely urban, minority sample is not representative of the U.S. population, and within our large sample, there were participants with missing data who had lower HL on average, although this group represented only 5% of the sample. Finally, different HL tools have noncomplete concordance, which has been seen when comparing the BHLS with more objective tools.^20,37 Furthermore, certain in-hospital clinical scenarios (eg, recent stroke or prolonged intensive care unit stay) may present unique challenges in establishing a baseline HL level. However, the BHLS was used in this study because of its greater feasibility.

In conclusion, this study is the first to evaluate the relationship between low HL and LOS. The findings suggest that HL may play a role in shaping outcomes in the inpatient setting and that targeting interventions toward screened patients may be a pathway toward mitigating adverse effects. Our findings need to be replicated in larger, more representative samples, and further work understanding subpopulations within the low HL population is needed. Future work should measure this association in diverse inpatient settings (eg, psychiatric, surgical, and specialty), in addition to assessing associations between HL and other important in-hospital outcome measures, including mortality and discharge disposition.

Acknowledgments

The authors thank the Hospitalist Project team for their assistance with data collection. The authors especially thank Chuanhong Liao and Ashley Snyder for assistance with statistical analyses; Andrea Flores, Ainoa Coltri, and Tom Best for their assistance with data management. The authors would also like to thank Nicole Twu for her help with preparing and editing the manuscript.

Disclosures

Dr. Jaffee was supported by a Calvin Fentress Research Fellowship and NIH R25MH094612. Dr. Press was supported by a career development award (NHLBI K23HL118151). This work was also supported by a seed grant from the Center for Health Administration Studies. All other authors declare no conflicts of interest.

Health literacy (HL), defined as patients’ ability to understand health information and make health decisions,¹ is a prevalent problem in the outpatient and inpatient settings.^2,3 In both settings, low HL has adverse implications for self-care including interpreting health labels⁴ and taking medications correctly.⁵ Among outpatient cohorts, HL has been associated with worse outcomes and acute care utilization.⁶ Associations with low HL include increased hospitalizations,⁷ rehospitalizations,^8,9 emergency department visits,¹⁰ and decreased preventative care use.¹¹ Among the elderly, low HL is associated with increased mortality¹² and decreased self-perception of health.¹³

A systematic review revealed that most high-quality HL outcome studies were conducted in the outpatient setting.⁶ There have been very few studies assessing effects of low HL in an acute-care setting.^7,14 These studies have evaluated postdischarge outcomes, including admissions or readmissions,^7-9 and medication knowledge.¹⁴ To the best of our knowledge, there are no studies evaluating associations between HL and hospital length of stay (LOS).

LOS has received much attention as providers and payers focus more on resource utilization and eliminating adverse effects of prolonged hospitalization.¹⁵ LOS is multifactorial, depending on clinical characteristics like disease severity, as well as on sociocultural, demographic, and geographic factors.¹⁶ Despite evidence that LOS reductions translate into improved resource allocation and potentially fewer complications, there remains a tension between the appropriate LOS and one that is too short for a given condition.¹⁷

Because low HL is associated with inefficient resource utilization, we hypothesized that low HL would be associated with increased LOS after controlling for illness severity. Our objectives were to evaluate the association between low HL and LOS and whether such an association was modified by illness severity and sociodemographics.

Study Design, Setting, Participants

An in-hospital, cohort study design of patients who were admitted or transferred to the general medicine service at the University of Chicago between October 2012 and November 2015 and screened for inclusion as part of a large, ongoing study of inpatient care quality was conducted.¹⁸ Exclusion criteria included observation status, age under 18 years, non-English speaking, and repeat participants. Those who died during hospitalization or whose discharge status was missing were excluded because the primary goal was to examine the association of HL and time to discharge, which could not be evaluated among those who died. We excluded participants with LOS >30 days to limit overly influential effects of extreme outliers (1% of the population).

Variables

HL was screened using the Brief Health Literacy Screen (BHLS), a validated, 3-question verbal survey not requiring adequate visual acuity to assess HL.^19,20 The 3 questions are as follows: (1) “How confident are you filling out medical forms by yourself?”, (2) “How often do you have someone help you read hospital materials?”, and (3) “How often do you have problems learning about your medical condition because of difficulty understanding written information?” Responses to the questions were scored on a 5-point Likert scale in which higher scores corresponded to higher HL.^21,22 The scores for each of the 3 questions were summed to yield a range between 3 and 15. On the individual questions, prior work has demonstrated improved test performance with a cutoff of ≤3, which corresponds to a response of “some of the time” or “somewhat”; therefore, when the 3 questions were summed together, scores of ≤9 were considered indicative of low HL.^21,23

For severity of illness adjustment, we used relative weights derived from the 3M (3M, Maplewood, MN) All Patient Refined Diagnosis Related Groups (APR-DRG) classification system, which uses administrative data to classify the severity. The APR-DRG system assigns each admission to a DRG based on principal diagnosis; for each DRG, patients are then subdivided into 4 severity classes based on age, comorbidity, and interactions between these variables and the admitting diagnosis.²⁴ Using the base DRG and severity score, the system assigns relative weights that reflect differences in expected hospital resource utilization.

LOS was derived from hospital administrative data and counted from the date of admission to the hospital. Participants who were discharged on the day of admission were counted as having an LOS of 1. Insurance status (Medicare, Medicaid, no payer, private) also was obtained from administrative data. Age, sex (male or female), education (junior high or less, some high school, high school graduate, some college, college graduate, postgraduate), and race (black/African American, white, Asian or Pacific Islander [including Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, other Asian, Native Hawaiian, Guam/Chamorro, Samoan, other Pacific], American Indian or Alaskan Native, multiple race) were obtained from administrative data based on information provided by the patient. Participants with missing data on any of the sociodemographic variables or on the APR-DRG score were excluded from the analysis.

Statistical Analysis

χ² and 2-tailed t tests were used to compare categorical and continuous variables, respectively. Multivariate linear regressions were employed to measure associations between the independent variables (HL, illness severity, race, gender, education, and insurance status) and the dependent variable, LOS. Independent variables were chosen for clinical significance and retained in the model regardless of statistical significance. The adjusted R² values of models with and without the HL variable included were reported to provide information on the contribution of HL to the overall model.

Because LOS was observed to be right skewed and residuals of the untransformed regression were observed to be non-normally distributed, the decision was made to natural log transform LOS, which is consistent with previous hospital LOS studies.¹⁶ Regression coefficients and confidence intervals were then transformed into percentage estimates using the following equation: 100(e^β–1). Adjusted R² was reported for the transformed regression.

The APR-DRG relative weight was treated as a continuous variable. Sociodemographic variables were dichotomized as follows: female vs male; high school graduates vs not; African American vs not; Medicaid/no payer vs Medicare/private payer. Age was not included in the multivariate model because it has been incorporated into the weighted APR-DRG illness severity scores.

Each of the sociodemographic variables and the APR-DRG score were examined for effect modification via the same multivariate linear equation described above, with the addition of an interaction term. A separate regression was performed with an interaction term between age (dichotomized at ≥65) and HL to investigate whether age modified the association between HL and LOS. Finally, we explored whether effects were isolated to long vs short LOS by dividing the sample based on the mean LOS (≥6 days) and performing separate multivariate comparisons.

Sensitivity analyses were performed to exclude those with LOS greater than the 90th percentile and those with APR-DRG score greater than the 90th percentile; age was added to the model as a continuous variable to evaluate whether the illness severity score fully adjusted for the effects of age on LOS. Furthermore, we compared the participants with missing data to those with complete data across both dependent and independent variables. Alpha was set at 0.05; analyses were performed using Stata Version 14 (Stata, College Station, TX).

A total of 5983 participants met inclusion criteria and completed the HL assessment; of these participants, 75 (1%) died during hospitalization, 9 (0.2%) had missing discharge status, and 79 (1%) had LOS >30 days. Two hundred eighty (5%) were missing data on sociodemographic variables or APR-DRG score. Of the remaining (n = 5540), the mean age was 57 years (standard deviation [SD] = 19 years), over half of participants were female (57%), and the majority were African American (73%) and had graduated from high school (81%). The sample was divided into those with private insurance (25%), those with Medicare (46%), and those with Medicaid (26%); 2% had no payer. The mean APR-DRG score was 1.3 (SD = 1.2), and the scores ranged from 0.3 to 15.8.

On the BHLS screen for HL, 20% (1104/5540) had inadequate HL. Participants with low HL had higher weighted illness severity scores (average 1.4 vs 1.3; P = 0.003). Participants with low HL were also more likely to be 65 or older (55% vs 33%; P < 0.001), non-high school graduates (35% vs 15%; P < 0.001), and African American (78% vs 72%; P < 0.001), and to have Medicare or private insurance (75% vs 71%; P = 0.02). There was no significant difference with respect to gender (54% male vs 57% female; P = 0.1)

To our knowledge, this study is the first to evaluate the association between low HL and an important in-hospital outcome measure, hospital LOS. We found that low HL was associated with a longer hospital LOS, a result which remained significant when controlling for severity of illness and sociodemographic variables and when testing the model for sensitivity to the highest values of LOS and illness severity. Additionally, the association of HL with LOS appeared concentrated among participants with shorter LOS. Relative to other predictors, the contribution of HL to the overall LOS model was small, as evidenced by the change in adjusted R² values with HL excluded.

Among the covariates, only gender modified the association between HL and LOS; the findings suggested that men were more susceptible to the effect of low HL on increased LOS. Illness severity and other sociodemographics, including age ≥65, did not appear to modify the association. We also found that being African American and having Medicaid or no insurance were associated with a significantly shorter LOS in multivariate analysis.

Previous work suggested that the adverse health effects of low HL may be mediated through several pathways, including health knowledge, self-efficacy, health skills, and illness stigma.^25-27 The finding of a small but significant relationship between HL and LOS was not surprising given these known associations; nevertheless, there may be an additional patient-dependent effect of low HL on LOS not discovered here. For instance, patients with poor health knowledge and self-efficacy might stay in the hospital longer if they or their providers do not feel comfortable with their self-care ability.

This finding may be useful in developing hospital-based interventions. HL-specific interventions, several of which have been tested in the inpatient setting,^14,28,29 have shown promise toward improving health knowledge,³⁰ disease severity,³¹ and health resource utilization.³²

Those with low HL may lack the self-efficacy to participate in discharge planning; in fact, previous work has related low HL to posthospital readmissions.^8,9 Conversely, patients with low HL might struggle to engage in the inpatient milieu, advocating for shorter LOS if they feel alienated by the inpatient experience.

These possibilities show that LOS is a complex measure shown to depend on patient-level characteristics and on provider-based, geographical, and sociocultural factors.^16,33 With these forces at play, additional effects of lower levels of HL may be lost without phenotyping patients by both level of HL and related characteristics, such as self-efficacy, health skills, and stigma. By gathering these additional data, future work should explore whether subpopulations of patients with low HL may be at risk for too-short vs too-long hospital admissions.

For instance, in this study, both race and Medicaid insurance were associated with shorter LOS. Being African American was associated with shorter LOS in our study but has been found to be associated with longer LOS in another study specifically focused on diabetes.³⁴ Prior findings found uninsured patients have shorter LOS.³⁵ Therefore, these findings in our study are difficult to explain without further work to understand whether there are health disparities in the way patients are cared for during hospitalization that may shorten or lengthen their LOS because of factors outside of their clinical need.

The finding that gender modified the effect of low HL on LOS was unexpected. There were similar proportions of men and women with low HL. There is evidence to support that women make the majority of health decisions for themselves and their familes³⁶; therefore, there may be unmeasured aspects of HL that provide an advantage for female vs male inpatients. Furthermore, omitted confounders, such as social support, may not fully capture potential gender-related differences. Future work is needed to understand the role of gender in relationship to HL and LOS.

Limitations of this study include its observational, single-centered design with information derived from administrative data; positive and negative confounding cannot be ruled out. For instance, we did not control for complex aspects affecting LOS, such as discharge disposition and goals of care (eg, aggressive care after discharge vs hospice). To address this limitation, multivariate analyses were performed, which were adjusted for illness severity scores and took into account both comorbidity and severity of the current illness. Additionally, although it is important to study such populations, our largely urban, minority sample is not representative of the U.S. population, and within our large sample, there were participants with missing data who had lower HL on average, although this group represented only 5% of the sample. Finally, different HL tools have noncomplete concordance, which has been seen when comparing the BHLS with more objective tools.^20,37 Furthermore, certain in-hospital clinical scenarios (eg, recent stroke or prolonged intensive care unit stay) may present unique challenges in establishing a baseline HL level. However, the BHLS was used in this study because of its greater feasibility.

In conclusion, this study is the first to evaluate the relationship between low HL and LOS. The findings suggest that HL may play a role in shaping outcomes in the inpatient setting and that targeting interventions toward screened patients may be a pathway toward mitigating adverse effects. Our findings need to be replicated in larger, more representative samples, and further work understanding subpopulations within the low HL population is needed. Future work should measure this association in diverse inpatient settings (eg, psychiatric, surgical, and specialty), in addition to assessing associations between HL and other important in-hospital outcome measures, including mortality and discharge disposition.

Acknowledgments

The authors thank the Hospitalist Project team for their assistance with data collection. The authors especially thank Chuanhong Liao and Ashley Snyder for assistance with statistical analyses; Andrea Flores, Ainoa Coltri, and Tom Best for their assistance with data management. The authors would also like to thank Nicole Twu for her help with preparing and editing the manuscript.

Disclosures

Dr. Jaffee was supported by a Calvin Fentress Research Fellowship and NIH R25MH094612. Dr. Press was supported by a career development award (NHLBI K23HL118151). This work was also supported by a seed grant from the Center for Health Administration Studies. All other authors declare no conflicts of interest.

Evidence suggests that troponin-only testing is the superior strategy to diagnose acute myocardial infarction (AMI).¹ Because of this, in February 2015, the Choosing Wisely^® campaign issued a recommendation to use troponin I or T to diagnose AMI, and not to test for myoglobin or creatine kinase-MB (CK-MB).² This recommendation was in line with guidelines from the American Heart Association and the American College of Cardiology, which recommended that myoglobin and CK-MB are not useful and offer no benefit for the diagnosis of acute coronary syndrome.³ Some institutions have developed interventions to promote troponin-only testing, reporting substantial cost savings and no negative consequences.^4,5

Despite these successes, it is likely that institutions vary with respect to the adoption of the Choosing Wisely^® troponin-only testing recommendation.⁶ Implementing this recommendation requires both promoting clinician behavior change and a strong institutional culture of high-value care.⁷ Understanding the variation across institutions of troponin-only testing could inform how to promote high-value care recommendations nationwide. We aimed to describe patterns of troponin, myoglobin, and CK-MB testing in a sample of academic teaching hospitals before and after the Choosing Wisely^® recommendation.

Troponin, myoglobin, and CK-MB ordering data were extracted from Vizient’s (formerly University HealthSystem Consortium, Chicago, IL) Clinical Database/Resource Manager (CDB/RM^®) for all patients with a principal discharge diagnosis of AMI at all hospitals reporting all 36 months from the fourth quarter of 2013 through the third quarter of 2016. This period includes time both before and after the Choosing Wisely^® recommendation, which was released in the first quarter of 2015. Vizient’s CDB/RM contains ordering data for 300 academic medical centers and their affiliated hospitals and includes the discharge diagnoses for patients cared for by these institutions. Only patients with a principal discharge diagnosis of AMI were included because the Choosing Wisely^® recommendation is specific with regard to troponin-only testing for the diagnosis of AMI. Patients with a principal diagnosis code for subcategories of myocardial ischemia (eg, stable angina, unstable angina) were not included because of the large number of diagnosis codes for these subcategories (more than 100 in the International Classification of Diseases, Ninth Revision and the International Classification of Diseases, Tenth Revision) and because the variation in their use across institutions within the dataset limited the utility of using these codes to consistently and accurately identify patients with myocardial ischemia. Moreover, the diagnosis of AMI encompasses the subcategories of myocardial ischemia.⁸

Hospital rates of ordering cardiac biomarkers (troponin-only or troponin and myoglobin/CK-MB) were determined overall for the entire study period and for each quarter of the study period based on the total patients with a discharge diagnosis of AMI. For each quarter of the 12 study quarters, all the hospitals were divided into tertiles based on their rate of troponin-only testing per discharge diagnosis of AMI. Hospitals were then classified into 3 groups based on their tertile ranking over the full 12 study quarters. The first group included hospitals whose rate of troponin-only testing placed them in the top tertile for each and all quarters throughout the study period. The second group included hospitals whose troponin-only testing rate placed them in the bottom tertile for each and all quarters throughout the study period. The third group included hospitals whose troponin-only testing rate each quarter led to either an increase or decrease in their tertile ranking throughout the study period. χ² tests were used to test for bivariate associations among hospitals based on their rate of troponin-only testing and hospital size (number of beds), their regional geographic location, the volume of AMI patients seen at the hospital, whether the primary physician during the hospitalization was a cardiologist or other provider, and the hospitals’ quality ratings. Quality rating was based on an internal Vizient rating and the “Best Hospitals for Cardiology and Heart Surgery Rankings” as published in the US News & World Report.⁹ The Vizient quality rating is based on a composite score that combines scores from the domains of quality (hospital quality incentive scores), safety (patient safety indicators), patient-centeredness (Hospital Consumer Assessment of Healthcare Providers and Systems Hospital Survey), and equity (distribution of care by race/ethnicity, gender, and age). Simple slopes were calculated to determine the rate of change in troponin-only testing for each study quarter, and Student t tests were used to compare the rates of change of these simple slopes across study quarters.

Pattern of Troponin-Only Testing by Hospital Size

Pattern of Troponin-Only Testing by Geographic Region

The rate of troponin-only testing also varied and was statistically significantly different when comparing the 3 groups of hospitals across geographic regions of the country (P < 0.0001). Of the hospitals in the top tertile of troponin-only testing throughout the study period, the majority were in the Midwest (n = 6) and Mid-Atlantic (n = 5) regions. However, the rate of troponin-only testing for AMI in this group was highest in hospitals in the West (86/100 patients) and/or Southeast (75/100 patients) regions, although this rate was based on a small number of hospitals in these geographic areas (n = 1 in the West, n = 2 in the Southeast). Of hospitals in the bottom tertile of troponin-only testing throughout the study period, the majority were in the Mid-Atlantic region (n = 10). Hospitals that increased their troponin-only testing during the study period were predominantly in the Midwest (n = 12) and Mid-Atlantic regions (n = 11; Table), with the hospitals in the Midwest having the highest rate of troponin-only testing in this group.

Pattern of Troponin-Only Testing by Volume of AMI Patients

Of the hospitals in the top tertile of troponin-only testing during the study period, the majority cared for ≥1500 AMI patients (n = 9), but interestingly, among these hospitals, those caring for a smaller volume of AMI patients all had higher rates of troponin-only testing per 100 patients (P < 0.0001; Table). There was no other obvious pattern of troponin-only testing based on the volume of AMI patients cared for in hospitals in either the bottom tertile of troponin-only testing or hospitals that improved troponin-only testing during the study period.

Pattern of Troponin-Only Testing by Physician Type

Of the hospitals in the top tertile of troponin-only testing throughout the study period, those where a cardiologist cared for patients with AMI had higher rates of troponin-only testing (71/100 patients) than did hospitals where patients were cared for by a noncardiologist (60/100 patients). However, of the hospitals that improved their troponin-only testing during the study period, higher rates of troponin-only testing were seen in hospitals where patients were cared for by a noncardiologist (48/100 patients) compared with patients cared for by a cardiologist (34/100 patients; Table). These differences in hospital rates of troponin-only testing during the study period based on physician type were statistically significant (P < 0.0001; Table).

Pattern of Troponin-Only Testing by Quality Rating

Hospitals that were in the top tertile of troponin-only testing and were rated highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report had higher rates of troponin-only testing per 100 patients than did hospitals in the top tertile that were not ranked highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report. However, the majority of hospitals in the top tertile of troponin-only testing were not rated highly by Vizient (n = 15) or recognized as a top hospital by the US News & World Report (n = 16). The large majority of hospitals in the bottom tertile of troponin-only testing were not recognized as high-quality hospitals by Vizient (n = 32) or the US News & World Report (n = 31). Of the hospitals that improved their troponin-only testing during the study period, the majority were not recognized as high-quality hospitals by Vizient (n = 33) or the US News & World Report (n = 36), but among this group, those hospitals recognized by Vizient as high quality (n = 5) had the highest rate of troponin-only testing (57/100 patients). The differences in the rate of troponin-only testing across the different groups of hospitals and quality ratings were statistically significant (P < 0.0001; Table).

The Effect of Choosing Wisely^® on Troponin-Only Testing

Disclosure 

No other individuals besides the authors contributed to this work. This project was not funded or supported by any external grant or agency. Dr. Prochaska’s institute received funding from the Agency for Research Healthcare and Quality for a K12 Career Development Grant (AHRQ K12 HS023007) outside the submitted work. Dr. Hohmann and Dr Modes have nothing to disclose. Dr. Arora receives financial compensation as a member of the Board of Directors for the American Board of Internal Medicine and has received grant funding from the ABIM Foundation. She also receives royalties from McGraw Hill.

Evidence suggests that troponin-only testing is the superior strategy to diagnose acute myocardial infarction (AMI).¹ Because of this, in February 2015, the Choosing Wisely^® campaign issued a recommendation to use troponin I or T to diagnose AMI, and not to test for myoglobin or creatine kinase-MB (CK-MB).² This recommendation was in line with guidelines from the American Heart Association and the American College of Cardiology, which recommended that myoglobin and CK-MB are not useful and offer no benefit for the diagnosis of acute coronary syndrome.³ Some institutions have developed interventions to promote troponin-only testing, reporting substantial cost savings and no negative consequences.^4,5

Despite these successes, it is likely that institutions vary with respect to the adoption of the Choosing Wisely^® troponin-only testing recommendation.⁶ Implementing this recommendation requires both promoting clinician behavior change and a strong institutional culture of high-value care.⁷ Understanding the variation across institutions of troponin-only testing could inform how to promote high-value care recommendations nationwide. We aimed to describe patterns of troponin, myoglobin, and CK-MB testing in a sample of academic teaching hospitals before and after the Choosing Wisely^® recommendation.

Troponin, myoglobin, and CK-MB ordering data were extracted from Vizient’s (formerly University HealthSystem Consortium, Chicago, IL) Clinical Database/Resource Manager (CDB/RM^®) for all patients with a principal discharge diagnosis of AMI at all hospitals reporting all 36 months from the fourth quarter of 2013 through the third quarter of 2016. This period includes time both before and after the Choosing Wisely^® recommendation, which was released in the first quarter of 2015. Vizient’s CDB/RM contains ordering data for 300 academic medical centers and their affiliated hospitals and includes the discharge diagnoses for patients cared for by these institutions. Only patients with a principal discharge diagnosis of AMI were included because the Choosing Wisely^® recommendation is specific with regard to troponin-only testing for the diagnosis of AMI. Patients with a principal diagnosis code for subcategories of myocardial ischemia (eg, stable angina, unstable angina) were not included because of the large number of diagnosis codes for these subcategories (more than 100 in the International Classification of Diseases, Ninth Revision and the International Classification of Diseases, Tenth Revision) and because the variation in their use across institutions within the dataset limited the utility of using these codes to consistently and accurately identify patients with myocardial ischemia. Moreover, the diagnosis of AMI encompasses the subcategories of myocardial ischemia.⁸

Hospital rates of ordering cardiac biomarkers (troponin-only or troponin and myoglobin/CK-MB) were determined overall for the entire study period and for each quarter of the study period based on the total patients with a discharge diagnosis of AMI. For each quarter of the 12 study quarters, all the hospitals were divided into tertiles based on their rate of troponin-only testing per discharge diagnosis of AMI. Hospitals were then classified into 3 groups based on their tertile ranking over the full 12 study quarters. The first group included hospitals whose rate of troponin-only testing placed them in the top tertile for each and all quarters throughout the study period. The second group included hospitals whose troponin-only testing rate placed them in the bottom tertile for each and all quarters throughout the study period. The third group included hospitals whose troponin-only testing rate each quarter led to either an increase or decrease in their tertile ranking throughout the study period. χ² tests were used to test for bivariate associations among hospitals based on their rate of troponin-only testing and hospital size (number of beds), their regional geographic location, the volume of AMI patients seen at the hospital, whether the primary physician during the hospitalization was a cardiologist or other provider, and the hospitals’ quality ratings. Quality rating was based on an internal Vizient rating and the “Best Hospitals for Cardiology and Heart Surgery Rankings” as published in the US News & World Report.⁹ The Vizient quality rating is based on a composite score that combines scores from the domains of quality (hospital quality incentive scores), safety (patient safety indicators), patient-centeredness (Hospital Consumer Assessment of Healthcare Providers and Systems Hospital Survey), and equity (distribution of care by race/ethnicity, gender, and age). Simple slopes were calculated to determine the rate of change in troponin-only testing for each study quarter, and Student t tests were used to compare the rates of change of these simple slopes across study quarters.

Pattern of Troponin-Only Testing by Hospital Size

Pattern of Troponin-Only Testing by Geographic Region

The rate of troponin-only testing also varied and was statistically significantly different when comparing the 3 groups of hospitals across geographic regions of the country (P < 0.0001). Of the hospitals in the top tertile of troponin-only testing throughout the study period, the majority were in the Midwest (n = 6) and Mid-Atlantic (n = 5) regions. However, the rate of troponin-only testing for AMI in this group was highest in hospitals in the West (86/100 patients) and/or Southeast (75/100 patients) regions, although this rate was based on a small number of hospitals in these geographic areas (n = 1 in the West, n = 2 in the Southeast). Of hospitals in the bottom tertile of troponin-only testing throughout the study period, the majority were in the Mid-Atlantic region (n = 10). Hospitals that increased their troponin-only testing during the study period were predominantly in the Midwest (n = 12) and Mid-Atlantic regions (n = 11; Table), with the hospitals in the Midwest having the highest rate of troponin-only testing in this group.

Pattern of Troponin-Only Testing by Volume of AMI Patients

Of the hospitals in the top tertile of troponin-only testing during the study period, the majority cared for ≥1500 AMI patients (n = 9), but interestingly, among these hospitals, those caring for a smaller volume of AMI patients all had higher rates of troponin-only testing per 100 patients (P < 0.0001; Table). There was no other obvious pattern of troponin-only testing based on the volume of AMI patients cared for in hospitals in either the bottom tertile of troponin-only testing or hospitals that improved troponin-only testing during the study period.

Pattern of Troponin-Only Testing by Physician Type

Of the hospitals in the top tertile of troponin-only testing throughout the study period, those where a cardiologist cared for patients with AMI had higher rates of troponin-only testing (71/100 patients) than did hospitals where patients were cared for by a noncardiologist (60/100 patients). However, of the hospitals that improved their troponin-only testing during the study period, higher rates of troponin-only testing were seen in hospitals where patients were cared for by a noncardiologist (48/100 patients) compared with patients cared for by a cardiologist (34/100 patients; Table). These differences in hospital rates of troponin-only testing during the study period based on physician type were statistically significant (P < 0.0001; Table).

Pattern of Troponin-Only Testing by Quality Rating

Hospitals that were in the top tertile of troponin-only testing and were rated highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report had higher rates of troponin-only testing per 100 patients than did hospitals in the top tertile that were not ranked highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report. However, the majority of hospitals in the top tertile of troponin-only testing were not rated highly by Vizient (n = 15) or recognized as a top hospital by the US News & World Report (n = 16). The large majority of hospitals in the bottom tertile of troponin-only testing were not recognized as high-quality hospitals by Vizient (n = 32) or the US News & World Report (n = 31). Of the hospitals that improved their troponin-only testing during the study period, the majority were not recognized as high-quality hospitals by Vizient (n = 33) or the US News & World Report (n = 36), but among this group, those hospitals recognized by Vizient as high quality (n = 5) had the highest rate of troponin-only testing (57/100 patients). The differences in the rate of troponin-only testing across the different groups of hospitals and quality ratings were statistically significant (P < 0.0001; Table).

The Effect of Choosing Wisely^® on Troponin-Only Testing

Disclosure 

No other individuals besides the authors contributed to this work. This project was not funded or supported by any external grant or agency. Dr. Prochaska’s institute received funding from the Agency for Research Healthcare and Quality for a K12 Career Development Grant (AHRQ K12 HS023007) outside the submitted work. Dr. Hohmann and Dr Modes have nothing to disclose. Dr. Arora receives financial compensation as a member of the Board of Directors for the American Board of Internal Medicine and has received grant funding from the ABIM Foundation. She also receives royalties from McGraw Hill.

Evidence suggests that troponin-only testing is the superior strategy to diagnose acute myocardial infarction (AMI).¹ Because of this, in February 2015, the Choosing Wisely^® campaign issued a recommendation to use troponin I or T to diagnose AMI, and not to test for myoglobin or creatine kinase-MB (CK-MB).² This recommendation was in line with guidelines from the American Heart Association and the American College of Cardiology, which recommended that myoglobin and CK-MB are not useful and offer no benefit for the diagnosis of acute coronary syndrome.³ Some institutions have developed interventions to promote troponin-only testing, reporting substantial cost savings and no negative consequences.^4,5

Despite these successes, it is likely that institutions vary with respect to the adoption of the Choosing Wisely^® troponin-only testing recommendation.⁶ Implementing this recommendation requires both promoting clinician behavior change and a strong institutional culture of high-value care.⁷ Understanding the variation across institutions of troponin-only testing could inform how to promote high-value care recommendations nationwide. We aimed to describe patterns of troponin, myoglobin, and CK-MB testing in a sample of academic teaching hospitals before and after the Choosing Wisely^® recommendation.

Troponin, myoglobin, and CK-MB ordering data were extracted from Vizient’s (formerly University HealthSystem Consortium, Chicago, IL) Clinical Database/Resource Manager (CDB/RM^®) for all patients with a principal discharge diagnosis of AMI at all hospitals reporting all 36 months from the fourth quarter of 2013 through the third quarter of 2016. This period includes time both before and after the Choosing Wisely^® recommendation, which was released in the first quarter of 2015. Vizient’s CDB/RM contains ordering data for 300 academic medical centers and their affiliated hospitals and includes the discharge diagnoses for patients cared for by these institutions. Only patients with a principal discharge diagnosis of AMI were included because the Choosing Wisely^® recommendation is specific with regard to troponin-only testing for the diagnosis of AMI. Patients with a principal diagnosis code for subcategories of myocardial ischemia (eg, stable angina, unstable angina) were not included because of the large number of diagnosis codes for these subcategories (more than 100 in the International Classification of Diseases, Ninth Revision and the International Classification of Diseases, Tenth Revision) and because the variation in their use across institutions within the dataset limited the utility of using these codes to consistently and accurately identify patients with myocardial ischemia. Moreover, the diagnosis of AMI encompasses the subcategories of myocardial ischemia.⁸

Hospital rates of ordering cardiac biomarkers (troponin-only or troponin and myoglobin/CK-MB) were determined overall for the entire study period and for each quarter of the study period based on the total patients with a discharge diagnosis of AMI. For each quarter of the 12 study quarters, all the hospitals were divided into tertiles based on their rate of troponin-only testing per discharge diagnosis of AMI. Hospitals were then classified into 3 groups based on their tertile ranking over the full 12 study quarters. The first group included hospitals whose rate of troponin-only testing placed them in the top tertile for each and all quarters throughout the study period. The second group included hospitals whose troponin-only testing rate placed them in the bottom tertile for each and all quarters throughout the study period. The third group included hospitals whose troponin-only testing rate each quarter led to either an increase or decrease in their tertile ranking throughout the study period. χ² tests were used to test for bivariate associations among hospitals based on their rate of troponin-only testing and hospital size (number of beds), their regional geographic location, the volume of AMI patients seen at the hospital, whether the primary physician during the hospitalization was a cardiologist or other provider, and the hospitals’ quality ratings. Quality rating was based on an internal Vizient rating and the “Best Hospitals for Cardiology and Heart Surgery Rankings” as published in the US News & World Report.⁹ The Vizient quality rating is based on a composite score that combines scores from the domains of quality (hospital quality incentive scores), safety (patient safety indicators), patient-centeredness (Hospital Consumer Assessment of Healthcare Providers and Systems Hospital Survey), and equity (distribution of care by race/ethnicity, gender, and age). Simple slopes were calculated to determine the rate of change in troponin-only testing for each study quarter, and Student t tests were used to compare the rates of change of these simple slopes across study quarters.

Pattern of Troponin-Only Testing by Hospital Size

Pattern of Troponin-Only Testing by Geographic Region

The rate of troponin-only testing also varied and was statistically significantly different when comparing the 3 groups of hospitals across geographic regions of the country (P < 0.0001). Of the hospitals in the top tertile of troponin-only testing throughout the study period, the majority were in the Midwest (n = 6) and Mid-Atlantic (n = 5) regions. However, the rate of troponin-only testing for AMI in this group was highest in hospitals in the West (86/100 patients) and/or Southeast (75/100 patients) regions, although this rate was based on a small number of hospitals in these geographic areas (n = 1 in the West, n = 2 in the Southeast). Of hospitals in the bottom tertile of troponin-only testing throughout the study period, the majority were in the Mid-Atlantic region (n = 10). Hospitals that increased their troponin-only testing during the study period were predominantly in the Midwest (n = 12) and Mid-Atlantic regions (n = 11; Table), with the hospitals in the Midwest having the highest rate of troponin-only testing in this group.

Pattern of Troponin-Only Testing by Volume of AMI Patients

Of the hospitals in the top tertile of troponin-only testing during the study period, the majority cared for ≥1500 AMI patients (n = 9), but interestingly, among these hospitals, those caring for a smaller volume of AMI patients all had higher rates of troponin-only testing per 100 patients (P < 0.0001; Table). There was no other obvious pattern of troponin-only testing based on the volume of AMI patients cared for in hospitals in either the bottom tertile of troponin-only testing or hospitals that improved troponin-only testing during the study period.

Pattern of Troponin-Only Testing by Physician Type

Of the hospitals in the top tertile of troponin-only testing throughout the study period, those where a cardiologist cared for patients with AMI had higher rates of troponin-only testing (71/100 patients) than did hospitals where patients were cared for by a noncardiologist (60/100 patients). However, of the hospitals that improved their troponin-only testing during the study period, higher rates of troponin-only testing were seen in hospitals where patients were cared for by a noncardiologist (48/100 patients) compared with patients cared for by a cardiologist (34/100 patients; Table). These differences in hospital rates of troponin-only testing during the study period based on physician type were statistically significant (P < 0.0001; Table).

Pattern of Troponin-Only Testing by Quality Rating

Hospitals that were in the top tertile of troponin-only testing and were rated highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report had higher rates of troponin-only testing per 100 patients than did hospitals in the top tertile that were not ranked highly by Vizient’s quality rating or recognized as a top hospital by the US News & World Report. However, the majority of hospitals in the top tertile of troponin-only testing were not rated highly by Vizient (n = 15) or recognized as a top hospital by the US News & World Report (n = 16). The large majority of hospitals in the bottom tertile of troponin-only testing were not recognized as high-quality hospitals by Vizient (n = 32) or the US News & World Report (n = 31). Of the hospitals that improved their troponin-only testing during the study period, the majority were not recognized as high-quality hospitals by Vizient (n = 33) or the US News & World Report (n = 36), but among this group, those hospitals recognized by Vizient as high quality (n = 5) had the highest rate of troponin-only testing (57/100 patients). The differences in the rate of troponin-only testing across the different groups of hospitals and quality ratings were statistically significant (P < 0.0001; Table).

The Effect of Choosing Wisely^® on Troponin-Only Testing

Disclosure 

No other individuals besides the authors contributed to this work. This project was not funded or supported by any external grant or agency. Dr. Prochaska’s institute received funding from the Agency for Research Healthcare and Quality for a K12 Career Development Grant (AHRQ K12 HS023007) outside the submitted work. Dr. Hohmann and Dr Modes have nothing to disclose. Dr. Arora receives financial compensation as a member of the Board of Directors for the American Board of Internal Medicine and has received grant funding from the ABIM Foundation. She also receives royalties from McGraw Hill.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.