Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Prior studies have found that up to 70% of acid‐suppressive medication (ASM) use in the hospital is not indicated, most commonly for stress ulcer prophylaxis in patients outside of the intensive care unit (ICU).[1, 2, 3, 4, 5, 6, 7] Accordingly, reducing inappropriate use of ASM for stress ulcer prophylaxis in hospitalized patients is 1 of the 5 opportunities for improved healthcare value identified by the Society of Hospital Medicine as part of the American Board of Internal Medicine's Choosing Wisely campaign.[8]

We designed and tested a computerized clinical decision support (CDS) intervention with the goal of reducing use of ASM for stress ulcer prophylaxis in hospitalized patients outside the ICU at an academic medical center.

METHODS

Intervention

Our study was implemented in 2 phases (Figure 1).

Figure 1

Baseline Phase

The purpose of the first phase was to obtain baseline data on ASM use prior to implementing our CDS tool designed to influence prescribing. During this baseline phase, a computerized prompt was activated through our POE system whenever a clinician initiated an order for ASM (histamine 2 receptor antagonists or proton pump inhibitors), asking the clinician to select the reason/reasons for the order based on the following predefined response options: (1) active/recent upper gastrointestinal bleed, (2) continuing preadmission medication, (3) Helicobacter pylori treatment, (4) prophylaxis in patient on medications that increase bleeding risk, (5) stress ulcer prophylaxis, (6) suspected/known peptic ulcer disease, gastritis, esophagitis, gastroesophageal reflux disease, and (7) other, with a free‐text box to input the indication. This indications prompt was rolled out to the entire medical center on September 12, 2011 and remained active for the duration of the study period.

Intervention Phase

In the second phase of the study, if a clinician selected stress ulcer prophylaxis as the only indication for ordering ASM, a CDS prompt alerted the clinician that Stress ulcer prophylaxis is not recommended for patients outside of the intensive care unit (ASHP Therapeutic Guidelines on Stress Ulcer Prophylaxis. Am J Health‐Syst Pharm. 1999, 56:347‐79). The clinician could then select either, For use in ICUOrder Medication, Choose Other Indication, or Cancel Order. This CDS prompt was rolled out in a staggered manner to the East Campus on January 3, 2012, followed by the West Campus on April 3, 2012.

RESULTS

There were 26,400 adult admissions during the study period, and 22,330 discrete orders for ASM. Overall, 12,056 (46%) admissions had at least 1 charge for ASM. Admission characteristics were similar before and after the intervention on each campus (Table 1).

Table 2 shows the indications chosen each time ASM was ordered, stratified by campus and study phase. Although selection of stress ulcer prophylaxis decreased on both campuses during the intervention phase, selection of continuing preadmission medication increased.

Table 3 shows the unadjusted comparison of outcomes between baseline and intervention phases on each campus. Use of ASM with stress ulcer prophylaxis as the only indication decreased during the intervention phase on both campuses. There was a nonsignificant reduction in overall rates of use on both campuses, and use on discharge was unchanged. Figure 2 demonstrates the unadjusted and modeled monthly rates of admissions with at least 1 ASM order with stress ulcer prophylaxis selected as the only indication, stratified by campus. After adjusting for the admission characteristics in Table 1, during the intervention phase on both campuses there was a significant immediate reduction in the odds of receiving an ASM with stress ulcer prophylaxis selected as the only indication (East Campus odds ratio [OR]: 0.36, 95% confidence interval [CI]: 0.180.71; West Campus OR: 0.41, 95% CI: 0.280.60), and a significant change in trend compared to the baseline phase (East Campus 1.5% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.002; West Campus 0.9% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.02).

Figure 2

DISCUSSION

In this single‐center study, we found that a computerized CDS intervention resulted in a significant reduction in use of ASM for the sole purpose of stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. We found low rates of use for the isolated purpose of stress ulcer prophylaxis even before the intervention, and continuing preadmission medication was the most commonly selected indication throughout the study.

Although overall rates of ASM use declined after the intervention, the change was not statistically significant, and was not of the same magnitude as the decline in rates of use for the purpose of stress ulcer prophylaxis. This suggests that our intervention, in part, led to substitution of 1 indication for another. The indication that increased the most after rollout on both campuses was continuing preadmission medication. There are at least 2 possibilities for this finding: (1) the intervention prompted physicians to more accurately record the indication, or (2) physicians falsified the indication in order to execute the order. To explore these possibilities, we reviewed the charts of a random sample of 100 admissions during each of the baseline and intervention phases where continuing preadmission medication was selected as an indication for an ASM order. We found that 6/100 orders in the baseline phase and 7/100 orders in the intervention phase incorrectly indicated that the patient was on ASM prior to admission (P=0.77). This suggests that scenario 1 above is the more likely explanation for the increased use of this indication, and that the intervention, in part, simply unmasked the true rate of use at our medical center for the isolated purpose of stress ulcer prophylaxis.

These findings have implications for others attempting to use computerized CDS to better understand physician prescribing. They suggest that information collected through computer‐based interaction with clinicians at the point of care may not always be accurate or complete. As institutions increasingly use similar interventions to drive behavior, information obtained from such interaction should be validated, and when possible, patient outcomes should be measured.

Our findings suggest that rates of ASM use for the purpose of stress ulcer prophylaxis in the hospital may have declined over the last decade. Studies demonstrating that up to 70% of inpatient use of ASM was inappropriate were conducted 5 to 10 years ago.[1, 2, 3, 4, 5] Since then, studies have demonstrated risk of nosocomial infections in patients on ASM.[9, 10, 11] It is possible that the low rate of use for stress ulcer prophylaxis in our study is attributable to awareness of the risks of these medications, and limited our ability to detect differences in overall use. It is also possible, however, that a portion of the admissions with continuation of preadmission medication as the indication were started on these medications during a prior hospitalization. Thus, some portion of preadmission use is likely to represent failed medication reconciliation during a prior discharge. In this context, hospitalization may serve as an opportunity to evaluate the indication for ASM use even when these medications show up as preadmission medications.

There are additional limitations. First, the single‐center nature limits generalizability. Second, the first phase of our study, designed to obtain baseline data on ASM use, may have led to changes in prescribing prior to implementation of our CDS tool. Additionally, we did not validate the accuracy of each of the chosen indications, or the site of initial prescription in the case of preadmission exposure. Last, our study was not powered to investigate changes in rates of nosocomial gastrointestinal bleeding or nosocomial pneumonia owing to the infrequent nature of these complications.

In conclusion, we designed a simple computerized CDS intervention that was associated with a reduction in ASM use for stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. The majority of inpatient use represented continuation of preadmission medication, suggesting that interventions to improve the appropriateness of ASM prescribing should span the continuum of care. Future studies should investigate whether it is worthwhile and appropriate to reevaluate continued use of preadmission ASM during an inpatient stay.

Prior studies have found that up to 70% of acid‐suppressive medication (ASM) use in the hospital is not indicated, most commonly for stress ulcer prophylaxis in patients outside of the intensive care unit (ICU).[1, 2, 3, 4, 5, 6, 7] Accordingly, reducing inappropriate use of ASM for stress ulcer prophylaxis in hospitalized patients is 1 of the 5 opportunities for improved healthcare value identified by the Society of Hospital Medicine as part of the American Board of Internal Medicine's Choosing Wisely campaign.[8]

We designed and tested a computerized clinical decision support (CDS) intervention with the goal of reducing use of ASM for stress ulcer prophylaxis in hospitalized patients outside the ICU at an academic medical center.

METHODS

Intervention

Our study was implemented in 2 phases (Figure 1).

Figure 1

Baseline Phase

The purpose of the first phase was to obtain baseline data on ASM use prior to implementing our CDS tool designed to influence prescribing. During this baseline phase, a computerized prompt was activated through our POE system whenever a clinician initiated an order for ASM (histamine 2 receptor antagonists or proton pump inhibitors), asking the clinician to select the reason/reasons for the order based on the following predefined response options: (1) active/recent upper gastrointestinal bleed, (2) continuing preadmission medication, (3) Helicobacter pylori treatment, (4) prophylaxis in patient on medications that increase bleeding risk, (5) stress ulcer prophylaxis, (6) suspected/known peptic ulcer disease, gastritis, esophagitis, gastroesophageal reflux disease, and (7) other, with a free‐text box to input the indication. This indications prompt was rolled out to the entire medical center on September 12, 2011 and remained active for the duration of the study period.

Intervention Phase

In the second phase of the study, if a clinician selected stress ulcer prophylaxis as the only indication for ordering ASM, a CDS prompt alerted the clinician that Stress ulcer prophylaxis is not recommended for patients outside of the intensive care unit (ASHP Therapeutic Guidelines on Stress Ulcer Prophylaxis. Am J Health‐Syst Pharm. 1999, 56:347‐79). The clinician could then select either, For use in ICUOrder Medication, Choose Other Indication, or Cancel Order. This CDS prompt was rolled out in a staggered manner to the East Campus on January 3, 2012, followed by the West Campus on April 3, 2012.

RESULTS

There were 26,400 adult admissions during the study period, and 22,330 discrete orders for ASM. Overall, 12,056 (46%) admissions had at least 1 charge for ASM. Admission characteristics were similar before and after the intervention on each campus (Table 1).

Table 2 shows the indications chosen each time ASM was ordered, stratified by campus and study phase. Although selection of stress ulcer prophylaxis decreased on both campuses during the intervention phase, selection of continuing preadmission medication increased.

Table 3 shows the unadjusted comparison of outcomes between baseline and intervention phases on each campus. Use of ASM with stress ulcer prophylaxis as the only indication decreased during the intervention phase on both campuses. There was a nonsignificant reduction in overall rates of use on both campuses, and use on discharge was unchanged. Figure 2 demonstrates the unadjusted and modeled monthly rates of admissions with at least 1 ASM order with stress ulcer prophylaxis selected as the only indication, stratified by campus. After adjusting for the admission characteristics in Table 1, during the intervention phase on both campuses there was a significant immediate reduction in the odds of receiving an ASM with stress ulcer prophylaxis selected as the only indication (East Campus odds ratio [OR]: 0.36, 95% confidence interval [CI]: 0.180.71; West Campus OR: 0.41, 95% CI: 0.280.60), and a significant change in trend compared to the baseline phase (East Campus 1.5% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.002; West Campus 0.9% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.02).

Figure 2

DISCUSSION

In this single‐center study, we found that a computerized CDS intervention resulted in a significant reduction in use of ASM for the sole purpose of stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. We found low rates of use for the isolated purpose of stress ulcer prophylaxis even before the intervention, and continuing preadmission medication was the most commonly selected indication throughout the study.

Although overall rates of ASM use declined after the intervention, the change was not statistically significant, and was not of the same magnitude as the decline in rates of use for the purpose of stress ulcer prophylaxis. This suggests that our intervention, in part, led to substitution of 1 indication for another. The indication that increased the most after rollout on both campuses was continuing preadmission medication. There are at least 2 possibilities for this finding: (1) the intervention prompted physicians to more accurately record the indication, or (2) physicians falsified the indication in order to execute the order. To explore these possibilities, we reviewed the charts of a random sample of 100 admissions during each of the baseline and intervention phases where continuing preadmission medication was selected as an indication for an ASM order. We found that 6/100 orders in the baseline phase and 7/100 orders in the intervention phase incorrectly indicated that the patient was on ASM prior to admission (P=0.77). This suggests that scenario 1 above is the more likely explanation for the increased use of this indication, and that the intervention, in part, simply unmasked the true rate of use at our medical center for the isolated purpose of stress ulcer prophylaxis.

These findings have implications for others attempting to use computerized CDS to better understand physician prescribing. They suggest that information collected through computer‐based interaction with clinicians at the point of care may not always be accurate or complete. As institutions increasingly use similar interventions to drive behavior, information obtained from such interaction should be validated, and when possible, patient outcomes should be measured.

Our findings suggest that rates of ASM use for the purpose of stress ulcer prophylaxis in the hospital may have declined over the last decade. Studies demonstrating that up to 70% of inpatient use of ASM was inappropriate were conducted 5 to 10 years ago.[1, 2, 3, 4, 5] Since then, studies have demonstrated risk of nosocomial infections in patients on ASM.[9, 10, 11] It is possible that the low rate of use for stress ulcer prophylaxis in our study is attributable to awareness of the risks of these medications, and limited our ability to detect differences in overall use. It is also possible, however, that a portion of the admissions with continuation of preadmission medication as the indication were started on these medications during a prior hospitalization. Thus, some portion of preadmission use is likely to represent failed medication reconciliation during a prior discharge. In this context, hospitalization may serve as an opportunity to evaluate the indication for ASM use even when these medications show up as preadmission medications.

There are additional limitations. First, the single‐center nature limits generalizability. Second, the first phase of our study, designed to obtain baseline data on ASM use, may have led to changes in prescribing prior to implementation of our CDS tool. Additionally, we did not validate the accuracy of each of the chosen indications, or the site of initial prescription in the case of preadmission exposure. Last, our study was not powered to investigate changes in rates of nosocomial gastrointestinal bleeding or nosocomial pneumonia owing to the infrequent nature of these complications.

In conclusion, we designed a simple computerized CDS intervention that was associated with a reduction in ASM use for stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. The majority of inpatient use represented continuation of preadmission medication, suggesting that interventions to improve the appropriateness of ASM prescribing should span the continuum of care. Future studies should investigate whether it is worthwhile and appropriate to reevaluate continued use of preadmission ASM during an inpatient stay.

Prior studies have found that up to 70% of acid‐suppressive medication (ASM) use in the hospital is not indicated, most commonly for stress ulcer prophylaxis in patients outside of the intensive care unit (ICU).[1, 2, 3, 4, 5, 6, 7] Accordingly, reducing inappropriate use of ASM for stress ulcer prophylaxis in hospitalized patients is 1 of the 5 opportunities for improved healthcare value identified by the Society of Hospital Medicine as part of the American Board of Internal Medicine's Choosing Wisely campaign.[8]

We designed and tested a computerized clinical decision support (CDS) intervention with the goal of reducing use of ASM for stress ulcer prophylaxis in hospitalized patients outside the ICU at an academic medical center.

METHODS

Intervention

Our study was implemented in 2 phases (Figure 1).

Figure 1

Baseline Phase

The purpose of the first phase was to obtain baseline data on ASM use prior to implementing our CDS tool designed to influence prescribing. During this baseline phase, a computerized prompt was activated through our POE system whenever a clinician initiated an order for ASM (histamine 2 receptor antagonists or proton pump inhibitors), asking the clinician to select the reason/reasons for the order based on the following predefined response options: (1) active/recent upper gastrointestinal bleed, (2) continuing preadmission medication, (3) Helicobacter pylori treatment, (4) prophylaxis in patient on medications that increase bleeding risk, (5) stress ulcer prophylaxis, (6) suspected/known peptic ulcer disease, gastritis, esophagitis, gastroesophageal reflux disease, and (7) other, with a free‐text box to input the indication. This indications prompt was rolled out to the entire medical center on September 12, 2011 and remained active for the duration of the study period.

Intervention Phase

In the second phase of the study, if a clinician selected stress ulcer prophylaxis as the only indication for ordering ASM, a CDS prompt alerted the clinician that Stress ulcer prophylaxis is not recommended for patients outside of the intensive care unit (ASHP Therapeutic Guidelines on Stress Ulcer Prophylaxis. Am J Health‐Syst Pharm. 1999, 56:347‐79). The clinician could then select either, For use in ICUOrder Medication, Choose Other Indication, or Cancel Order. This CDS prompt was rolled out in a staggered manner to the East Campus on January 3, 2012, followed by the West Campus on April 3, 2012.

RESULTS

There were 26,400 adult admissions during the study period, and 22,330 discrete orders for ASM. Overall, 12,056 (46%) admissions had at least 1 charge for ASM. Admission characteristics were similar before and after the intervention on each campus (Table 1).

Table 2 shows the indications chosen each time ASM was ordered, stratified by campus and study phase. Although selection of stress ulcer prophylaxis decreased on both campuses during the intervention phase, selection of continuing preadmission medication increased.

Table 3 shows the unadjusted comparison of outcomes between baseline and intervention phases on each campus. Use of ASM with stress ulcer prophylaxis as the only indication decreased during the intervention phase on both campuses. There was a nonsignificant reduction in overall rates of use on both campuses, and use on discharge was unchanged. Figure 2 demonstrates the unadjusted and modeled monthly rates of admissions with at least 1 ASM order with stress ulcer prophylaxis selected as the only indication, stratified by campus. After adjusting for the admission characteristics in Table 1, during the intervention phase on both campuses there was a significant immediate reduction in the odds of receiving an ASM with stress ulcer prophylaxis selected as the only indication (East Campus odds ratio [OR]: 0.36, 95% confidence interval [CI]: 0.180.71; West Campus OR: 0.41, 95% CI: 0.280.60), and a significant change in trend compared to the baseline phase (East Campus 1.5% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.002; West Campus 0.9% daily decrease in odds of receiving ASM solely for stress ulcer prophylaxis, P=0.02).

Figure 2

DISCUSSION

In this single‐center study, we found that a computerized CDS intervention resulted in a significant reduction in use of ASM for the sole purpose of stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. We found low rates of use for the isolated purpose of stress ulcer prophylaxis even before the intervention, and continuing preadmission medication was the most commonly selected indication throughout the study.

Although overall rates of ASM use declined after the intervention, the change was not statistically significant, and was not of the same magnitude as the decline in rates of use for the purpose of stress ulcer prophylaxis. This suggests that our intervention, in part, led to substitution of 1 indication for another. The indication that increased the most after rollout on both campuses was continuing preadmission medication. There are at least 2 possibilities for this finding: (1) the intervention prompted physicians to more accurately record the indication, or (2) physicians falsified the indication in order to execute the order. To explore these possibilities, we reviewed the charts of a random sample of 100 admissions during each of the baseline and intervention phases where continuing preadmission medication was selected as an indication for an ASM order. We found that 6/100 orders in the baseline phase and 7/100 orders in the intervention phase incorrectly indicated that the patient was on ASM prior to admission (P=0.77). This suggests that scenario 1 above is the more likely explanation for the increased use of this indication, and that the intervention, in part, simply unmasked the true rate of use at our medical center for the isolated purpose of stress ulcer prophylaxis.

These findings have implications for others attempting to use computerized CDS to better understand physician prescribing. They suggest that information collected through computer‐based interaction with clinicians at the point of care may not always be accurate or complete. As institutions increasingly use similar interventions to drive behavior, information obtained from such interaction should be validated, and when possible, patient outcomes should be measured.

Our findings suggest that rates of ASM use for the purpose of stress ulcer prophylaxis in the hospital may have declined over the last decade. Studies demonstrating that up to 70% of inpatient use of ASM was inappropriate were conducted 5 to 10 years ago.[1, 2, 3, 4, 5] Since then, studies have demonstrated risk of nosocomial infections in patients on ASM.[9, 10, 11] It is possible that the low rate of use for stress ulcer prophylaxis in our study is attributable to awareness of the risks of these medications, and limited our ability to detect differences in overall use. It is also possible, however, that a portion of the admissions with continuation of preadmission medication as the indication were started on these medications during a prior hospitalization. Thus, some portion of preadmission use is likely to represent failed medication reconciliation during a prior discharge. In this context, hospitalization may serve as an opportunity to evaluate the indication for ASM use even when these medications show up as preadmission medications.

There are additional limitations. First, the single‐center nature limits generalizability. Second, the first phase of our study, designed to obtain baseline data on ASM use, may have led to changes in prescribing prior to implementation of our CDS tool. Additionally, we did not validate the accuracy of each of the chosen indications, or the site of initial prescription in the case of preadmission exposure. Last, our study was not powered to investigate changes in rates of nosocomial gastrointestinal bleeding or nosocomial pneumonia owing to the infrequent nature of these complications.

In conclusion, we designed a simple computerized CDS intervention that was associated with a reduction in ASM use for stress ulcer prophylaxis in patients outside the ICU, a nonsignificant reduction in overall use, and no change in use on discharge. The majority of inpatient use represented continuation of preadmission medication, suggesting that interventions to improve the appropriateness of ASM prescribing should span the continuum of care. Future studies should investigate whether it is worthwhile and appropriate to reevaluate continued use of preadmission ASM during an inpatient stay.

In this issue of the Journal of Hospital Medicine, Butcher and colleagues report on residents' perceptions of a rapid response team's (RRT) impact on their training.[1] RRTs mobilize key clinicians in an attempt to rescue acutely decompensating hospitalized patients. Early recognition is essential, and most systems allow any concerned health professional to activate the RRT. Although the evidence for benefit is somewhat controversial,[2, 3] an overwhelming majority of hospitals have implemented RRTs.[4, 5]

The use of RRTs in teaching hospitals raises important concerns. The ability of nurses and other professionals to activate the RRT without need for prior approval from a physician could potentially undermine resident physician autonomy. Residents may feel that their clinical judgment has been usurped or second guessed. Whether nurse led or physician led, RRTs always introduce new members to the care team.[6] These new team members share in decision making, which may theoretically reduce residents' opportunities to hone their decision‐making skills when caring for potentially critically ill patients.

Despite these potential disadvantages, Butcher and colleagues report that the vast majority of residents found working with the RRT to be a valuable educational experience and disagreed that the RRT decreased their clinical autonomy. Interestingly, surgical residents were less likely to agree that working with the RRT was a valuable educational experience and much more likely to feel that nurses should contact them before activating the RRT.

The results of the study by Butcher et al. highlight several evolving paradigms in medical education and quality improvement. Over the past 10 to 15 years, and fostered in large part by Accreditation Council for Graduate Medical Education (ACGME) duty‐hour revisions,[7] teaching hospitals have moved away from the traditional practice of using residents primarily to fill their clinical service needs to an approach that treats residents more as learners. Resident training requires clinical care, but the provision of clinical care in teaching hospitals does not necessarily require residents. At the same time, healthcare organizations have moved away from the traditional culture characterized by reliance on individual skill, physician autonomy, and steep hierarchies, to an enlightened culture emphasizing teamwork with flattened hierarchies and systems redesigned to provide safe and effective care.[8]

For the most part, the paradigm shifts in medical education and quality improvement have been aligned. In fact, the primary goal of duty‐hour policy revisions was to improve patient safety.[9] Yet, Butcher and colleagues' study highlights the need to continuously and deliberately integrate our efforts to enhance medical education and quality of care, and more rigorously study the effects. Rather than be pleasantly surprised that residents understand the intrinsic value of an RRT to patient care and their education, we should ensure that residents understand the rationale for an RRT and consider using the RRT to complement other efforts to educate resident physicians in managing unstable patients. RRTs introduce a wonderful opportunity to develop novel interprofessional curricula. Learning objectives should include the management of common clinical syndromes represented in RRT calls, but should also focus on communication, leadership, and other essential teamwork skills. Simulation‐based training is an ideal teaching strategy for these objectives, and prior studies support the effectiveness of this approach.[10, 11]

The ACGME has now implemented the Next Accreditation System (NAS) across all specialties. Of the 22 reporting milestones within internal medicine, 12 relate directly to quality improvement and patient safety objectives, whereas 6 relate directly to pathophysiology and disease management.[12] Educating residents on systems of care is further highlighted by the Clinical Learning Environment Review (CLER), a key component of the NAS. The CLER program uses site visits to identify teaching hospitals' efforts to engage residents in 6 focus areas: patient safety; healthcare quality; transitions of care; supervision; duty hours, fatigue management, and mitigation; and professionalism.[13] CLER site visits include discussions and observations with hospital executive leadership, residents, graduate medical education leadership, nursing, and other hospital staff. The CLER program raises the bar for integrating medical education and quality improvement efforts even further. Quality improvement activities that previously supported an informal curriculum must now be made explicit to, and deliberately engage, our residents. Teaching hospitals are being tasked with including residents in safety initiatives and on all quality committees, especially those with cross‐departmental boundaries such as the Emergency Response Team/RRT Committee. Residents should meaningfully participate, and whenever possible, lead quality improvement projects, the focus of which may ideally be identified by residents themselves. An important resource for medical educators is the Quality and Safety Educators Academy, a program developed by the Society of Hospital Medicine and the Alliance for Academic Internal Medicine, which provides educators with the knowledge and tools to integrate quality improvement and patient safety objectives into their training programs.[14]

In conclusion, we are reassured that residents understand the intrinsic value of an RRT to patient care and their education. We encourage medical educators to use RRTs as an opportunity to develop interprofessional curricula, including those that aim to enhance teamwork skills. Beyond curricular innovation, quality‐improvement activities in teaching hospitals must deliberately engage our residents at every level of the organization.

In this issue of the Journal of Hospital Medicine, Butcher and colleagues report on residents' perceptions of a rapid response team's (RRT) impact on their training.[1] RRTs mobilize key clinicians in an attempt to rescue acutely decompensating hospitalized patients. Early recognition is essential, and most systems allow any concerned health professional to activate the RRT. Although the evidence for benefit is somewhat controversial,[2, 3] an overwhelming majority of hospitals have implemented RRTs.[4, 5]

The use of RRTs in teaching hospitals raises important concerns. The ability of nurses and other professionals to activate the RRT without need for prior approval from a physician could potentially undermine resident physician autonomy. Residents may feel that their clinical judgment has been usurped or second guessed. Whether nurse led or physician led, RRTs always introduce new members to the care team.[6] These new team members share in decision making, which may theoretically reduce residents' opportunities to hone their decision‐making skills when caring for potentially critically ill patients.

Despite these potential disadvantages, Butcher and colleagues report that the vast majority of residents found working with the RRT to be a valuable educational experience and disagreed that the RRT decreased their clinical autonomy. Interestingly, surgical residents were less likely to agree that working with the RRT was a valuable educational experience and much more likely to feel that nurses should contact them before activating the RRT.

The results of the study by Butcher et al. highlight several evolving paradigms in medical education and quality improvement. Over the past 10 to 15 years, and fostered in large part by Accreditation Council for Graduate Medical Education (ACGME) duty‐hour revisions,[7] teaching hospitals have moved away from the traditional practice of using residents primarily to fill their clinical service needs to an approach that treats residents more as learners. Resident training requires clinical care, but the provision of clinical care in teaching hospitals does not necessarily require residents. At the same time, healthcare organizations have moved away from the traditional culture characterized by reliance on individual skill, physician autonomy, and steep hierarchies, to an enlightened culture emphasizing teamwork with flattened hierarchies and systems redesigned to provide safe and effective care.[8]

For the most part, the paradigm shifts in medical education and quality improvement have been aligned. In fact, the primary goal of duty‐hour policy revisions was to improve patient safety.[9] Yet, Butcher and colleagues' study highlights the need to continuously and deliberately integrate our efforts to enhance medical education and quality of care, and more rigorously study the effects. Rather than be pleasantly surprised that residents understand the intrinsic value of an RRT to patient care and their education, we should ensure that residents understand the rationale for an RRT and consider using the RRT to complement other efforts to educate resident physicians in managing unstable patients. RRTs introduce a wonderful opportunity to develop novel interprofessional curricula. Learning objectives should include the management of common clinical syndromes represented in RRT calls, but should also focus on communication, leadership, and other essential teamwork skills. Simulation‐based training is an ideal teaching strategy for these objectives, and prior studies support the effectiveness of this approach.[10, 11]

The ACGME has now implemented the Next Accreditation System (NAS) across all specialties. Of the 22 reporting milestones within internal medicine, 12 relate directly to quality improvement and patient safety objectives, whereas 6 relate directly to pathophysiology and disease management.[12] Educating residents on systems of care is further highlighted by the Clinical Learning Environment Review (CLER), a key component of the NAS. The CLER program uses site visits to identify teaching hospitals' efforts to engage residents in 6 focus areas: patient safety; healthcare quality; transitions of care; supervision; duty hours, fatigue management, and mitigation; and professionalism.[13] CLER site visits include discussions and observations with hospital executive leadership, residents, graduate medical education leadership, nursing, and other hospital staff. The CLER program raises the bar for integrating medical education and quality improvement efforts even further. Quality improvement activities that previously supported an informal curriculum must now be made explicit to, and deliberately engage, our residents. Teaching hospitals are being tasked with including residents in safety initiatives and on all quality committees, especially those with cross‐departmental boundaries such as the Emergency Response Team/RRT Committee. Residents should meaningfully participate, and whenever possible, lead quality improvement projects, the focus of which may ideally be identified by residents themselves. An important resource for medical educators is the Quality and Safety Educators Academy, a program developed by the Society of Hospital Medicine and the Alliance for Academic Internal Medicine, which provides educators with the knowledge and tools to integrate quality improvement and patient safety objectives into their training programs.[14]

In conclusion, we are reassured that residents understand the intrinsic value of an RRT to patient care and their education. We encourage medical educators to use RRTs as an opportunity to develop interprofessional curricula, including those that aim to enhance teamwork skills. Beyond curricular innovation, quality‐improvement activities in teaching hospitals must deliberately engage our residents at every level of the organization.

In this issue of the Journal of Hospital Medicine, Butcher and colleagues report on residents' perceptions of a rapid response team's (RRT) impact on their training.[1] RRTs mobilize key clinicians in an attempt to rescue acutely decompensating hospitalized patients. Early recognition is essential, and most systems allow any concerned health professional to activate the RRT. Although the evidence for benefit is somewhat controversial,[2, 3] an overwhelming majority of hospitals have implemented RRTs.[4, 5]

The use of RRTs in teaching hospitals raises important concerns. The ability of nurses and other professionals to activate the RRT without need for prior approval from a physician could potentially undermine resident physician autonomy. Residents may feel that their clinical judgment has been usurped or second guessed. Whether nurse led or physician led, RRTs always introduce new members to the care team.[6] These new team members share in decision making, which may theoretically reduce residents' opportunities to hone their decision‐making skills when caring for potentially critically ill patients.

Despite these potential disadvantages, Butcher and colleagues report that the vast majority of residents found working with the RRT to be a valuable educational experience and disagreed that the RRT decreased their clinical autonomy. Interestingly, surgical residents were less likely to agree that working with the RRT was a valuable educational experience and much more likely to feel that nurses should contact them before activating the RRT.

The results of the study by Butcher et al. highlight several evolving paradigms in medical education and quality improvement. Over the past 10 to 15 years, and fostered in large part by Accreditation Council for Graduate Medical Education (ACGME) duty‐hour revisions,[7] teaching hospitals have moved away from the traditional practice of using residents primarily to fill their clinical service needs to an approach that treats residents more as learners. Resident training requires clinical care, but the provision of clinical care in teaching hospitals does not necessarily require residents. At the same time, healthcare organizations have moved away from the traditional culture characterized by reliance on individual skill, physician autonomy, and steep hierarchies, to an enlightened culture emphasizing teamwork with flattened hierarchies and systems redesigned to provide safe and effective care.[8]

For the most part, the paradigm shifts in medical education and quality improvement have been aligned. In fact, the primary goal of duty‐hour policy revisions was to improve patient safety.[9] Yet, Butcher and colleagues' study highlights the need to continuously and deliberately integrate our efforts to enhance medical education and quality of care, and more rigorously study the effects. Rather than be pleasantly surprised that residents understand the intrinsic value of an RRT to patient care and their education, we should ensure that residents understand the rationale for an RRT and consider using the RRT to complement other efforts to educate resident physicians in managing unstable patients. RRTs introduce a wonderful opportunity to develop novel interprofessional curricula. Learning objectives should include the management of common clinical syndromes represented in RRT calls, but should also focus on communication, leadership, and other essential teamwork skills. Simulation‐based training is an ideal teaching strategy for these objectives, and prior studies support the effectiveness of this approach.[10, 11]

The ACGME has now implemented the Next Accreditation System (NAS) across all specialties. Of the 22 reporting milestones within internal medicine, 12 relate directly to quality improvement and patient safety objectives, whereas 6 relate directly to pathophysiology and disease management.[12] Educating residents on systems of care is further highlighted by the Clinical Learning Environment Review (CLER), a key component of the NAS. The CLER program uses site visits to identify teaching hospitals' efforts to engage residents in 6 focus areas: patient safety; healthcare quality; transitions of care; supervision; duty hours, fatigue management, and mitigation; and professionalism.[13] CLER site visits include discussions and observations with hospital executive leadership, residents, graduate medical education leadership, nursing, and other hospital staff. The CLER program raises the bar for integrating medical education and quality improvement efforts even further. Quality improvement activities that previously supported an informal curriculum must now be made explicit to, and deliberately engage, our residents. Teaching hospitals are being tasked with including residents in safety initiatives and on all quality committees, especially those with cross‐departmental boundaries such as the Emergency Response Team/RRT Committee. Residents should meaningfully participate, and whenever possible, lead quality improvement projects, the focus of which may ideally be identified by residents themselves. An important resource for medical educators is the Quality and Safety Educators Academy, a program developed by the Society of Hospital Medicine and the Alliance for Academic Internal Medicine, which provides educators with the knowledge and tools to integrate quality improvement and patient safety objectives into their training programs.[14]

In conclusion, we are reassured that residents understand the intrinsic value of an RRT to patient care and their education. We encourage medical educators to use RRTs as an opportunity to develop interprofessional curricula, including those that aim to enhance teamwork skills. Beyond curricular innovation, quality‐improvement activities in teaching hospitals must deliberately engage our residents at every level of the organization.

The adoption of electronic health records (EHRs) in US hospitals continues to rise steeply, with nearly 60% of all hospitals having at least a basic EHR as of 2014.[1] EHRs bring with them the ability to inform and guide clinicians as they make decisions. In theory, this form of clinical decision support (CDS) ensures quality of care, reduces adverse events, and improves efficiency; in practice, experience in the field paints a mixed picture.[2, 3] This issue of the Journal of Hospital Medicine presents 3 examples of CDS that illustrate the distance between what we see as CDS' full potential and current limitations.

In the study by Herzig et al.[4] investigators took on the challenge of implementing stress ulcer prophylaxis guidelines developed by the Society of Hospital Medicine. The investigators first demonstrated that targeted electronic prompts captured patients' indications for acid suppressive therapy, and could be used to prohibit prescribers from ordering acid suppressive therapy among patients outside the intensive care unit (ICU) setting. Through an elegant interrupted time series study design deployed across 2 hospital campuses, the investigators were able to demonstrate immediate and clinically significant reduction in acid suppressive therapy outside the ICU. They further found that the impact of this reduction was augmented over time, suggesting that the electronic prompts had a sustained impact on provider ordering behavior. However, below the headlineand relevant to the limitations of CDSthe investigators noted that much of the reduction in the use of acid suppressive therapy for stress ulcer prophylaxis could be accounted for by providers' choice of another acceptable indication (eg, continuing preadmission medication). The authors speculated that the CDS intervention prompted providers to more accurately record the indication for acid suppressive therapy. It is also possible that providers simply chose an alternate indication to circumvent the decision‐support step. Perhaps as a result of these 2 offsetting factors, the actual use of acid suppressive therapy, regardless of indication, only decreased in a modest and statistically nonsignificant way, casting the true effectiveness of this CDS intervention into question.

Two other studies in this issue of the Journal of Hospital Medicine[5, 6] provide valuable insights into interactions between social and technical factors[7, 8, 9, 10] that determine the success or failure in the use of technology such as CDS to drive organizational performance. At the technical end of this sociotechnical spectrum, the study by Knight et al.[5] illustrated that a minimally configured and visually unintuitive medication decision‐support system resulted in a high number of alerts (approximately 17% of studied orders), leading to the well‐reported phenomena of alert fatigue and substantially lower response rate compared to those reported in the literature.[11, 12, 13] Moreover, the analysis suggested that response to these alerts were particularly muted among situations that were particularly high risk, including the patient being older, patient having a greater length of stay, care being delivered in the internal medicine service, resident physician being the prescriber, and the medication being on the Institute for Safe Medication Practices list of high‐alert medications. The investigators concluded that a redesign of the medication decision‐support system was needed.