Jonathan H. Chen, MD, PhD

Iatrogenic pneumothorax (IAP) is a complication of invasive procedures that is associated with substantial morbidity and some mortality.[1] IAP is often avoidable, and in many cases can be prevented through adherence to evidence‐based guidelines and procedural techniques known to reduce the incidence of IAP.[2] IAP may occur with a subclavian (SC) or internal jugular (IJ) central venous catheter (CVC) insertion, but is more frequently associated with the SC approach.[3] Ultrasound guidance during IJ CVC insertion is associated with a lower risk as compared to guidance by anatomical landmarks.[4, 5] Other bedside procedures that are known to cause IAP include thoracentesis. This risk can also be reduced with the use of ultrasound guidance.[6]

Including simulation in training for CVC insertion has been demonstrated in meta‐analyses to improve both learner outcomes, including simulator performance and perceived confidence, and patient outcomes, including fewer failed CVC attempts and reduced incidence of IAP.[7] Even brief simulation workshops lasting less than two hours can improve patient safety during CVC insertion.[8]

The implementation of ultrasound‐based simulation and improved adherence to the actual use of ultrasound at the bedside can be motivated by tying competency‐based educational objectives (eg, CVC insertion) to clinical outcomes (ie, rates of IAP) and tracking both as part of a continuous quality‐improvement cycle.[9] Adherence to best practices for CVC insertion can also be improved through standardizing hospital‐wide policies and hands‐on training.[10] Involving many stakeholders, including nurses, physicians, nurse practioners and physician assistants, in a multidisciplinary team has been shown to help alter entrenched behaviors and reduce the incidence of central‐line associated bloodstream infections through long‐term adherence to evidence‐based interventions.[11]

LOCAL PROBLEM

The Agency for Healthcare Research and Quality (AHRQ) has designed Patient Safety Indicators (PSIs) (http://qualityindicators.ahrq.gov/modules/psi_resources.aspx) that use hospital discharge coding data to quantify rates of hospital complications. The PSI for IAP has been validated in the literature, and has been adopted by the National Quality Forum and the Centers for Medicare and Medicaid Services as a tool for measuring hospital quality and safety.[12, 13]

Our hospital is a member of the University HealthSystem Consortium (UHC) (https://www.uhc.edu/), an alliance of 116 academic medical centers that share and compare patient outcome data. The UHC Clinical Database allows a hospital to identify areas of underperformance by the PSI as compared to peers, with the understanding that potential variation among hospitals in coding is a potential problem in using the PSI.

Despite this, the PSI can highlight areas where quality‐improvement efforts might be best directed. In 2005 and 2006, our hospital was ranked within the lowest UHC performance quartile for all‐cause IAP PSI.

During FY 2006 (September 2005August 2006), root‐cause analysis on cases of IAP at our hospital found that CVC insertion (40%) was the most common procedure associated with IAP, with SC insertion causing 69% of CVC‐associated IAP. Other common procedures associated with IAP were operative/pacemaker (30%), thoracentesis (25%), and ventilator associated (5%). Ultrasound was not used in 2/5 cases of IJ CVC placement and 3/5 thoracentesis cases. Only 44% of CVC insertions had a procedure note.

METHODS

The format of this article is based on the standards for quality‐improvement reporting excellence guidelines for the reporting of studies on the effectiveness of quality‐improvement interventions.[14]

RESULTS

Outcomes

Preintervention (January 1, 2006 to December 31, 2006)

There were a total of 26 cases of IAP in 2006. Of these, 15 (58%) were associated with CVC insertion (Figure 1). The single procedure associated with the largest proportion of IAP was SC CVC insertion (11 cases, 42% of all IAP cases). Eleven CVC‐associated IAPs were significant enough to require chest tube placement. Our hospital recorded a total of 1593 CVC insertions (ICD‐9 codes 38.93 and 38.97) in 2006.

Figure 1

Postintervention (January 1, 2008 to December 31, 2014)

There were a total of 80 cases of IAP over 7 years, of which 24 (30%) were associated with CVC insertion. Of these, 16 required chest tube placement. In the last 4 years of the postintervention period (20112014), there were only 5 cases of CVC‐associated IAP requiring chest tube placement (Figure 1). There were a total of 12,000 CVC insertions recorded over the same period.

We successfully met both our short‐ and long‐term goals. Our preintervention CVC‐associated IAP rate was 0.94%, and our post‐intervention rate during 2008 was 0.44%, a short‐term reduction of 53% (P=0.088). Our average postintervention CVC‐associated IAP rate for the years 2008 through 2014 was 0.13%, a significant long‐term reduction of 86% (P<0.0001) (Table 2). The decrease in CVC‐associated IAP rates occurred despite an older patient population (P<0.001) and a higher CMI (P<0.001) in postintervention patients who received a CVC (Table 1). Special cause variation corresponding to a change in our process is demonstrated in Figure 2. The preintervention average number of procedures between IAP was 114.8 and increased to 460.7 in the postintervention period.

Table 2.Summary of Outcomes

	Total CVC (n=95)		Subclavian (n=29)			Internal Jugular (n=58)					Femoral (n=8)
NOTE: Chart review of CVC placement in ICU during 2014 showing high compliance to intervention: US guided IJ placement, standard procedure note completion, and supervision, and significant long‐term reduction in CVC‐associated IAP rate and all‐cause IAP rate. A total of 100 patient charts were randomly selected and reviewed. Five patients receiving a PICC were not included in the analysis. All PICC lines were placed by trained registered nurses without supervision under US guidance. Of note, proportions of CVC types are not accurate representation; additional subclavian and internal jugular lines were randomly selected and reviewed. Abbreviations: CVC, central venous catheterization; IAP, iatrogenic pneumothorax; ICU intensive are unit; IJ, internal jugular; MICU, medical intensive care unit; PICC, peripherally inserted central catheter; SICU, surgical intensive care unit; US, ultrasound. *Surgical ICU includes all surgery primary teams including trauma, general surgery, cardiothoracic surgery, and neurosurgery.
Compliance to intervention
US guided	68.1%		20.7%			86.2%					100.0%
Procedure note completion	90.4%		93.1%			86.2%					100.0%
Supervision	70.2%		77.8%			73.1%					87.5%
Level of training
Resident	61.1%		58.6%			60.3%					75.0%
Fellow	25.3%		27.6%			24.1%					25.0%
Attending	4.2%		6.9%			3.4%					0.0%
Advance practitioner	3.2%		3.4%			3.4%					0.0%
Unknown	6.3%		3.4%			8.6%					0.0%
Supervisor type
Resident	3.0%		4.8%			2.6%					0.0%
Fellow	54.5%		33.3%			57.9%					100.0%
Attending	42.4%		61.9%			39.5%					0.0%
Location of CVC Placement	Internal Jugular (n=25)					Subclavian (n=25)
MICU	32.0%					32.0%
SICU*	40.0%					52.0%
Operating room	28.0%					16.0%
Average no. of attempts/procedure	1.4					1.5
Indications for subclavian insertion (n=25)
Trauma/surgical site	60.0%
Need for additional access	16.0%
Unsuccessful IJ placement	4.0%
Unclear	20.0%
Iatrogenic Pneumothorax Rate (20062014)	Year
	2006	2007		2008	2009		2010	2011	2012			2013	2014
% of CVC insertions associated with IAP	0.94	1.49		0.44	0.13		0.20	0.07	0.04			0.11	0.07
All‐cause IAP per 1,000 discharges	1.32	1.29		0.98	0.71		0.83	0.49	0.13			0.35	0.23
	Preintervention				Postintervention					P Value
CVC‐ associated IAP short term (2006 vs 2008)	0.94%				0.44%					0.088
CVC‐associated IAP long term (2006 vs 20082014)	0.94%				0.13%					<0.0001
All‐cause IAP per 1,000 discharges short term (2006 vs 2008)	1.32				0.98					<0.0001
All‐cause IAP per 1,000 discharges long term (2006 vs 2008‐14)	1.32				0.52					<0.0001

Figure 2

With the decrease in CVC‐associated IAP, we also saw a decrease in our all‐cause IAP rate per 1000 discharges from 1.32 in 2006 to 0.98 in 2008. This represents a 26% short‐term reduction (P<0.0001). We also saw a decrease in our all‐cause IAP rate per 1000 discharges to 0.52 from 2008 to 2014, representing a 61% long‐term reduction (P<0.0001). This decrease in all‐cause IAP postintervention occurred despite an older patient population (P<0.01) for all discharges. Our hospital is now in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014.

After our multifaceted intervention in 2007, there was substantially more and consistent documentation of CVC procedure notes from less than 500 in 2006 to greater than 2000 in 2009. The distribution of CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed a decrease in the proportion of femoral lines from 15% to 11%, increase in IJ lines from 31% to 49%, and a decrease in SC from 54% to 40% (P=0.001). The distribution of IJ CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed an increase in the proportion of procedures with ultrasound documentation from 13% to 93% (P<0.001) (Figure 3).

Figure 3

In our ICU 2014 audit, the majority of CVC lines were placed by residents under supervision (>70%), and most used the standard CVC note template to document the procedure (90%). Of the total CVC approach, 66% were IJ and 4% were SC. Eighty‐six percent used ultrasound during IJ placement. The majority of SC insertions were placed in the surgical ICU and had clear indications (80%) for placement. Of those, 75% were due to trauma (limited access to neck) or surgery (interfering with surgical site) (Table 2).

DISCUSSION

CONCLUSIONS

We are the first, to our knowledge, to report a measurable improvement in reducing IAP patient outcomes that has been sustained for over 7 years. Our hospital is in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014. A multidisciplinary quality‐improvement team, focused on evidence, patient safety, and standardization, can use a multifaceted intervention to sustainably improve patient outcomes. Promoting ultrasound‐guided IJ catheterization as the CVC insertion method of choice significantly reduced our hospital's rate of CVC‐associated IAP.

Iatrogenic pneumothorax (IAP) is a complication of invasive procedures that is associated with substantial morbidity and some mortality.[1] IAP is often avoidable, and in many cases can be prevented through adherence to evidence‐based guidelines and procedural techniques known to reduce the incidence of IAP.[2] IAP may occur with a subclavian (SC) or internal jugular (IJ) central venous catheter (CVC) insertion, but is more frequently associated with the SC approach.[3] Ultrasound guidance during IJ CVC insertion is associated with a lower risk as compared to guidance by anatomical landmarks.[4, 5] Other bedside procedures that are known to cause IAP include thoracentesis. This risk can also be reduced with the use of ultrasound guidance.[6]

Including simulation in training for CVC insertion has been demonstrated in meta‐analyses to improve both learner outcomes, including simulator performance and perceived confidence, and patient outcomes, including fewer failed CVC attempts and reduced incidence of IAP.[7] Even brief simulation workshops lasting less than two hours can improve patient safety during CVC insertion.[8]

The implementation of ultrasound‐based simulation and improved adherence to the actual use of ultrasound at the bedside can be motivated by tying competency‐based educational objectives (eg, CVC insertion) to clinical outcomes (ie, rates of IAP) and tracking both as part of a continuous quality‐improvement cycle.[9] Adherence to best practices for CVC insertion can also be improved through standardizing hospital‐wide policies and hands‐on training.[10] Involving many stakeholders, including nurses, physicians, nurse practioners and physician assistants, in a multidisciplinary team has been shown to help alter entrenched behaviors and reduce the incidence of central‐line associated bloodstream infections through long‐term adherence to evidence‐based interventions.[11]

LOCAL PROBLEM

The Agency for Healthcare Research and Quality (AHRQ) has designed Patient Safety Indicators (PSIs) (http://qualityindicators.ahrq.gov/modules/psi_resources.aspx) that use hospital discharge coding data to quantify rates of hospital complications. The PSI for IAP has been validated in the literature, and has been adopted by the National Quality Forum and the Centers for Medicare and Medicaid Services as a tool for measuring hospital quality and safety.[12, 13]

Our hospital is a member of the University HealthSystem Consortium (UHC) (https://www.uhc.edu/), an alliance of 116 academic medical centers that share and compare patient outcome data. The UHC Clinical Database allows a hospital to identify areas of underperformance by the PSI as compared to peers, with the understanding that potential variation among hospitals in coding is a potential problem in using the PSI.

Despite this, the PSI can highlight areas where quality‐improvement efforts might be best directed. In 2005 and 2006, our hospital was ranked within the lowest UHC performance quartile for all‐cause IAP PSI.

During FY 2006 (September 2005August 2006), root‐cause analysis on cases of IAP at our hospital found that CVC insertion (40%) was the most common procedure associated with IAP, with SC insertion causing 69% of CVC‐associated IAP. Other common procedures associated with IAP were operative/pacemaker (30%), thoracentesis (25%), and ventilator associated (5%). Ultrasound was not used in 2/5 cases of IJ CVC placement and 3/5 thoracentesis cases. Only 44% of CVC insertions had a procedure note.

METHODS

The format of this article is based on the standards for quality‐improvement reporting excellence guidelines for the reporting of studies on the effectiveness of quality‐improvement interventions.[14]

RESULTS

Outcomes

Preintervention (January 1, 2006 to December 31, 2006)

There were a total of 26 cases of IAP in 2006. Of these, 15 (58%) were associated with CVC insertion (Figure 1). The single procedure associated with the largest proportion of IAP was SC CVC insertion (11 cases, 42% of all IAP cases). Eleven CVC‐associated IAPs were significant enough to require chest tube placement. Our hospital recorded a total of 1593 CVC insertions (ICD‐9 codes 38.93 and 38.97) in 2006.

Figure 1

Postintervention (January 1, 2008 to December 31, 2014)

There were a total of 80 cases of IAP over 7 years, of which 24 (30%) were associated with CVC insertion. Of these, 16 required chest tube placement. In the last 4 years of the postintervention period (20112014), there were only 5 cases of CVC‐associated IAP requiring chest tube placement (Figure 1). There were a total of 12,000 CVC insertions recorded over the same period.

We successfully met both our short‐ and long‐term goals. Our preintervention CVC‐associated IAP rate was 0.94%, and our post‐intervention rate during 2008 was 0.44%, a short‐term reduction of 53% (P=0.088). Our average postintervention CVC‐associated IAP rate for the years 2008 through 2014 was 0.13%, a significant long‐term reduction of 86% (P<0.0001) (Table 2). The decrease in CVC‐associated IAP rates occurred despite an older patient population (P<0.001) and a higher CMI (P<0.001) in postintervention patients who received a CVC (Table 1). Special cause variation corresponding to a change in our process is demonstrated in Figure 2. The preintervention average number of procedures between IAP was 114.8 and increased to 460.7 in the postintervention period.

Table 2.Summary of Outcomes

	Total CVC (n=95)		Subclavian (n=29)			Internal Jugular (n=58)					Femoral (n=8)
NOTE: Chart review of CVC placement in ICU during 2014 showing high compliance to intervention: US guided IJ placement, standard procedure note completion, and supervision, and significant long‐term reduction in CVC‐associated IAP rate and all‐cause IAP rate. A total of 100 patient charts were randomly selected and reviewed. Five patients receiving a PICC were not included in the analysis. All PICC lines were placed by trained registered nurses without supervision under US guidance. Of note, proportions of CVC types are not accurate representation; additional subclavian and internal jugular lines were randomly selected and reviewed. Abbreviations: CVC, central venous catheterization; IAP, iatrogenic pneumothorax; ICU intensive are unit; IJ, internal jugular; MICU, medical intensive care unit; PICC, peripherally inserted central catheter; SICU, surgical intensive care unit; US, ultrasound. *Surgical ICU includes all surgery primary teams including trauma, general surgery, cardiothoracic surgery, and neurosurgery.
Compliance to intervention
US guided	68.1%		20.7%			86.2%					100.0%
Procedure note completion	90.4%		93.1%			86.2%					100.0%
Supervision	70.2%		77.8%			73.1%					87.5%
Level of training
Resident	61.1%		58.6%			60.3%					75.0%
Fellow	25.3%		27.6%			24.1%					25.0%
Attending	4.2%		6.9%			3.4%					0.0%
Advance practitioner	3.2%		3.4%			3.4%					0.0%
Unknown	6.3%		3.4%			8.6%					0.0%
Supervisor type
Resident	3.0%		4.8%			2.6%					0.0%
Fellow	54.5%		33.3%			57.9%					100.0%
Attending	42.4%		61.9%			39.5%					0.0%
Location of CVC Placement	Internal Jugular (n=25)					Subclavian (n=25)
MICU	32.0%					32.0%
SICU*	40.0%					52.0%
Operating room	28.0%					16.0%
Average no. of attempts/procedure	1.4					1.5
Indications for subclavian insertion (n=25)
Trauma/surgical site	60.0%
Need for additional access	16.0%
Unsuccessful IJ placement	4.0%
Unclear	20.0%
Iatrogenic Pneumothorax Rate (20062014)	Year
	2006	2007		2008	2009		2010	2011	2012			2013	2014
% of CVC insertions associated with IAP	0.94	1.49		0.44	0.13		0.20	0.07	0.04			0.11	0.07
All‐cause IAP per 1,000 discharges	1.32	1.29		0.98	0.71		0.83	0.49	0.13			0.35	0.23
	Preintervention				Postintervention					P Value
CVC‐ associated IAP short term (2006 vs 2008)	0.94%				0.44%					0.088
CVC‐associated IAP long term (2006 vs 20082014)	0.94%				0.13%					<0.0001
All‐cause IAP per 1,000 discharges short term (2006 vs 2008)	1.32				0.98					<0.0001
All‐cause IAP per 1,000 discharges long term (2006 vs 2008‐14)	1.32				0.52					<0.0001

Figure 2

With the decrease in CVC‐associated IAP, we also saw a decrease in our all‐cause IAP rate per 1000 discharges from 1.32 in 2006 to 0.98 in 2008. This represents a 26% short‐term reduction (P<0.0001). We also saw a decrease in our all‐cause IAP rate per 1000 discharges to 0.52 from 2008 to 2014, representing a 61% long‐term reduction (P<0.0001). This decrease in all‐cause IAP postintervention occurred despite an older patient population (P<0.01) for all discharges. Our hospital is now in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014.

After our multifaceted intervention in 2007, there was substantially more and consistent documentation of CVC procedure notes from less than 500 in 2006 to greater than 2000 in 2009. The distribution of CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed a decrease in the proportion of femoral lines from 15% to 11%, increase in IJ lines from 31% to 49%, and a decrease in SC from 54% to 40% (P=0.001). The distribution of IJ CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed an increase in the proportion of procedures with ultrasound documentation from 13% to 93% (P<0.001) (Figure 3).

Figure 3

In our ICU 2014 audit, the majority of CVC lines were placed by residents under supervision (>70%), and most used the standard CVC note template to document the procedure (90%). Of the total CVC approach, 66% were IJ and 4% were SC. Eighty‐six percent used ultrasound during IJ placement. The majority of SC insertions were placed in the surgical ICU and had clear indications (80%) for placement. Of those, 75% were due to trauma (limited access to neck) or surgery (interfering with surgical site) (Table 2).

DISCUSSION

CONCLUSIONS

We are the first, to our knowledge, to report a measurable improvement in reducing IAP patient outcomes that has been sustained for over 7 years. Our hospital is in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014. A multidisciplinary quality‐improvement team, focused on evidence, patient safety, and standardization, can use a multifaceted intervention to sustainably improve patient outcomes. Promoting ultrasound‐guided IJ catheterization as the CVC insertion method of choice significantly reduced our hospital's rate of CVC‐associated IAP.

Iatrogenic pneumothorax (IAP) is a complication of invasive procedures that is associated with substantial morbidity and some mortality.[1] IAP is often avoidable, and in many cases can be prevented through adherence to evidence‐based guidelines and procedural techniques known to reduce the incidence of IAP.[2] IAP may occur with a subclavian (SC) or internal jugular (IJ) central venous catheter (CVC) insertion, but is more frequently associated with the SC approach.[3] Ultrasound guidance during IJ CVC insertion is associated with a lower risk as compared to guidance by anatomical landmarks.[4, 5] Other bedside procedures that are known to cause IAP include thoracentesis. This risk can also be reduced with the use of ultrasound guidance.[6]

Including simulation in training for CVC insertion has been demonstrated in meta‐analyses to improve both learner outcomes, including simulator performance and perceived confidence, and patient outcomes, including fewer failed CVC attempts and reduced incidence of IAP.[7] Even brief simulation workshops lasting less than two hours can improve patient safety during CVC insertion.[8]

The implementation of ultrasound‐based simulation and improved adherence to the actual use of ultrasound at the bedside can be motivated by tying competency‐based educational objectives (eg, CVC insertion) to clinical outcomes (ie, rates of IAP) and tracking both as part of a continuous quality‐improvement cycle.[9] Adherence to best practices for CVC insertion can also be improved through standardizing hospital‐wide policies and hands‐on training.[10] Involving many stakeholders, including nurses, physicians, nurse practioners and physician assistants, in a multidisciplinary team has been shown to help alter entrenched behaviors and reduce the incidence of central‐line associated bloodstream infections through long‐term adherence to evidence‐based interventions.[11]

LOCAL PROBLEM

The Agency for Healthcare Research and Quality (AHRQ) has designed Patient Safety Indicators (PSIs) (http://qualityindicators.ahrq.gov/modules/psi_resources.aspx) that use hospital discharge coding data to quantify rates of hospital complications. The PSI for IAP has been validated in the literature, and has been adopted by the National Quality Forum and the Centers for Medicare and Medicaid Services as a tool for measuring hospital quality and safety.[12, 13]

Our hospital is a member of the University HealthSystem Consortium (UHC) (https://www.uhc.edu/), an alliance of 116 academic medical centers that share and compare patient outcome data. The UHC Clinical Database allows a hospital to identify areas of underperformance by the PSI as compared to peers, with the understanding that potential variation among hospitals in coding is a potential problem in using the PSI.

Despite this, the PSI can highlight areas where quality‐improvement efforts might be best directed. In 2005 and 2006, our hospital was ranked within the lowest UHC performance quartile for all‐cause IAP PSI.

During FY 2006 (September 2005August 2006), root‐cause analysis on cases of IAP at our hospital found that CVC insertion (40%) was the most common procedure associated with IAP, with SC insertion causing 69% of CVC‐associated IAP. Other common procedures associated with IAP were operative/pacemaker (30%), thoracentesis (25%), and ventilator associated (5%). Ultrasound was not used in 2/5 cases of IJ CVC placement and 3/5 thoracentesis cases. Only 44% of CVC insertions had a procedure note.

METHODS

The format of this article is based on the standards for quality‐improvement reporting excellence guidelines for the reporting of studies on the effectiveness of quality‐improvement interventions.[14]

RESULTS

Outcomes

Preintervention (January 1, 2006 to December 31, 2006)

There were a total of 26 cases of IAP in 2006. Of these, 15 (58%) were associated with CVC insertion (Figure 1). The single procedure associated with the largest proportion of IAP was SC CVC insertion (11 cases, 42% of all IAP cases). Eleven CVC‐associated IAPs were significant enough to require chest tube placement. Our hospital recorded a total of 1593 CVC insertions (ICD‐9 codes 38.93 and 38.97) in 2006.

Figure 1

Postintervention (January 1, 2008 to December 31, 2014)

There were a total of 80 cases of IAP over 7 years, of which 24 (30%) were associated with CVC insertion. Of these, 16 required chest tube placement. In the last 4 years of the postintervention period (20112014), there were only 5 cases of CVC‐associated IAP requiring chest tube placement (Figure 1). There were a total of 12,000 CVC insertions recorded over the same period.

We successfully met both our short‐ and long‐term goals. Our preintervention CVC‐associated IAP rate was 0.94%, and our post‐intervention rate during 2008 was 0.44%, a short‐term reduction of 53% (P=0.088). Our average postintervention CVC‐associated IAP rate for the years 2008 through 2014 was 0.13%, a significant long‐term reduction of 86% (P<0.0001) (Table 2). The decrease in CVC‐associated IAP rates occurred despite an older patient population (P<0.001) and a higher CMI (P<0.001) in postintervention patients who received a CVC (Table 1). Special cause variation corresponding to a change in our process is demonstrated in Figure 2. The preintervention average number of procedures between IAP was 114.8 and increased to 460.7 in the postintervention period.

Table 2.Summary of Outcomes

	Total CVC (n=95)		Subclavian (n=29)			Internal Jugular (n=58)					Femoral (n=8)
NOTE: Chart review of CVC placement in ICU during 2014 showing high compliance to intervention: US guided IJ placement, standard procedure note completion, and supervision, and significant long‐term reduction in CVC‐associated IAP rate and all‐cause IAP rate. A total of 100 patient charts were randomly selected and reviewed. Five patients receiving a PICC were not included in the analysis. All PICC lines were placed by trained registered nurses without supervision under US guidance. Of note, proportions of CVC types are not accurate representation; additional subclavian and internal jugular lines were randomly selected and reviewed. Abbreviations: CVC, central venous catheterization; IAP, iatrogenic pneumothorax; ICU intensive are unit; IJ, internal jugular; MICU, medical intensive care unit; PICC, peripherally inserted central catheter; SICU, surgical intensive care unit; US, ultrasound. *Surgical ICU includes all surgery primary teams including trauma, general surgery, cardiothoracic surgery, and neurosurgery.
Compliance to intervention
US guided	68.1%		20.7%			86.2%					100.0%
Procedure note completion	90.4%		93.1%			86.2%					100.0%
Supervision	70.2%		77.8%			73.1%					87.5%
Level of training
Resident	61.1%		58.6%			60.3%					75.0%
Fellow	25.3%		27.6%			24.1%					25.0%
Attending	4.2%		6.9%			3.4%					0.0%
Advance practitioner	3.2%		3.4%			3.4%					0.0%
Unknown	6.3%		3.4%			8.6%					0.0%
Supervisor type
Resident	3.0%		4.8%			2.6%					0.0%
Fellow	54.5%		33.3%			57.9%					100.0%
Attending	42.4%		61.9%			39.5%					0.0%
Location of CVC Placement	Internal Jugular (n=25)					Subclavian (n=25)
MICU	32.0%					32.0%
SICU*	40.0%					52.0%
Operating room	28.0%					16.0%
Average no. of attempts/procedure	1.4					1.5
Indications for subclavian insertion (n=25)
Trauma/surgical site	60.0%
Need for additional access	16.0%
Unsuccessful IJ placement	4.0%
Unclear	20.0%
Iatrogenic Pneumothorax Rate (20062014)	Year
	2006	2007		2008	2009		2010	2011	2012			2013	2014
% of CVC insertions associated with IAP	0.94	1.49		0.44	0.13		0.20	0.07	0.04			0.11	0.07
All‐cause IAP per 1,000 discharges	1.32	1.29		0.98	0.71		0.83	0.49	0.13			0.35	0.23
	Preintervention				Postintervention					P Value
CVC‐ associated IAP short term (2006 vs 2008)	0.94%				0.44%					0.088
CVC‐associated IAP long term (2006 vs 20082014)	0.94%				0.13%					<0.0001
All‐cause IAP per 1,000 discharges short term (2006 vs 2008)	1.32				0.98					<0.0001
All‐cause IAP per 1,000 discharges long term (2006 vs 2008‐14)	1.32				0.52					<0.0001

Figure 2

With the decrease in CVC‐associated IAP, we also saw a decrease in our all‐cause IAP rate per 1000 discharges from 1.32 in 2006 to 0.98 in 2008. This represents a 26% short‐term reduction (P<0.0001). We also saw a decrease in our all‐cause IAP rate per 1000 discharges to 0.52 from 2008 to 2014, representing a 61% long‐term reduction (P<0.0001). This decrease in all‐cause IAP postintervention occurred despite an older patient population (P<0.01) for all discharges. Our hospital is now in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014.

After our multifaceted intervention in 2007, there was substantially more and consistent documentation of CVC procedure notes from less than 500 in 2006 to greater than 2000 in 2009. The distribution of CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed a decrease in the proportion of femoral lines from 15% to 11%, increase in IJ lines from 31% to 49%, and a decrease in SC from 54% to 40% (P=0.001). The distribution of IJ CVC procedure notes in the pre‐ (2006) versus postintervention (20082014) period showed an increase in the proportion of procedures with ultrasound documentation from 13% to 93% (P<0.001) (Figure 3).

Figure 3

In our ICU 2014 audit, the majority of CVC lines were placed by residents under supervision (>70%), and most used the standard CVC note template to document the procedure (90%). Of the total CVC approach, 66% were IJ and 4% were SC. Eighty‐six percent used ultrasound during IJ placement. The majority of SC insertions were placed in the surgical ICU and had clear indications (80%) for placement. Of those, 75% were due to trauma (limited access to neck) or surgery (interfering with surgical site) (Table 2).

DISCUSSION

CONCLUSIONS

We are the first, to our knowledge, to report a measurable improvement in reducing IAP patient outcomes that has been sustained for over 7 years. Our hospital is in the highest performance UHC quartile for all‐cause IAP in 2012 to 2014. A multidisciplinary quality‐improvement team, focused on evidence, patient safety, and standardization, can use a multifaceted intervention to sustainably improve patient outcomes. Promoting ultrasound‐guided IJ catheterization as the CVC insertion method of choice significantly reduced our hospital's rate of CVC‐associated IAP.

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.