Paul Haidet, MD, MPH

Interprofessional collaborative care (IPCC) involves members from different professions working together to enhance communication, coordination, and healthcare quality.[1, 2, 3] Because several current healthcare policy initiatives include financial incentives for increased quality of care, there has been resultant interest in the implementation of IPCC in healthcare systems.[4, 5] Unfortunately, many hospitals have found IPCC difficult to achieve. Hospital‐based medicine units are complex, time‐constrained environments requiring a high degree of collaboration and mutual decision‐making between nurses, physicians, therapists, pharmacists, care coordinators, and patients. In addition, despite recommendations for interprofessional collaborative care, the implementation and assessment of IPCC within this environment has not been well studied.[6, 7]

On academic internal medicine services, the majority of care decisions occur during rounds. Although rounds provide a common structure, the participants, length, location, and agenda of rounds tend to vary by institution and individual physician preference.[8, 9, 10, 11] Traditionally, ward rounds occur mostly in hallways and conference rooms rather than the patient's bedside.[12] Additionally, during rounds, nurse‐physician collaboration occurs infrequently, estimated at <10% of rounding time.[13] Recently, an increased focus on quality, safety, and collaboration has inspired the investigation and implementation of new methods to increase interprofessional collaboration during rounds, but many of these interventions occurred away from the patient's bedside.[14, 15] One trial of bedside interprofessional rounds (BIRs) by Curley et al. suggested improvements in patient‐level outcomes (cost and length of stay) versus traditional physician‐based rounds.[16] Although interprofessional nurse‐physician rounds at patients' bedsides may represent an ideal process, limited work has investigated this activity.[17]

A prerequisite for successful and sustained integration of BIRs is a shared conceptualization among physicians and nurses regarding the process. Such a shared conceptualization would include perceptions of benefits and barriers to implementation.[18] Currently, such perceptions have not been measured. In this study, we sought to evaluate perceptions of front‐line care providers on inpatient units, specifically nursing staff, attending physicians, and housestaff physicians, regarding the benefits and barriers to BIRs.

METHODS

RESULTS

Of the 171 surveys sent, 149 participants completed surveys (response rate 87%). Responses were received from 53/58 nursing staff (91% response), 21/28 attending physicians (75% response), and 75/85 housestaff physicians (88% response). Table 1 describes the participant response demographics.

DISCUSSION

In this study, we sought to compare perceptions of nurses and physicians on the benefits and barriers to BIRs. Nursing staff ranked each benefit higher than physicians, though rank orders of specific benefits were highly correlated. Highest‐ranked benefits related to coordination and communication more than quality or process benefits. Across groups, the highest‐ranked barriers to BIRs were related to time, whereas the lowest‐ranked factors were related to provider and patient discomfort. These results highlight important similarities and differences in perceptions between front‐line providers.

The highest‐ranked benefits were related to improved interprofessional communication and coordination. Combining interprofessional team members during care delivery allows for integrated understanding of daily care plans and clinical issues, and fosters collaboration and a team‐based atmosphere.[1, 20, 26] The lowest‐ranked benefits were related to more tangible measures, including length of stay, timely consultations, and judicious laboratory ordering. This finding contrasts with the limited literature demonstrating increased efficiency in general medicine units practicing IPCC.[16] These rankings may reflect a poor understanding or self‐assessment of outcome measures by healthcare providers, representing a potential focus for educational initiatives. Future investigations using objective assessment methods of outcomes and collaboration will provide a more accurate understanding of these findings.

The highest‐ranked barriers were related to time and systems issues. Several studies of physician‐based bedside rounds have identified systems‐ and time‐related issues as primary limiting barriers.[22, 24] In units without colocalization of patients and providers, finding receptive times for BIRs can be difficult. Although time‐related issues could be addressed by decreasing patient‐provider ratios, these changes require substantial investment in resources. A reasonable degree of improvement in efficiency and coordination is expected following acclimation to BIRs or by addressing modifiable systems factors to increase this activity. Less costly interventions, such as tailoring provider schedules, prescheduling patient rounding times, and geographic colocalization of patients and providers may be more feasible. However, the clinical microsystems within which medicine patients are cared for are often chaotic and disorganized at the infrastructural and cultural levels, which may be less influenced by surface‐level interventions. Such interventions may be ward specific and require customization to individual team needs.

The lowest‐ranked barriers to BIRs were related to provider‐ and patient‐related factors, including comfort level of patients and providers. Prior work on bedside rounds has identified physicians who are apprehensive about performing bedside rounds, but those who experience this activity are more likely to be comfortable with it.[12, 28] Our results from a culture where BIRs occur on nearly two‐thirds of patients suggest provider discomfort is not a predominant barrier.[22, 29] Additionally, educators have raised concerns about patient discomfort with bedside rounds, but nearly all studies evaluating patients' perspectives reveal patient preference for bedside case presentations over activities occurring in alternative locations.[30, 31, 32] Little work has investigated patient preference for BIRs as per our definition; our participants do not believe patients are discomforted by BIRs, building upon evidence in the literature for patient preferences regarding bedside activities.

Nursing staff perceptions of the benefits and culture related to BIRs were more positive than physicians. We hypothesize several reasons for this disparity. First, nursing staff may have more experience with observing and understanding the positive impact of BIRs and therefore are more likely to understand the positive ramifications. Alternatively, nursing staff may be satisfied with active integration into traditional physician‐centric decisions. Additionally, the professional culture and educational foundation of the nursing culture is based upon a patient‐centered approach and therefore may be more aligned with the goals of BIRs. Last, physicians may have competing priorities, favoring productivity and didactic learning rather than interprofessional collaboration. Further investigation is required to understand differences between nurses and physicians, in addition to other providers integral to BIRs (eg, care coordinators, pharmacists). Regardless, during the implementation of interprofessional collaborative care models, our findings suggest initial challenges, and the focus of educational initiatives may necessitate acclimating physician groups to benefits identified by front‐line nursing staff.

There are several limitations to our study. We investigated the perceptions of medicine nurses and physicians in 1 teaching hospital, limiting generalizability to other specialties, other vital professional groups, and nonteaching hospitals. Additionally, BIRs has been a focus of our hospital for several years. Therefore, perceived barriers may differ in BIRs‐nave hospitals. Second, although pilot‐tested for content, the construct validity of the instrument was not rigorously assessed, and the instrument was not designed to measure benefits and barriers not explicitly identified during pilot testing. Last, although surveys were anonymous, the possibility of social desirability bias exists, thereby limiting accuracy.

For over a century, physician‐led rounds have been the preferred modality for point‐of‐care decision making.[10, 15, 32, 33] BIRs address our growing understanding of patient‐centered care. Future efforts should address the quality of collaboration and current hospital and unit structures hindering patient‐centered IPCC and patient outcomes.

Interprofessional collaborative care (IPCC) involves members from different professions working together to enhance communication, coordination, and healthcare quality.[1, 2, 3] Because several current healthcare policy initiatives include financial incentives for increased quality of care, there has been resultant interest in the implementation of IPCC in healthcare systems.[4, 5] Unfortunately, many hospitals have found IPCC difficult to achieve. Hospital‐based medicine units are complex, time‐constrained environments requiring a high degree of collaboration and mutual decision‐making between nurses, physicians, therapists, pharmacists, care coordinators, and patients. In addition, despite recommendations for interprofessional collaborative care, the implementation and assessment of IPCC within this environment has not been well studied.[6, 7]

On academic internal medicine services, the majority of care decisions occur during rounds. Although rounds provide a common structure, the participants, length, location, and agenda of rounds tend to vary by institution and individual physician preference.[8, 9, 10, 11] Traditionally, ward rounds occur mostly in hallways and conference rooms rather than the patient's bedside.[12] Additionally, during rounds, nurse‐physician collaboration occurs infrequently, estimated at <10% of rounding time.[13] Recently, an increased focus on quality, safety, and collaboration has inspired the investigation and implementation of new methods to increase interprofessional collaboration during rounds, but many of these interventions occurred away from the patient's bedside.[14, 15] One trial of bedside interprofessional rounds (BIRs) by Curley et al. suggested improvements in patient‐level outcomes (cost and length of stay) versus traditional physician‐based rounds.[16] Although interprofessional nurse‐physician rounds at patients' bedsides may represent an ideal process, limited work has investigated this activity.[17]

A prerequisite for successful and sustained integration of BIRs is a shared conceptualization among physicians and nurses regarding the process. Such a shared conceptualization would include perceptions of benefits and barriers to implementation.[18] Currently, such perceptions have not been measured. In this study, we sought to evaluate perceptions of front‐line care providers on inpatient units, specifically nursing staff, attending physicians, and housestaff physicians, regarding the benefits and barriers to BIRs.

METHODS

RESULTS

Of the 171 surveys sent, 149 participants completed surveys (response rate 87%). Responses were received from 53/58 nursing staff (91% response), 21/28 attending physicians (75% response), and 75/85 housestaff physicians (88% response). Table 1 describes the participant response demographics.

DISCUSSION

In this study, we sought to compare perceptions of nurses and physicians on the benefits and barriers to BIRs. Nursing staff ranked each benefit higher than physicians, though rank orders of specific benefits were highly correlated. Highest‐ranked benefits related to coordination and communication more than quality or process benefits. Across groups, the highest‐ranked barriers to BIRs were related to time, whereas the lowest‐ranked factors were related to provider and patient discomfort. These results highlight important similarities and differences in perceptions between front‐line providers.

The highest‐ranked benefits were related to improved interprofessional communication and coordination. Combining interprofessional team members during care delivery allows for integrated understanding of daily care plans and clinical issues, and fosters collaboration and a team‐based atmosphere.[1, 20, 26] The lowest‐ranked benefits were related to more tangible measures, including length of stay, timely consultations, and judicious laboratory ordering. This finding contrasts with the limited literature demonstrating increased efficiency in general medicine units practicing IPCC.[16] These rankings may reflect a poor understanding or self‐assessment of outcome measures by healthcare providers, representing a potential focus for educational initiatives. Future investigations using objective assessment methods of outcomes and collaboration will provide a more accurate understanding of these findings.

The highest‐ranked barriers were related to time and systems issues. Several studies of physician‐based bedside rounds have identified systems‐ and time‐related issues as primary limiting barriers.[22, 24] In units without colocalization of patients and providers, finding receptive times for BIRs can be difficult. Although time‐related issues could be addressed by decreasing patient‐provider ratios, these changes require substantial investment in resources. A reasonable degree of improvement in efficiency and coordination is expected following acclimation to BIRs or by addressing modifiable systems factors to increase this activity. Less costly interventions, such as tailoring provider schedules, prescheduling patient rounding times, and geographic colocalization of patients and providers may be more feasible. However, the clinical microsystems within which medicine patients are cared for are often chaotic and disorganized at the infrastructural and cultural levels, which may be less influenced by surface‐level interventions. Such interventions may be ward specific and require customization to individual team needs.

The lowest‐ranked barriers to BIRs were related to provider‐ and patient‐related factors, including comfort level of patients and providers. Prior work on bedside rounds has identified physicians who are apprehensive about performing bedside rounds, but those who experience this activity are more likely to be comfortable with it.[12, 28] Our results from a culture where BIRs occur on nearly two‐thirds of patients suggest provider discomfort is not a predominant barrier.[22, 29] Additionally, educators have raised concerns about patient discomfort with bedside rounds, but nearly all studies evaluating patients' perspectives reveal patient preference for bedside case presentations over activities occurring in alternative locations.[30, 31, 32] Little work has investigated patient preference for BIRs as per our definition; our participants do not believe patients are discomforted by BIRs, building upon evidence in the literature for patient preferences regarding bedside activities.

Nursing staff perceptions of the benefits and culture related to BIRs were more positive than physicians. We hypothesize several reasons for this disparity. First, nursing staff may have more experience with observing and understanding the positive impact of BIRs and therefore are more likely to understand the positive ramifications. Alternatively, nursing staff may be satisfied with active integration into traditional physician‐centric decisions. Additionally, the professional culture and educational foundation of the nursing culture is based upon a patient‐centered approach and therefore may be more aligned with the goals of BIRs. Last, physicians may have competing priorities, favoring productivity and didactic learning rather than interprofessional collaboration. Further investigation is required to understand differences between nurses and physicians, in addition to other providers integral to BIRs (eg, care coordinators, pharmacists). Regardless, during the implementation of interprofessional collaborative care models, our findings suggest initial challenges, and the focus of educational initiatives may necessitate acclimating physician groups to benefits identified by front‐line nursing staff.

There are several limitations to our study. We investigated the perceptions of medicine nurses and physicians in 1 teaching hospital, limiting generalizability to other specialties, other vital professional groups, and nonteaching hospitals. Additionally, BIRs has been a focus of our hospital for several years. Therefore, perceived barriers may differ in BIRs‐nave hospitals. Second, although pilot‐tested for content, the construct validity of the instrument was not rigorously assessed, and the instrument was not designed to measure benefits and barriers not explicitly identified during pilot testing. Last, although surveys were anonymous, the possibility of social desirability bias exists, thereby limiting accuracy.

For over a century, physician‐led rounds have been the preferred modality for point‐of‐care decision making.[10, 15, 32, 33] BIRs address our growing understanding of patient‐centered care. Future efforts should address the quality of collaboration and current hospital and unit structures hindering patient‐centered IPCC and patient outcomes.

Interprofessional collaborative care (IPCC) involves members from different professions working together to enhance communication, coordination, and healthcare quality.[1, 2, 3] Because several current healthcare policy initiatives include financial incentives for increased quality of care, there has been resultant interest in the implementation of IPCC in healthcare systems.[4, 5] Unfortunately, many hospitals have found IPCC difficult to achieve. Hospital‐based medicine units are complex, time‐constrained environments requiring a high degree of collaboration and mutual decision‐making between nurses, physicians, therapists, pharmacists, care coordinators, and patients. In addition, despite recommendations for interprofessional collaborative care, the implementation and assessment of IPCC within this environment has not been well studied.[6, 7]

On academic internal medicine services, the majority of care decisions occur during rounds. Although rounds provide a common structure, the participants, length, location, and agenda of rounds tend to vary by institution and individual physician preference.[8, 9, 10, 11] Traditionally, ward rounds occur mostly in hallways and conference rooms rather than the patient's bedside.[12] Additionally, during rounds, nurse‐physician collaboration occurs infrequently, estimated at <10% of rounding time.[13] Recently, an increased focus on quality, safety, and collaboration has inspired the investigation and implementation of new methods to increase interprofessional collaboration during rounds, but many of these interventions occurred away from the patient's bedside.[14, 15] One trial of bedside interprofessional rounds (BIRs) by Curley et al. suggested improvements in patient‐level outcomes (cost and length of stay) versus traditional physician‐based rounds.[16] Although interprofessional nurse‐physician rounds at patients' bedsides may represent an ideal process, limited work has investigated this activity.[17]

A prerequisite for successful and sustained integration of BIRs is a shared conceptualization among physicians and nurses regarding the process. Such a shared conceptualization would include perceptions of benefits and barriers to implementation.[18] Currently, such perceptions have not been measured. In this study, we sought to evaluate perceptions of front‐line care providers on inpatient units, specifically nursing staff, attending physicians, and housestaff physicians, regarding the benefits and barriers to BIRs.

METHODS

RESULTS

Of the 171 surveys sent, 149 participants completed surveys (response rate 87%). Responses were received from 53/58 nursing staff (91% response), 21/28 attending physicians (75% response), and 75/85 housestaff physicians (88% response). Table 1 describes the participant response demographics.

DISCUSSION

In this study, we sought to compare perceptions of nurses and physicians on the benefits and barriers to BIRs. Nursing staff ranked each benefit higher than physicians, though rank orders of specific benefits were highly correlated. Highest‐ranked benefits related to coordination and communication more than quality or process benefits. Across groups, the highest‐ranked barriers to BIRs were related to time, whereas the lowest‐ranked factors were related to provider and patient discomfort. These results highlight important similarities and differences in perceptions between front‐line providers.

The highest‐ranked benefits were related to improved interprofessional communication and coordination. Combining interprofessional team members during care delivery allows for integrated understanding of daily care plans and clinical issues, and fosters collaboration and a team‐based atmosphere.[1, 20, 26] The lowest‐ranked benefits were related to more tangible measures, including length of stay, timely consultations, and judicious laboratory ordering. This finding contrasts with the limited literature demonstrating increased efficiency in general medicine units practicing IPCC.[16] These rankings may reflect a poor understanding or self‐assessment of outcome measures by healthcare providers, representing a potential focus for educational initiatives. Future investigations using objective assessment methods of outcomes and collaboration will provide a more accurate understanding of these findings.

The highest‐ranked barriers were related to time and systems issues. Several studies of physician‐based bedside rounds have identified systems‐ and time‐related issues as primary limiting barriers.[22, 24] In units without colocalization of patients and providers, finding receptive times for BIRs can be difficult. Although time‐related issues could be addressed by decreasing patient‐provider ratios, these changes require substantial investment in resources. A reasonable degree of improvement in efficiency and coordination is expected following acclimation to BIRs or by addressing modifiable systems factors to increase this activity. Less costly interventions, such as tailoring provider schedules, prescheduling patient rounding times, and geographic colocalization of patients and providers may be more feasible. However, the clinical microsystems within which medicine patients are cared for are often chaotic and disorganized at the infrastructural and cultural levels, which may be less influenced by surface‐level interventions. Such interventions may be ward specific and require customization to individual team needs.

The lowest‐ranked barriers to BIRs were related to provider‐ and patient‐related factors, including comfort level of patients and providers. Prior work on bedside rounds has identified physicians who are apprehensive about performing bedside rounds, but those who experience this activity are more likely to be comfortable with it.[12, 28] Our results from a culture where BIRs occur on nearly two‐thirds of patients suggest provider discomfort is not a predominant barrier.[22, 29] Additionally, educators have raised concerns about patient discomfort with bedside rounds, but nearly all studies evaluating patients' perspectives reveal patient preference for bedside case presentations over activities occurring in alternative locations.[30, 31, 32] Little work has investigated patient preference for BIRs as per our definition; our participants do not believe patients are discomforted by BIRs, building upon evidence in the literature for patient preferences regarding bedside activities.

Nursing staff perceptions of the benefits and culture related to BIRs were more positive than physicians. We hypothesize several reasons for this disparity. First, nursing staff may have more experience with observing and understanding the positive impact of BIRs and therefore are more likely to understand the positive ramifications. Alternatively, nursing staff may be satisfied with active integration into traditional physician‐centric decisions. Additionally, the professional culture and educational foundation of the nursing culture is based upon a patient‐centered approach and therefore may be more aligned with the goals of BIRs. Last, physicians may have competing priorities, favoring productivity and didactic learning rather than interprofessional collaboration. Further investigation is required to understand differences between nurses and physicians, in addition to other providers integral to BIRs (eg, care coordinators, pharmacists). Regardless, during the implementation of interprofessional collaborative care models, our findings suggest initial challenges, and the focus of educational initiatives may necessitate acclimating physician groups to benefits identified by front‐line nursing staff.

There are several limitations to our study. We investigated the perceptions of medicine nurses and physicians in 1 teaching hospital, limiting generalizability to other specialties, other vital professional groups, and nonteaching hospitals. Additionally, BIRs has been a focus of our hospital for several years. Therefore, perceived barriers may differ in BIRs‐nave hospitals. Second, although pilot‐tested for content, the construct validity of the instrument was not rigorously assessed, and the instrument was not designed to measure benefits and barriers not explicitly identified during pilot testing. Last, although surveys were anonymous, the possibility of social desirability bias exists, thereby limiting accuracy.

For over a century, physician‐led rounds have been the preferred modality for point‐of‐care decision making.[10, 15, 32, 33] BIRs address our growing understanding of patient‐centered care. Future efforts should address the quality of collaboration and current hospital and unit structures hindering patient‐centered IPCC and patient outcomes.

Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7

Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16

Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.

Methods

Development of the Risk Assessment Tool

A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.

Figure 1

Results

Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).

Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.

The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).

Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).

Discussion

We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.

Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.

As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035

When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135

Figure 2

Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.

A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.

Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.

In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.

Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7

Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16

Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.

Methods

Development of the Risk Assessment Tool

A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.

Figure 1

Results

Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).

Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.

The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).

Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).

Discussion

We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.

Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.

As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035

When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135

Figure 2

Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.

A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.

Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.

In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.

Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7

Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16

Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.

Methods

Development of the Risk Assessment Tool

A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.

Figure 1

Results

Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).

Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.

The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).

Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).

Discussion

We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.

Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.

As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035

When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135

Figure 2

Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.

A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.

Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.

In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.