User login
Assessment Tool in the Hands of Medical Residents
Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7
Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16
Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.
Methods
Development of the Risk Assessment Tool
A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.
Construction of Clinical Vignettes
Approval was obtained by the Pennsylvania State University Institutional Review Board. Since previous research demonstrates the utility of clinical vignettes to study the effectiveness of guideline application and decision making, we used a series of 21 randomly selected and de‐identified clinical vignettes to portray a range of real‐world patient admission scenarios.2022 We identified individuals who had been admitted to the Hershey Medical Center using data from the inpatient electronic health record and applying the following inclusion criteria: age >17 years, and admission to a general medical service from the Emergency Department during a 14‐day period in 2008. Since more than 80% of patients admitted to our medical service are admitted through the Emergency Department and residents place all of the admission orders, our goal was to use vignettes that were typical of patients they commonly admit. We attached a paper form of our institution's VTE prophylaxis strategy (Figure 1) to each vignette.
Data Collection
A 1‐hour noon conference titled VTE Workshop was conducted by one of the authors (M.J.B.) during the first quarter of the 2008 academic year. We asked the medical residents to apply the VTE prophylaxis protocol to 21 vignettes during this session. In order to determine the appropriate time allotted to complete the vignettes, each case was completed by M.J.B. and three medical residents (one intern, one second year, and one third year) prior to conducting the VTE workshop. Based on these data, we determined that the median time to complete each vignette was 2 minutes and 15 seconds (range 30 seconds to 7 minutes). Therefore, we assumed that the 21 vignettes could be completed within 1 hour. At the beginning of the conference, the residents were provided with 10 minutes of verbal instruction about how to apply the VTE risk assessment tool. They were instructed to provide a total risk score (070 points) and, based on the total score, to classify each patient as low, moderate, or high risk for VTE. Following the risk assignment, they were instructed to document any absolute and relative contraindications. Finally, they were asked to select the most appropriate VTE prophylaxis plan according to the recommendations of the protocol. Vignettes were considered complete if they had an assessment and plan for >75% of the cases.
Prior to conducting this study, there had been no formal orientation regarding use of the VTE risk assessment tool or incorporation of it into our institution's computerized order entry system. Average attendance for the noon conference is between 20 and 30 house staff, approximately one‐third of the entire residency. Medical students were excluded from the study. All respondents voluntarily and anonymously performed the assessments, and indicated on the front of their vignette packet their level of training as PGY‐1, ‐2, or ‐3. The sessions were overseen by one of the authors to ensure that no communication occurred among the residents.
Data Analysis
We constructed a database with five variables collected from each resident's VTE risk assessment form: 1) a total risk score, 2) a risk classification (low, medium, or high), 3) the number and type of absolute contraindications to pharmacological prophylaxis, 4) the number and type of relative contraindications to pharmacological prophylaxis, and 5) a VTE prophylaxis plan. The lead author also performed these assessments of the 21 vignettes 1 month prior to the resident session. In power calculations performed prior to the session, we determined that the study would need at least 300 observations in order to calculate inter‐rater reliability.23 With the estimation that between 20 and 30 residents would attend, we determined that 21 vignettes would exceed the minimum required observations to allow for an accurate calculation of inter‐rater reliability.
The total risk score was treated as a continuous variable for which the intra‐class correlation (ICC) was calculated. The ICC is used to assess the consistency, or conformity, of measurements made by multiple observers measuring the same quantity.24 Risk stratification, presence of absolute and relative contraindications, and VTE plan were treated as categorical variables. For these, we used Cohen's kappa to assess variability in resident ratings. The kappa score has been used in other studies to determine inter‐rater reliability using similar VTE risk assessment tools.18, 19 Finally, adherence to our hospital's protocol was determined by comparing the residents' VTE plans with the lead author's VTE plans for each of the 21 vignettes. We used SAS 9.1.3 for all statistical analyses (SAS Institute, Cary, NC).
Results
Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).
| Risk Stratification | Resident no./total (%) | Attending no./total (%) |
|---|---|---|
| Low | 130/479 (27) | 3/21 (14) |
| Moderate | 183/479 (38) | 7/21 (33) |
| High | 166/479 (34) | 11/21 (52) |
Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.
| Attending Classification | Total | Appropriate Risk Assessment No. (%) | SCDs Only No. (%) | Heparin Only | Both Heparin and SCDs | Ambulation |
|---|---|---|---|---|---|---|
| ||||||
| Low risk | 115 | 86 (75) | 11 (10) | 7 (6) | 0 | 93 (84)a |
| Moderate risk | 138 | 85 (62) | 28 (21)b | 44 (33)a | 16 (12) | 47 (35) |
| High risk | 230 | 138 (60) | 39 (18)b | 69 (31) | 88 (40)a | 27 (12) |
| Total | 483 | 309 (64) | 78 (16) | 120 (26) | 104 (22) | 167 (36) |
The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).
Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).
| Risk Score | Stratification | Absolute Contraindication | Relative Contraindication | VTE Plan | ||
|---|---|---|---|---|---|---|
| ||||||
| Aggregate | ICC | 0.66 | ||||
| Kappa | 0.51 | 0.29 | 0.23 | 0.28 | ||
| Intern | ICC | 0.63 | ||||
| Kappa | 0.47 | NA | NA | 0.23 | ||
| Seniora | ICC | 0.73 | ||||
| Kappa | 0.61 | NA | NA | 0.35 | ||
Discussion
We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.
Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.
As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035
When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135
Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.
A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.
Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.
In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.
Acknowledgements
The authors thank Lisabeth V. Scalzi, MD, MS, Lora Moyer, Kevin McKenna MD, Hammid Al‐Mondhiry, Lucille Anderson, MD, Kathleen Williams, Kevin Larraway, Cynthia Chuang, MD, MS, the residents of the Internal Medicine and Combined Medicine/Pediatrics residencies, and the division of General Internal Medicine at Hershey Medical Center.
- ,,.Autopsy‐verified pulmonary embolism in a surgical department: analysis of the period from 1951 to 1988.Br J Surg.1991;78(7):849–852.
- , , , , et al.Fatal pulmonary embolism in hospitalised patients: a necropsy review.J Clin Pathol.2004;57(12):1254–1257.
- , , , et al.Prevention of venous thromboembolism: American College of Chest Physicians Evidence‐Based Clinical Practice Guidelines (8th edition).Chest.2008;133(6 suppl):381S–453S.
- ,.Coronary artery vasculitis and myocardial infarction with systemic lupus erythematosus.NY State J Med.1974;74(5):873–874.
- , , , et al.Venous thromboembolism prophylaxis in acutely ill hospitalized medical patients: findings from the International Medical Prevention Registry on Venous Thromboembolism.Chest.2007;132(3):936–945.
- , , , et al.Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross‐sectional study.Lancet.2008;371(9610):387–394.
- , , , .Venous thromboembolic events in hospitalised medical patients.Thromb Haemost.2009;102(3):505–510.
- ,,.Compliance with recommended prophylaxis for venous thromboembolism: improving the use and rate of uptake of clinical practice guidelines.J Thromb Haemost.2004;2(2):221–227.
- , , , et al.A systematic review of strategies to improve prophylaxis for venous thromboembolism in hospitals.Ann Surg.2005;241(3):397–415.
- ,.Prevention of venous thromboembolism: consensus, controversies, and challenges.Hematol Am Soc Hematol Educ Program.2009:286–292.
- ,,.International perspective on venous thromboembolism prophylaxis in surgery.Semin Thromb Hemost.1991;17(4):322–325.
- , , , et al.Clinical assessment of venous thromboembolic risk in surgical patients.Semin Thromb Hemost.1991;17(suppl 3):304–312.
- , et al.Risk factor assessment in the management of patients with suspected deep venous thrombosis.Int Angiol.2000;19(1):47–51.
- ,,.Effective risk stratification of surgical and nonsurgical patients for venous thromboembolic disease.Semin Hematol.2001;38(2 suppl 5):12–19.
- .Thrombosis risk assessment as a guide to quality patient care.Dis Mon.2005;51(2–3):70–78.
- ,.Designing and implementing effective venous thromboembolism prevention protocols: lessons from collaborative efforts.J Thromb Thrombolysis.2010;29(2):159–166.
- ,,.Emergency medicine practitioner knowledge and use of decision rules for the evaluation of patients with suspected pulmonary embolism: variations by practice setting and training level.Acad Emerg Med.2007;14(1):53–57.
- , , , , , , , .Optimizing prevention of hospital‐acquired (HA) venous thromboembolism (VTE): prospective validation of a VTE risk assessment model (RAM).J Hosp Med.2010;5(1):10–18..
- , , , et al.Development and testing of a DVT risk assessment tool: providing evidence of validity and reliability.Worldviews Evid Based Nurs.2007;4(1):14–20.
- , , , et al.A novel educational strategy to enhance internal medicine residents' familial colorectal cancer knowledge and risk assessment skills.Am J Gastroenterol.2005;100(3):677–684.
- ,, , , , .Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?JAMA.1999;282(9):867–874.
- , , , et al.Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality.JAMA.2000;283(13):1715–1722.
- .Psychometric Theory.New Delhi:Tate McGraw‐Hill;1981.
- ,.Intraclass correlations: uses in assessing rater reliability.Psychol Bull.1979;86(2):420–428.
- ,,.Innovative approaches to increase deep vein thrombosis prophylaxis rate resulting in a decrease in hospital‐acquired deep vein thrombosis at a tertiary‐care teaching hospital.J Hosp Med.2008;3(2):148–155.
- ,,.Improved use of thromboprophylaxis for deep vein thrombosis following an educational intervention.J Hosp Med.2006;1(6):331–338.
- , , , et al.Minimizing errors of omission: behavioural reenforcement of heparin to avert venous emboli: the BEHAVE study.Crit Care Med.2006;34(3):694–699.
- , , , et al.Electronic alerts to prevent venous thromboembolism among hospitalized patients.N Engl J Med.2005;352(10):969–977.
- , , , .Getting a validated guideline into local practice: implementation and audit of the SIGN guideline on the prevention of deep vein thrombosis in a district general hospital.Scott Med J.1998;43(1):23–25.
- ,,. Improving the Reliability of Health Care. IHI Innovation Series white paper 2004. Available at: www.IHI.org/IHI/Results/WhitePapers/Improving theReliabilityof HealthCare.htm. Accessed October 4,2010.
- , , , et al.Thrombosis prophylaxis in medical patients: a retrospective review of clinical practice patterns.Haematologica.2002;87(7):746–750; discussion 250.
- ,.Genetic epidemiology: systemic lupus erythematosus.Arthritis Res.2001;3(6):331–336.
- ,,.Missed opportunities for prevention of venous thromboembolism: an evaluation of the use of thromboprophylaxis guidelines.Chest.2001;120(6):1964–1971.
- , , , , .Underuse of venous thromboembolism prophylaxis for general surgery patients: physician practices in the community hospital setting.Arch Intern Med.1998;158(17):1909–1912.
- ,,.Venous thromboembolism prophylaxis used by consultant general surgeons in Scotland.J R Coll Surg Edinb.2001;46(6):329–333.
- ,.Ordering errors by first‐year residents: evidence of learning from mistakes.Mo Med.2004;101(2):128–131.
- ,.Prevention of medication errors: teaching and training.Br J Clin Pharmacol.2009;67(6):656–661.
- ,,.Reducing medication errors in a surgical residency training program.Am Surg.2004;70(5):467–471.
- , , , et al.Why don't physicians follow clinical practice guidelines? A framework for improvement.JAMA.1999;282(15):1458–1465.
- , , , , .Predicting computerized physician order entry system adoption in US hospitals: can the federal mandate be met?Int J Med Inform.2008;77(8):539–545.
- ,.Implementation of computerized physician order entry in seven countries.Health Aff (Millwood).2009;28(2):404–414.
Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7
Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16
Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.
Methods
Development of the Risk Assessment Tool
A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.
Construction of Clinical Vignettes
Approval was obtained by the Pennsylvania State University Institutional Review Board. Since previous research demonstrates the utility of clinical vignettes to study the effectiveness of guideline application and decision making, we used a series of 21 randomly selected and de‐identified clinical vignettes to portray a range of real‐world patient admission scenarios.2022 We identified individuals who had been admitted to the Hershey Medical Center using data from the inpatient electronic health record and applying the following inclusion criteria: age >17 years, and admission to a general medical service from the Emergency Department during a 14‐day period in 2008. Since more than 80% of patients admitted to our medical service are admitted through the Emergency Department and residents place all of the admission orders, our goal was to use vignettes that were typical of patients they commonly admit. We attached a paper form of our institution's VTE prophylaxis strategy (Figure 1) to each vignette.
Data Collection
A 1‐hour noon conference titled VTE Workshop was conducted by one of the authors (M.J.B.) during the first quarter of the 2008 academic year. We asked the medical residents to apply the VTE prophylaxis protocol to 21 vignettes during this session. In order to determine the appropriate time allotted to complete the vignettes, each case was completed by M.J.B. and three medical residents (one intern, one second year, and one third year) prior to conducting the VTE workshop. Based on these data, we determined that the median time to complete each vignette was 2 minutes and 15 seconds (range 30 seconds to 7 minutes). Therefore, we assumed that the 21 vignettes could be completed within 1 hour. At the beginning of the conference, the residents were provided with 10 minutes of verbal instruction about how to apply the VTE risk assessment tool. They were instructed to provide a total risk score (070 points) and, based on the total score, to classify each patient as low, moderate, or high risk for VTE. Following the risk assignment, they were instructed to document any absolute and relative contraindications. Finally, they were asked to select the most appropriate VTE prophylaxis plan according to the recommendations of the protocol. Vignettes were considered complete if they had an assessment and plan for >75% of the cases.
Prior to conducting this study, there had been no formal orientation regarding use of the VTE risk assessment tool or incorporation of it into our institution's computerized order entry system. Average attendance for the noon conference is between 20 and 30 house staff, approximately one‐third of the entire residency. Medical students were excluded from the study. All respondents voluntarily and anonymously performed the assessments, and indicated on the front of their vignette packet their level of training as PGY‐1, ‐2, or ‐3. The sessions were overseen by one of the authors to ensure that no communication occurred among the residents.
Data Analysis
We constructed a database with five variables collected from each resident's VTE risk assessment form: 1) a total risk score, 2) a risk classification (low, medium, or high), 3) the number and type of absolute contraindications to pharmacological prophylaxis, 4) the number and type of relative contraindications to pharmacological prophylaxis, and 5) a VTE prophylaxis plan. The lead author also performed these assessments of the 21 vignettes 1 month prior to the resident session. In power calculations performed prior to the session, we determined that the study would need at least 300 observations in order to calculate inter‐rater reliability.23 With the estimation that between 20 and 30 residents would attend, we determined that 21 vignettes would exceed the minimum required observations to allow for an accurate calculation of inter‐rater reliability.
The total risk score was treated as a continuous variable for which the intra‐class correlation (ICC) was calculated. The ICC is used to assess the consistency, or conformity, of measurements made by multiple observers measuring the same quantity.24 Risk stratification, presence of absolute and relative contraindications, and VTE plan were treated as categorical variables. For these, we used Cohen's kappa to assess variability in resident ratings. The kappa score has been used in other studies to determine inter‐rater reliability using similar VTE risk assessment tools.18, 19 Finally, adherence to our hospital's protocol was determined by comparing the residents' VTE plans with the lead author's VTE plans for each of the 21 vignettes. We used SAS 9.1.3 for all statistical analyses (SAS Institute, Cary, NC).
Results
Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).
| Risk Stratification | Resident no./total (%) | Attending no./total (%) |
|---|---|---|
| Low | 130/479 (27) | 3/21 (14) |
| Moderate | 183/479 (38) | 7/21 (33) |
| High | 166/479 (34) | 11/21 (52) |
Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.
| Attending Classification | Total | Appropriate Risk Assessment No. (%) | SCDs Only No. (%) | Heparin Only | Both Heparin and SCDs | Ambulation |
|---|---|---|---|---|---|---|
| ||||||
| Low risk | 115 | 86 (75) | 11 (10) | 7 (6) | 0 | 93 (84)a |
| Moderate risk | 138 | 85 (62) | 28 (21)b | 44 (33)a | 16 (12) | 47 (35) |
| High risk | 230 | 138 (60) | 39 (18)b | 69 (31) | 88 (40)a | 27 (12) |
| Total | 483 | 309 (64) | 78 (16) | 120 (26) | 104 (22) | 167 (36) |
The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).
Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).
| Risk Score | Stratification | Absolute Contraindication | Relative Contraindication | VTE Plan | ||
|---|---|---|---|---|---|---|
| ||||||
| Aggregate | ICC | 0.66 | ||||
| Kappa | 0.51 | 0.29 | 0.23 | 0.28 | ||
| Intern | ICC | 0.63 | ||||
| Kappa | 0.47 | NA | NA | 0.23 | ||
| Seniora | ICC | 0.73 | ||||
| Kappa | 0.61 | NA | NA | 0.35 | ||
Discussion
We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.
Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.
As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035
When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135
Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.
A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.
Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.
In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.
Acknowledgements
The authors thank Lisabeth V. Scalzi, MD, MS, Lora Moyer, Kevin McKenna MD, Hammid Al‐Mondhiry, Lucille Anderson, MD, Kathleen Williams, Kevin Larraway, Cynthia Chuang, MD, MS, the residents of the Internal Medicine and Combined Medicine/Pediatrics residencies, and the division of General Internal Medicine at Hershey Medical Center.
Venous thromboembolic events (VTE) are a significant cause of mortality in hospitalized medical and surgical patients.1, 2 The incidence of hospital‐acquired deep vein thrombosis (DVT) is 1040% among medical or general surgical patients in the absence of VTE prophylaxis.3 Approximately 5075% of these cases are preventable with appropriate prophylaxis.3, 4 To reduce hospital‐acquired VTE, the American College of Chest Physicians (ACCP) has published VTE prevention guidelines regularly since 1986. The latest version of the ACCP guidelines recommends that every hospital have an institution‐wide policy that encourages use of VTE prevention strategies. Nevertheless, current evidence demonstrates that VTE prophylaxis remains underutilized in at‐risk patients.5, 6 For example, the ENDORSE study showed that only 39% of at‐risk medical patients received appropriate VTE prophylaxis.6 A more recent study estimated that 58% of hospital‐acquired VTEs were preventable with appropriate prophylaxis utilization.7
Such data demonstrate that a quality gap exists between VTE prophylaxis guideline recommendations and actual practice. Such a gap highlights the need to identify barriers to appropriate implementation of systems‐based strategies aimed at preventing VTE. A presumed barrier for adherence to any VTE protocol is the complexity involved in performing individualized risk assessments.810 All VTE prophylaxis strategies require the clinician to risk‐stratify each patient, identify contraindications to a prophylaxis strategy, and select an accepted strategy. While many VTE risk assessment protocols exist, they tend to fall into two categories: 1) a point‐based system, and 2) a simplified tiered system. Point‐based clinical prediction rules have been advocated by Caprini and others.1115 Such approaches require the clinician to assign points during the identification of VTE risk factors. The clinician must add the points to determine a patient's cumulative VTE risk and use the points to classify that risk as low, moderate, or high. Such point‐based systems are generally considered complex and may underestimate VTE risk, potentially leading to underutilization of prophylaxis strategies.16
Studies have demonstrated that complexity introduces variation into the decision‐making process.17 As a result, both the ACCP and SHM advocate for simplifying the VTE risk assessment process.4, 18 To date, several studies have demonstrated that attending physicians and nurses can reliably apply a VTE risk assessment tool, but none that measure how reliably residents can perform this task when using a point‐based tool.18, 19 For academic medical centers, information about the reliability of such tools is especially important, since they will often be applied by physicians‐in‐training, who have limited knowledge and experience with VTE guidelines and risk assessment. The goal of our study, therefore, was to use clinical vignettes to determine the reliability and protocol adherence of medical residents' application of an adapted point‐based VTE risk assessment tool, independent of other interventions.
Methods
Development of the Risk Assessment Tool
A multispecialty team adapted existing individualized VTE risk assessment tools based on one developed by Caprini.11 The VTE tool (Fig. 1) was designed to assist residents in making two essential determinations prior to ordering a prophylaxis plan. The first determination was the calculation of a total risk score (070 points). This score was determined by identifying and assigning a point value to all medical and surgical risk factors, and summing the points into 3 categories: low (01 point), moderate (24 points), and high (>4 points) VTE risk. The second determination was to identify any contraindications to pharmacological prophylaxis. Like other nonvalidated tools, our tool divided contraindications into absolute or relative. After making these two determinations, residents were encouraged to order 1 of 6 VTE prophylaxis plans. These plans were intended to balance VTE risk against risk factors for bleeding due to prophylaxis.
Construction of Clinical Vignettes
Approval was obtained by the Pennsylvania State University Institutional Review Board. Since previous research demonstrates the utility of clinical vignettes to study the effectiveness of guideline application and decision making, we used a series of 21 randomly selected and de‐identified clinical vignettes to portray a range of real‐world patient admission scenarios.2022 We identified individuals who had been admitted to the Hershey Medical Center using data from the inpatient electronic health record and applying the following inclusion criteria: age >17 years, and admission to a general medical service from the Emergency Department during a 14‐day period in 2008. Since more than 80% of patients admitted to our medical service are admitted through the Emergency Department and residents place all of the admission orders, our goal was to use vignettes that were typical of patients they commonly admit. We attached a paper form of our institution's VTE prophylaxis strategy (Figure 1) to each vignette.
Data Collection
A 1‐hour noon conference titled VTE Workshop was conducted by one of the authors (M.J.B.) during the first quarter of the 2008 academic year. We asked the medical residents to apply the VTE prophylaxis protocol to 21 vignettes during this session. In order to determine the appropriate time allotted to complete the vignettes, each case was completed by M.J.B. and three medical residents (one intern, one second year, and one third year) prior to conducting the VTE workshop. Based on these data, we determined that the median time to complete each vignette was 2 minutes and 15 seconds (range 30 seconds to 7 minutes). Therefore, we assumed that the 21 vignettes could be completed within 1 hour. At the beginning of the conference, the residents were provided with 10 minutes of verbal instruction about how to apply the VTE risk assessment tool. They were instructed to provide a total risk score (070 points) and, based on the total score, to classify each patient as low, moderate, or high risk for VTE. Following the risk assignment, they were instructed to document any absolute and relative contraindications. Finally, they were asked to select the most appropriate VTE prophylaxis plan according to the recommendations of the protocol. Vignettes were considered complete if they had an assessment and plan for >75% of the cases.
Prior to conducting this study, there had been no formal orientation regarding use of the VTE risk assessment tool or incorporation of it into our institution's computerized order entry system. Average attendance for the noon conference is between 20 and 30 house staff, approximately one‐third of the entire residency. Medical students were excluded from the study. All respondents voluntarily and anonymously performed the assessments, and indicated on the front of their vignette packet their level of training as PGY‐1, ‐2, or ‐3. The sessions were overseen by one of the authors to ensure that no communication occurred among the residents.
Data Analysis
We constructed a database with five variables collected from each resident's VTE risk assessment form: 1) a total risk score, 2) a risk classification (low, medium, or high), 3) the number and type of absolute contraindications to pharmacological prophylaxis, 4) the number and type of relative contraindications to pharmacological prophylaxis, and 5) a VTE prophylaxis plan. The lead author also performed these assessments of the 21 vignettes 1 month prior to the resident session. In power calculations performed prior to the session, we determined that the study would need at least 300 observations in order to calculate inter‐rater reliability.23 With the estimation that between 20 and 30 residents would attend, we determined that 21 vignettes would exceed the minimum required observations to allow for an accurate calculation of inter‐rater reliability.
The total risk score was treated as a continuous variable for which the intra‐class correlation (ICC) was calculated. The ICC is used to assess the consistency, or conformity, of measurements made by multiple observers measuring the same quantity.24 Risk stratification, presence of absolute and relative contraindications, and VTE plan were treated as categorical variables. For these, we used Cohen's kappa to assess variability in resident ratings. The kappa score has been used in other studies to determine inter‐rater reliability using similar VTE risk assessment tools.18, 19 Finally, adherence to our hospital's protocol was determined by comparing the residents' VTE plans with the lead author's VTE plans for each of the 21 vignettes. We used SAS 9.1.3 for all statistical analyses (SAS Institute, Cary, NC).
Results
Twenty‐six medical residents attended the conference. Three residents left without submitting their assessments and were excluded from the analysis. Of the 23 residents included in the analysis, 15 (65%) were interns, 5 (22%) second‐year residents, and 3 (13%) third‐year residents. A maximum of 483 observations (21 clinical vignettes and 23 residents) was possible. Six (1%) risk stratifications were missing, and 14 (3%) VTE prophylaxis plans were missing. Therefore, out of a possible 483 paired assessments and plans, complete data existed for 95% (469) of the observations. Residents risk‐stratified the vignettes as low risk for 27% of cases, moderate risk for 38%, and high risk 34%. These differed from those of the lead author, who stratified proportionately more vignettes as high risk (Table 1).
| Risk Stratification | Resident no./total (%) | Attending no./total (%) |
|---|---|---|
| Low | 130/479 (27) | 3/21 (14) |
| Moderate | 183/479 (38) | 7/21 (33) |
| High | 166/479 (34) | 11/21 (52) |
Of those vignette patients stratified as high risk, 77% (128/166) received some form of prophylaxis. Of those stratified as moderate risk, 66% (121/183) received some form of prophylaxis. Finally, of those stratified as low risk, 15% (20/130) received some form of prophylaxis. To explore the impact of the disparity in risk assessments between residents and attending physicians, we used the lead author's assessments as the standard for comparison, and determined that only 64% (309/479) of the observations were risk‐stratified correctly. To emphasize further the potential negative impact of these misclassifications, we determined that appropriate plans would have occurred only 47% of the time. Analysis of these data via risk category showed that low‐risk patients received appropriate prophylaxis 84% of the time. However, protocol adherence for moderate and high‐risk patients occurred only 33 and 40% of the time, respectively (Table 2). Making the assumption that those vignette patients at moderate and high risk who only received mechanical prophylaxis had appropriate contraindications to heparin prophylaxis, protocol adherence remained low at 54 and 58%, respectively.
| Attending Classification | Total | Appropriate Risk Assessment No. (%) | SCDs Only No. (%) | Heparin Only | Both Heparin and SCDs | Ambulation |
|---|---|---|---|---|---|---|
| ||||||
| Low risk | 115 | 86 (75) | 11 (10) | 7 (6) | 0 | 93 (84)a |
| Moderate risk | 138 | 85 (62) | 28 (21)b | 44 (33)a | 16 (12) | 47 (35) |
| High risk | 230 | 138 (60) | 39 (18)b | 69 (31) | 88 (40)a | 27 (12) |
| Total | 483 | 309 (64) | 78 (16) | 120 (26) | 104 (22) | 167 (36) |
The ICC for the total risk score was 0.66, and the kappa coefficient for risk stratification was 0.51 (95% CI 0.50, 0.53), both of which represent moderate agreement. Absolute and relative contraindications were identified 12% (57/483) and 13% (61/483) of the time, respectively. The kappa scores for absolute and relative contraindications were 0.29 and 0.23, respectively. The kappa score for the VTE plan was 0.28 and represents only fair agreement (Table 2).
Subgroup analysis of the 15 intern participants for ICC for the risk score was 0.63. The kappa scores for risk stratification and VTE plan were 0.47 and 0.23, respectively. The kappa scores for senior residents represent aggregate data of 168 observations of second‐ and third‐year residents. For senior residents, the kappa scores for risk stratification and VTE plan were 0.61 and 0.35, respectively (Table 3).
| Risk Score | Stratification | Absolute Contraindication | Relative Contraindication | VTE Plan | ||
|---|---|---|---|---|---|---|
| ||||||
| Aggregate | ICC | 0.66 | ||||
| Kappa | 0.51 | 0.29 | 0.23 | 0.28 | ||
| Intern | ICC | 0.63 | ||||
| Kappa | 0.47 | NA | NA | 0.23 | ||
| Seniora | ICC | 0.73 | ||||
| Kappa | 0.61 | NA | NA | 0.35 | ||
Discussion
We performed this study to determine how reliably our medical residents could apply a point‐based VTE risk assessment tool, similar to those published previously.11 We observed that early in the academic year, our residents were not able to use this tool reliably. While our study does not evaluate the effects of audit and feedback, reminder alerts, or educational interventions, an important first step toward quality improvement in VTE prophylaxis is to reduce variability in risk assessment and decision making. In this endeavor, our results differ markedly from those in the literature. For instance, one study used 3 trained nurses to employ a similar risk assessment tool, and found an ICC of 0.98 for overall assessments of VTE risk, but did not report protocol adherence.19 Another study found inter‐rater reliability to be high among 5 physician observers (kappa scores of 0.81 and 0.90 for risk stratification and VTE plan, respectively).18 These two studies evaluated the performance of experienced evaluators who employed different and simpler VTE risk assessment tools. Our study determined that the inter‐rater reliability of risk assessment and VTE plan between residents using a point‐based VTE risk assessment tool was significantly lower, at 0.51 and 0.28, respectively. There was marked disparity between the lead author's and residents' risk assessments of those deemed to be at low and high risk (Table 1). While both determined approximately one‐third of the patient vignettes to be at moderate risk, the residents misclassified those at high risk in comparison with the author's assessments. This underestimation of VTE risk could lead to profound underprophylaxis in at‐risk patients. To the extent that our findings represent those in other teaching hospitals, such errors could hinder VTE quality improvement efforts in such institutions.
Previous studies that successfully improved VTE prophylaxis rates coupled a risk assessment tool with provider education as well as audit‐and‐feedback interventions.25, 26 In one study, provider education occurred on the first day of every month with an orientation to the hospital's recommended VTE prevention strategies.26 Another study sought to improve the rates of VTE prophylaxis in medical intensive care (MICU) patients without performing individualized risk assessment.27 Using only weekly graphic feedback and verbal reminders to the medical team, it showed an improvement in VTE prophylaxis for 1 year. A third study improved VTE prophylaxis adherence and reduced VTE at 90 days using only reminder alerts.28 Interestingly, several studies reduced the incidence of VTE without employing any patient risk stratification.29 These studies suggest that improvement in VTE prophylaxis rates could have occurred as result of audit‐and‐feedback or reminder systems and perhaps independent of the reliable application of a risk assessment tool.29 The studies that used risk assessment tools with layered interventions make it difficult to interpret whether the tool or the layered interventions were responsible for the improvement in VTE outcomes. Ours is the first study to evaluate how reliably residents can apply a tool independent of other interventions. With only fair to moderate resident agreement in patient risk assessment and VTE plan, our study suggests that the complexity of a point‐based risk assessment protocol (as opposed to a simplified three‐tiered approach) may affect resident prescriptive behavior.
As a result, our study corroborates two things: first, in medical centers that rely on residents to perform VTE prevention using individualized risk assessment, a multilayered approach for VTE prevention must occur. Second, a passively disseminated VTE protocol in the form of a pocket card will most likely not create a sustained improvement in VTE prophylaxis rates or reduce VTE.3035
When addressing certain aspects of quality improvement and safety, teaching hospitals must recognize that their efforts largely rely on resident performance. The 2009 National Resident Matching Program data indicate that there are 22,427 intern positions available in the United States. Often it is the resident's responsibility to perform risk assessments and provide prophylaxis, possibly using a tool that is too complex to apply reliably. Several studies have determined that 65% of medical errors were committed by interns and that 35‐44% of those errors resulted from knowledge deficits.3638 In order to best improve adherence to clinical guidelines, strategies that result in changing physician behavior need to be implemented, and can include but are not limited to the ones found in Figure 2.39 Ideally, teaching centers with computerized order entry should embed the risk assessment process as part of an admission/transfer order set, with a reminder alert. The alert would be activated when at‐risk patients do not receive appropriate prophylaxis. Most alert systems require hospitals to have computerized order entry, which has achieved only 20% market penetration in US hospitals.40, 41 Therefore, some hospitals employ, or intend to employ, passively disseminated risk assessment tools in the form of pocket cards or preprinted forms. These methods are estimated to improve prophylaxis by only 50% and are therefore not considered to be highly reliable strategies.3135
Our study demonstrates only fair to moderate reliability of a point‐based VTE risk assessment tool when used by residents independent of other strategies. It also suggests that residents underestimate those at high risk. In addition, our residents' protocol adherence was suboptimal and would have resulted in appropriate prophylaxis approximately 50% of the time in patients at moderate or high VTE risk. Therefore, when risk assessment tools such as ours are used, it is imperative that frequent education be combined with real‐time patient identification strategies as well as audit and feedback, a process called measure‐vention.13, 14 This is especially true when the risk assessment process is not linked to a reminder system as part of computer‐assisted order entry protocols.
A limitation of our study was the lack of a control group. Since all the residents in attendance received the same clinical vignettes, it would have been of interest to see how the risk assessment tool performed compared with residents who did not have access to the tool. However, based on average noon conference attendance, it would have been difficult to achieve an adequate number of observations to calculate credible ICC and kappa scores. Other limitations include the high number of interns who completed the vignettes compared with senior residents, and the lack of additional attending reviewers to score the vignettes prior to the session. Ideally, in determining the accuracy of protocol adherence, we would have compared residents' determinations with those of several experts who had used an adjudication process in the event of disagreement. In our ongoing work, we are collecting data from a representative sample of attending physicians at our hospital to compare their assessments both with each other and with those of the residents.
Another issue in our design was that the study presented only a limited amount of medical information in the vignettes. In actual clinical circumstances, the amount of historical information is greater and more complicated. One could argue that the artificiality of clinical vignettes is not an accurate representation of resident performance when ordering VTE prophylaxis. However, this approach limits case‐mix variation, so residents should have been able to reach similar conclusions with the information given. Thus the limited information should have maximized residents' intentions to prescribe VTE prophylaxis, and kappa scores would likely be lower in real clinical settings. Finally, our kappa scores were calculated based on aggregate data of interns and residents; however, interns comprised almost two‐thirds of the resident participants. As reported in the results section, intern inter‐rater reliability was slightly lower compared with the senior resident subgroup, suggesting that the variability may be a result of less clinical experience of the interns. However, the study was not powered to assess differences in kappa scores for level of training.
In conclusion, we determined the inter‐rater reliability of an individualized, point‐based VTE risk assessment tool when used by medical residents unfamiliar with its use. Our study showed that under conditions of minimal education, a point‐based VTE assessment tool achieves only fair to moderate reliability. It also suggests that as a stand‐alone tool without a reminder alert, adherence to VTE prevention guidelines is suboptimal and might result in underprophylaxis of hospitalized medical patients at moderate or high VTE risk. In fact, with appropriate prophylaxis, rates were maximally estimated to be 55% (Table 2). Because of the high percentage of interns in the study, these results approximate intern application of a VTE prevention protocol independent of other interventions. Comparing reliability data from our study with those of others raises the question of whether the observed differences in kappa score are because other studies used highly trained observers or because their protocols were less complex. However, a recent study validated a simpler method of VTE risk grouping that performs well regardless of clinical experience.20 Future studies are needed to determine whether there is improved resident inter‐rater reliability using a point‐based risk assessment tool that is embedded into a computerized order entry system with electronic reminder alerts. Finally, in actual clinical settings, the question remains of whether kappa scores correlate with protocol adherence, prophylaxis rates, and VTE reduction when using point‐based tools. If not, then the use of simplified risk‐stratification tools and VTE measure‐vention strategies should be implemented.
Acknowledgements
The authors thank Lisabeth V. Scalzi, MD, MS, Lora Moyer, Kevin McKenna MD, Hammid Al‐Mondhiry, Lucille Anderson, MD, Kathleen Williams, Kevin Larraway, Cynthia Chuang, MD, MS, the residents of the Internal Medicine and Combined Medicine/Pediatrics residencies, and the division of General Internal Medicine at Hershey Medical Center.
- ,,.Autopsy‐verified pulmonary embolism in a surgical department: analysis of the period from 1951 to 1988.Br J Surg.1991;78(7):849–852.
- , , , , et al.Fatal pulmonary embolism in hospitalised patients: a necropsy review.J Clin Pathol.2004;57(12):1254–1257.
- , , , et al.Prevention of venous thromboembolism: American College of Chest Physicians Evidence‐Based Clinical Practice Guidelines (8th edition).Chest.2008;133(6 suppl):381S–453S.
- ,.Coronary artery vasculitis and myocardial infarction with systemic lupus erythematosus.NY State J Med.1974;74(5):873–874.
- , , , et al.Venous thromboembolism prophylaxis in acutely ill hospitalized medical patients: findings from the International Medical Prevention Registry on Venous Thromboembolism.Chest.2007;132(3):936–945.
- , , , et al.Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross‐sectional study.Lancet.2008;371(9610):387–394.
- , , , .Venous thromboembolic events in hospitalised medical patients.Thromb Haemost.2009;102(3):505–510.
- ,,.Compliance with recommended prophylaxis for venous thromboembolism: improving the use and rate of uptake of clinical practice guidelines.J Thromb Haemost.2004;2(2):221–227.
- , , , et al.A systematic review of strategies to improve prophylaxis for venous thromboembolism in hospitals.Ann Surg.2005;241(3):397–415.
- ,.Prevention of venous thromboembolism: consensus, controversies, and challenges.Hematol Am Soc Hematol Educ Program.2009:286–292.
- ,,.International perspective on venous thromboembolism prophylaxis in surgery.Semin Thromb Hemost.1991;17(4):322–325.
- , , , et al.Clinical assessment of venous thromboembolic risk in surgical patients.Semin Thromb Hemost.1991;17(suppl 3):304–312.
- , et al.Risk factor assessment in the management of patients with suspected deep venous thrombosis.Int Angiol.2000;19(1):47–51.
- ,,.Effective risk stratification of surgical and nonsurgical patients for venous thromboembolic disease.Semin Hematol.2001;38(2 suppl 5):12–19.
- .Thrombosis risk assessment as a guide to quality patient care.Dis Mon.2005;51(2–3):70–78.
- ,.Designing and implementing effective venous thromboembolism prevention protocols: lessons from collaborative efforts.J Thromb Thrombolysis.2010;29(2):159–166.
- ,,.Emergency medicine practitioner knowledge and use of decision rules for the evaluation of patients with suspected pulmonary embolism: variations by practice setting and training level.Acad Emerg Med.2007;14(1):53–57.
- , , , , , , , .Optimizing prevention of hospital‐acquired (HA) venous thromboembolism (VTE): prospective validation of a VTE risk assessment model (RAM).J Hosp Med.2010;5(1):10–18..
- , , , et al.Development and testing of a DVT risk assessment tool: providing evidence of validity and reliability.Worldviews Evid Based Nurs.2007;4(1):14–20.
- , , , et al.A novel educational strategy to enhance internal medicine residents' familial colorectal cancer knowledge and risk assessment skills.Am J Gastroenterol.2005;100(3):677–684.
- ,, , , , .Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?JAMA.1999;282(9):867–874.
- , , , et al.Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality.JAMA.2000;283(13):1715–1722.
- .Psychometric Theory.New Delhi:Tate McGraw‐Hill;1981.
- ,.Intraclass correlations: uses in assessing rater reliability.Psychol Bull.1979;86(2):420–428.
- ,,.Innovative approaches to increase deep vein thrombosis prophylaxis rate resulting in a decrease in hospital‐acquired deep vein thrombosis at a tertiary‐care teaching hospital.J Hosp Med.2008;3(2):148–155.
- ,,.Improved use of thromboprophylaxis for deep vein thrombosis following an educational intervention.J Hosp Med.2006;1(6):331–338.
- , , , et al.Minimizing errors of omission: behavioural reenforcement of heparin to avert venous emboli: the BEHAVE study.Crit Care Med.2006;34(3):694–699.
- , , , et al.Electronic alerts to prevent venous thromboembolism among hospitalized patients.N Engl J Med.2005;352(10):969–977.
- , , , .Getting a validated guideline into local practice: implementation and audit of the SIGN guideline on the prevention of deep vein thrombosis in a district general hospital.Scott Med J.1998;43(1):23–25.
- ,,. Improving the Reliability of Health Care. IHI Innovation Series white paper 2004. Available at: www.IHI.org/IHI/Results/WhitePapers/Improving theReliabilityof HealthCare.htm. Accessed October 4,2010.
- , , , et al.Thrombosis prophylaxis in medical patients: a retrospective review of clinical practice patterns.Haematologica.2002;87(7):746–750; discussion 250.
- ,.Genetic epidemiology: systemic lupus erythematosus.Arthritis Res.2001;3(6):331–336.
- ,,.Missed opportunities for prevention of venous thromboembolism: an evaluation of the use of thromboprophylaxis guidelines.Chest.2001;120(6):1964–1971.
- , , , , .Underuse of venous thromboembolism prophylaxis for general surgery patients: physician practices in the community hospital setting.Arch Intern Med.1998;158(17):1909–1912.
- ,,.Venous thromboembolism prophylaxis used by consultant general surgeons in Scotland.J R Coll Surg Edinb.2001;46(6):329–333.
- ,.Ordering errors by first‐year residents: evidence of learning from mistakes.Mo Med.2004;101(2):128–131.
- ,.Prevention of medication errors: teaching and training.Br J Clin Pharmacol.2009;67(6):656–661.
- ,,.Reducing medication errors in a surgical residency training program.Am Surg.2004;70(5):467–471.
- , , , et al.Why don't physicians follow clinical practice guidelines? A framework for improvement.JAMA.1999;282(15):1458–1465.
- , , , , .Predicting computerized physician order entry system adoption in US hospitals: can the federal mandate be met?Int J Med Inform.2008;77(8):539–545.
- ,.Implementation of computerized physician order entry in seven countries.Health Aff (Millwood).2009;28(2):404–414.
- ,,.Autopsy‐verified pulmonary embolism in a surgical department: analysis of the period from 1951 to 1988.Br J Surg.1991;78(7):849–852.
- , , , , et al.Fatal pulmonary embolism in hospitalised patients: a necropsy review.J Clin Pathol.2004;57(12):1254–1257.
- , , , et al.Prevention of venous thromboembolism: American College of Chest Physicians Evidence‐Based Clinical Practice Guidelines (8th edition).Chest.2008;133(6 suppl):381S–453S.
- ,.Coronary artery vasculitis and myocardial infarction with systemic lupus erythematosus.NY State J Med.1974;74(5):873–874.
- , , , et al.Venous thromboembolism prophylaxis in acutely ill hospitalized medical patients: findings from the International Medical Prevention Registry on Venous Thromboembolism.Chest.2007;132(3):936–945.
- , , , et al.Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross‐sectional study.Lancet.2008;371(9610):387–394.
- , , , .Venous thromboembolic events in hospitalised medical patients.Thromb Haemost.2009;102(3):505–510.
- ,,.Compliance with recommended prophylaxis for venous thromboembolism: improving the use and rate of uptake of clinical practice guidelines.J Thromb Haemost.2004;2(2):221–227.
- , , , et al.A systematic review of strategies to improve prophylaxis for venous thromboembolism in hospitals.Ann Surg.2005;241(3):397–415.
- ,.Prevention of venous thromboembolism: consensus, controversies, and challenges.Hematol Am Soc Hematol Educ Program.2009:286–292.
- ,,.International perspective on venous thromboembolism prophylaxis in surgery.Semin Thromb Hemost.1991;17(4):322–325.
- , , , et al.Clinical assessment of venous thromboembolic risk in surgical patients.Semin Thromb Hemost.1991;17(suppl 3):304–312.
- , et al.Risk factor assessment in the management of patients with suspected deep venous thrombosis.Int Angiol.2000;19(1):47–51.
- ,,.Effective risk stratification of surgical and nonsurgical patients for venous thromboembolic disease.Semin Hematol.2001;38(2 suppl 5):12–19.
- .Thrombosis risk assessment as a guide to quality patient care.Dis Mon.2005;51(2–3):70–78.
- ,.Designing and implementing effective venous thromboembolism prevention protocols: lessons from collaborative efforts.J Thromb Thrombolysis.2010;29(2):159–166.
- ,,.Emergency medicine practitioner knowledge and use of decision rules for the evaluation of patients with suspected pulmonary embolism: variations by practice setting and training level.Acad Emerg Med.2007;14(1):53–57.
- , , , , , , , .Optimizing prevention of hospital‐acquired (HA) venous thromboembolism (VTE): prospective validation of a VTE risk assessment model (RAM).J Hosp Med.2010;5(1):10–18..
- , , , et al.Development and testing of a DVT risk assessment tool: providing evidence of validity and reliability.Worldviews Evid Based Nurs.2007;4(1):14–20.
- , , , et al.A novel educational strategy to enhance internal medicine residents' familial colorectal cancer knowledge and risk assessment skills.Am J Gastroenterol.2005;100(3):677–684.
- ,, , , , .Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?JAMA.1999;282(9):867–874.
- , , , et al.Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality.JAMA.2000;283(13):1715–1722.
- .Psychometric Theory.New Delhi:Tate McGraw‐Hill;1981.
- ,.Intraclass correlations: uses in assessing rater reliability.Psychol Bull.1979;86(2):420–428.
- ,,.Innovative approaches to increase deep vein thrombosis prophylaxis rate resulting in a decrease in hospital‐acquired deep vein thrombosis at a tertiary‐care teaching hospital.J Hosp Med.2008;3(2):148–155.
- ,,.Improved use of thromboprophylaxis for deep vein thrombosis following an educational intervention.J Hosp Med.2006;1(6):331–338.
- , , , et al.Minimizing errors of omission: behavioural reenforcement of heparin to avert venous emboli: the BEHAVE study.Crit Care Med.2006;34(3):694–699.
- , , , et al.Electronic alerts to prevent venous thromboembolism among hospitalized patients.N Engl J Med.2005;352(10):969–977.
- , , , .Getting a validated guideline into local practice: implementation and audit of the SIGN guideline on the prevention of deep vein thrombosis in a district general hospital.Scott Med J.1998;43(1):23–25.
- ,,. Improving the Reliability of Health Care. IHI Innovation Series white paper 2004. Available at: www.IHI.org/IHI/Results/WhitePapers/Improving theReliabilityof HealthCare.htm. Accessed October 4,2010.
- , , , et al.Thrombosis prophylaxis in medical patients: a retrospective review of clinical practice patterns.Haematologica.2002;87(7):746–750; discussion 250.
- ,.Genetic epidemiology: systemic lupus erythematosus.Arthritis Res.2001;3(6):331–336.
- ,,.Missed opportunities for prevention of venous thromboembolism: an evaluation of the use of thromboprophylaxis guidelines.Chest.2001;120(6):1964–1971.
- , , , , .Underuse of venous thromboembolism prophylaxis for general surgery patients: physician practices in the community hospital setting.Arch Intern Med.1998;158(17):1909–1912.
- ,,.Venous thromboembolism prophylaxis used by consultant general surgeons in Scotland.J R Coll Surg Edinb.2001;46(6):329–333.
- ,.Ordering errors by first‐year residents: evidence of learning from mistakes.Mo Med.2004;101(2):128–131.
- ,.Prevention of medication errors: teaching and training.Br J Clin Pharmacol.2009;67(6):656–661.
- ,,.Reducing medication errors in a surgical residency training program.Am Surg.2004;70(5):467–471.
- , , , et al.Why don't physicians follow clinical practice guidelines? A framework for improvement.JAMA.1999;282(15):1458–1465.
- , , , , .Predicting computerized physician order entry system adoption in US hospitals: can the federal mandate be met?Int J Med Inform.2008;77(8):539–545.
- ,.Implementation of computerized physician order entry in seven countries.Health Aff (Millwood).2009;28(2):404–414.
Copyright © 2011 Society of Hospital Medicine
