User login
The Enhanced Care Program: Impact of a Care Transition Program on 30-Day Hospital Readmissions for Patients Discharged From an Acute Care Facility to Skilled Nursing Facilities
Public reporting of readmission rates on the Nursing Home Compare website is mandated to begin on October 1, 2017, with skilled nursing facilities (SNFs) set to receive a Medicare bonus or penalty beginning a year later.1 The Centers for Medicare & Medicaid Services (CMS) began public reporting of hospitals’ 30-day readmission rates for selected conditions in 2009, and the Patient Protection and Affordable Care Act of 2010 mandated financial penalties for excess readmissions through the Hospital Readmission Reduction Program.2 In response, most hospitals have focused on patients who return home following discharge. Innovative interventions have proven successful, such as the Transitional Care model developed by Naylor and Coleman’s Care Transitions Intervention.3-5 Approximately 20% of Medicare beneficiaries are discharged from hospitals to SNFs, and these patients have higher readmission rates than those discharged home. CMS reported that in 2010, 23.3% of those with an SNF stay were readmitted within 30 days, compared with 18.8% for those with other discharge dispositions.6
Some work has been undertaken in this arena. In 2012, the Center for Medicare and Medicaid Innovation (CMMI) and the Medicare-Medicaid Coordination Office jointly launched the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents.7 This partnership established 7 Enhanced Care and Coordination Provider organizations and was designed to improve care by reducing hospitalizations among long-stay, dual-eligible nursing facility residents at 143 nursing homes in 7 states.8 At the time of the most recent project report, there were mixed results regarding program effects on hospitalizations and spending, with 2 states showing strongly positive patterns, 3 states with reductions that were consistent though not statistically strong, and mixed results in the remaining states. Quality measures did not show any pattern suggesting a program effect.9 Interventions to Reduce Acute Care Transfers (INTERACT) II was a 6-month, collaborative, quality-improvement project implemented in 2009 at 30 nursing homes in 3 states.10 The project evaluation found a statistically significant, 17% decrease in self-reported hospital admissions among the 25 SNFs that completed the intervention, compared with the same 6 months in the prior year. The Cleveland Clinic recently reported favorable results implementing its Connected Care model, which relied on staff physicians and advanced practice professionals to visit patients 4 to 5 times per week and be on call 24/7 at 7 intervention SNFs.11 Through this intervention, it successfully reduced its 30-day hospital readmission rate from SNFs from 28.1% to 21.7% (P < 0.001), and the authors posed the question as to whether its model and results were reproducible in other healthcare systems.
Herein, we report on the results of a collaborative initiative named the Enhanced Care Program (ECP), which offers the services of clinical providers and administrative staff to assist with the care of patients at 8 partner SNFs. The 3 components of ECP (described below) were specifically designed to address commonly recognized gaps and opportunities in routine SNF care. In contrast to the Cleveland Clinic’s Connected Care model (which involved hospital-employed physicians serving as the SNF attendings and excluded patients followed by their own physicians), ECP was designed to integrate into a pluralistic, community model whereby independent physicians continued to follow their own patients at the SNFs. The Connected Care analysis compared participating versus nonparticipating SNFs; both the Connected Care model and the INTERACT II evaluation relied on pre–post comparisons; the CMMI evaluation used a difference-in-differences model to compare the outcomes of the program SNFs with those of a matched comparison group of nonparticipating SNFs. The evaluation of ECP differs from these other initiatives, using a concurrent comparison group of patients discharged to the same SNFs but who were not enrolled in ECP.
METHODS
Setting
Cedars-Sinai Medical Center (CSMC) is an 850-bed, acute care facility located in an urban area of Los Angeles. Eight SNFs, ranging in size from 49 to 150 beds and located between 0.6 and 2.2 miles from CSMC, were invited to partner with the ECP. The physician community encompasses more than 2000 physicians on the medical staff, including private practitioners, nonteaching hospitalists, full-time faculty hospitalists, and faculty specialists.
Study Design and Patients
This was an observational, retrospective cohort analysis of 30-day same-hospital readmissions among 3951 patients discharged from CSMC to 8 SNFs between January 1, 2014, and June 30, 2015. A total of 2394 patients were enrolled in the ECP, and 1557 patients were not enrolled.
ECP Enrollment Protocol
Every patient discharged from CSMC to 1 of the 8 partner SNFs was eligible to participate in the program. To respect the autonomy of the SNF attending physicians and to facilitate a collaborative relationship, the decision to enroll a patient in the ECP rested with the SNF attending physician. The ECP team maintained a database that tracked whether each SNF attending physician (1) opted to automatically enroll all his or her patients in the ECP, (2) opted to enroll patients on a case-by-case basis (in which case an ECP nurse practitioner [N
Program Description
Patients enrolled in the ECP experienced the standard care provided by the SNF staff and attending physicians plus a clinical care program delivered by 9 full-time NPs, 1 full-time pharmacist, 1 pharmacy technician, 1 full-time nurse educator, a program administrator, and a medical director.
The program included the following 3 major components:
1. Direct patient care and 24/7 NP availability: Program enrollment began with an on-site, bedside evaluation by an ECP NP at the SNF within 24 hours of arrival and continued with weekly NP rounding (or more frequently, if clinically indicated) on the patient. Each encounter included a review of the medical record; a dialogue with the patient’s SNF attending physician to formulate treatment plans and place orders; discussions with nurses, family members, and other caregivers; and documentation in the medical record. The ECP team was on-site at the SNFs 7 days a week and on call 24/7 to address questions and concerns. Patients remained enrolled in the ECP from SNF admission to discharge even if their stay extended beyond 30 days.
2. Medication reconciliation: The ECP pharmacy team completed a review of a patient’s SNF medication administration record (MAR) within 72 hours of SNF admission. This process involved the pharmacy technician gathering medication lists from the SNFs and CSMC and providing this information to the pharmacist for a medication reconciliation and clinical evaluation. Discrepancies and pharmacist recommendations were communicated to the ECP NPs, and all identified issues were resolved.
3. Educational in-services: Building upon the INTERACT II model, the ECP team identified high-yield, clinically relevant topics, which the ECP nurse educator turned into monthly educational sessions for the SNF nursing staff at each of the participating SNFs.10
Primary Outcome Measure
An inpatient readmission to CSMC within 30 days of the hospital discharge date was counted as a readmission, whether the patient returned directly from an SNF or was readmitted from home after an SNF discharge.
Data
ECP patients were identified using a log maintained by the ECP program manager. Non-ECP patients discharged to the same SNFs during the study period were identified from CSMC’s electronic registry of SNF discharges. Covariates known to be associated with increased risk of 30-day readmission were obtained from CSMC’s electronic data warehouse, including demographic information, length of stay (LOS) of index hospitalization, and payer.12 Eleven clinical service lines represented patients’ clinical conditions based on Medicare-Severity Diagnosis-Related groupings. The discharge severity of illness score was calculated using 3M All Patients Refined Diagnosis Related Group software, version 33.13
Analysis
Characteristics of the ECP and non-ECP patients were compared using the χ2 test. A multivariable logistic regression model with fixed effects for SNF was created to determine the program’s impact on 30-day hospital readmission, adjusting for patient characteristics. The Pearson χ2 goodness-of-fit test and the link test for model specification were used to evaluate model specification. The sensitivity of the results to differences in patient characteristics was assessed in 2 ways. First, the ECP and non-ECP populations were stratified based on race and/or ethnicity and payer, and the multivariable regression model was run within the strata associated with the highest readmission rates. Second, a propensity analysis using inverse probability of treatment weighting (IPTW) was performed to control for group differences. Results of all comparisons were considered statistically significant when P < 0.05. Stata version 13 was used to perform the main analyses.14 The propensity analysis was conducted using R version 3.2.3. The CSMC Institutional Review Board (IRB) determined that this study qualified as a quality-improvement activity and did not require IRB approval or exemption.
RESULTS
The average unadjusted 30-day readmission rate for ECP patients over the 18-month study period was 17.2%, compared to 23.0% for patients not enrolled in ECP (P < 0.001) (Figure 1). After adjusting for patient characteristics, ECP patients had 29% lower odds (95% confidence interval [CI], 0.60-0.85) of being readmitted to the medical center within 30 days than non-ECP patients at the same SNFs. The characteristics of the ECP and comparison patient cohorts are shown in Table 1. There were significant differences in sociodemographic characteristics: The ECP group had a higher proportion of non-Hispanic white patients, while the comparison group had a higher proportion of patients who were African American or Hispanic. ECP patients were more likely to prefer speaking English, while Russian, Farsi, and Spanish were preferred more frequently in the comparison group. There were also differences in payer mix, with the ECP group including proportionately more Medicare fee-for-service (52.9% vs 35.0%, P < 0.001), while the comparison group had a correspondingly larger proportion of dual-eligible (Medicare and Medicaid) patients (55.0% vs 35.1%, P < 0.001).
The largest clinical differences observed between the ECP and non-ECP groups were the proportions of patients in the clinical service lines of orthopedic surgery (28.7% vs 21.1%, P < 0.001), medical cardiology (7.4% vs 9.7%, P < 0.001), and surgery other than general surgery (5.8% vs 9.2%, P < 0.001). Despite these differences in case mix, no differences were seen between the 2 groups in discharge severity of illness or LOS of the index hospitalization. The distribution of index hospital LOS by quartile was the same, with the exception that the ECP group had a higher proportion of patients with longer LOS.
Sensitivity Analyses
The results were robust when tested within strata of the study population, including analyses limited to dual-eligible patients, African American patients, patients admitted to all except the highest volume facility, and patients admitted to any service line other than orthopedic surgery. Similar results were obtained when the study population was restricted to patients living within the medical center’s primary service area and to patients living in zip codes in which the proportion of adults living in households with income below 100% of the poverty level was 15% or greater (see Supplementary Material for results).
The effect of the program on readmission was also consistent when the full logistic regression model was run with IPTW using the propensity score. The evaluation of standardized cluster differences between the ECP and non-ECP groups before and after IPTW showed that the differences were reduced to <10% for being African American; speaking Russian or Farsi; having dual-eligible insurance coverage; having orthopedic surgery; being discharged from the clinical service lines of gastroenterology, pulmonary, other surgery, and other services; and having an index hospital LOS of 4 to 5 days or 10 or more days (results are provided in the Supplementary Material).
DISCUSSION
Hospitals continue to experience significant pressure to manage LOS, and SNFs and hospitals are being held accountable for readmission rates. The setting of this study is representative of many large, urban hospitals in the United States whose communities include a heterogeneous mix of hospitalists, primary care physicians who follow their patients in SNFs, and independent SNFs.15 The current regulations have not kept up with the increasing acuity and complexity of SNF patients. Specifically, Medicare guidelines allow the SNF attending physician up to 72 hours to complete a history and physical (or 7 days if he or she was the hospital attending physician for the index hospitalization) and only require monthly follow-up visits. It is the opinion of the ECP designers that these relatively lax requirements present unnecessary risk for vulnerable patients. While the INTERACT II model was focused largely on educational initiatives (with an advanced practice nurse available in a consultative role, as needed), the central tenet of ECP was similar to the Connected Care model in that the focus was on adding an extra layer of direct clinical support. Protocols that provided timely initial assessments by an NP (within 24 hours), weekly NP rounding (at a minimum), and 24/7 on-call availability all contributed to helping patients stay on track. Although the ECP had patients visited less frequently than the Connected Care model, and the Cleveland Clinic started with a higher baseline 30-day readmission rate from SNFs, similar overall reductions in 30-day readmissions were observed. The key point from both initiatives is that an increase in clinical touchpoints and ease of access to clinicians generates myriad opportunities to identify and address small issues before they become clinical emergencies requiring hospital transfers and readmissions.
Correcting medication discrepancies between hospital discharge summaries and SNF admission orders through a systematic medication reconciliation using a clinical pharmacist has previously been shown to improve outcomes.16-18 The ECP pharmacy technician and ECP clinical pharmacist discovered and corrected errors on a daily basis that ranged from incidental to potentially life-threatening. If the SNF staff does not provide the patient’s MAR within 48 hours of arrival, the pharmacy technician contacts the facility to obtain the information. As a result, all patients enrolled in the ECP during the study period received this intervention (unless they were rehospitalized or left the SNF before the process was completed), and 54% of ECP patients required some form of intervention after medication reconciliation was completed (data not shown).
This type of program requires hospital leadership and SNF administrators to be fully committed to developing strong working relationships, and in fact, there is evidence that SNF baseline readmission rates have a greater influence on patients’ risk of rehospitalization than the discharging hospital itself.19-21 Monthly educational in-services are delivered at the partner SNFs to enhance SNF nursing staff knowledge and clinical acumen. High-impact topics identified by the ECP team include the following: fall prevention, hand hygiene, venous thromboembolism, cardiovascular health, how to report change in condition, and advanced care planning, among others. While no formal pre–post assessments of the SNF nurses’ knowledge were conducted, a log of in-services was kept, subjective feedback was collected for performance improvement purposes, and continuing educational units were provided to the SNF nurses who attended.
This study has limitations. As a single-hospital study, generalizability may be limited. While adherence to the program components was closely monitored daily, service gaps may have occurred that were not captured. The program design makes it difficult to quantify the relative impact of the 3 program components on the outcome. Furthermore, the study was observational, so the differences in readmission rates may have been due to unmeasured variables. The decision to enroll patients in the ECP was made by each patient’s SNF attending physician, and those who chose to (or not to) participate in the program may manifest other, unmeasured practice patterns that made readmissions more or less likely. Participating physicians also had the option to enroll their patients on a case-by-case basis, introducing further potential bias in patient selection; however, <5% of physicians exercised this option. Patients may have also been readmitted to hospitals other than CSMC, producing an observed readmission rate for 1 or both groups that underrepresents the true outcome. On this point, while we did not systematically track these other-hospital readmissions for both groups, there is no reason to believe that this occurred preferentially for ECP or non-ECP patients.
Multiple sensitivity analyses were performed to address the observed differences between ECP and non-ECP patients. These included stratified examinations of variables differing between populations, examination of clustering effects between SNFs, and an analysis adjusted for the propensity to be included in the ECP. The calculated effect of the intervention on readmission remained robust, although we acknowledge that differences in the populations may persist and have influenced the outcomes even after controlling for multiple variables.22-25
In conclusion, the results of this intervention are compelling and add to the growing body of literature suggesting that a comprehensive, multipronged effort to enhance clinical oversight and coordination of care for SNF patients can improve outcomes. Given CMS’s plans to report SNF readmission rates in 2017 followed by the application of financial incentives in 2018, a favorable climate currently exists for greater coordination between hospitals and SNFs.26 We are currently undertaking an economic evaluation of the program.
Acknowledgments
The authors would like to thank the following people for their contributions: Mae Saunders, Rita Shane, Dr. Jon Kea, Miranda Li, the ECP NPs, the ECP pharmacy team, CSMC’s performance improvement team, and Alan Matus.
Disclosure
No conflicts of interest or disclosures.
1. Centers for Medicare & Medicaid Services (CMS), HHS. Medicare Program; Prospective Payment System and Consolidated Billing for Skilled Nursing Facilities (SNFs) for FY 2016, SNF Value-Based Purchasing Program, SNF Quality Reporting Program, and Staffing Data Collection. Final Rule. Fed Regist. 2015;80(149):46389-46477. PubMed
2. “Readmissions Reduction Program,” Centers for Medicare & Medicaid Services. http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed November 5, 2015.
3. Naylor MD, Brooten D, Campbell R, et al. Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial. JAMA. 1999;281:613-620. PubMed
4. Naylor MD, Brooten DA, Campbell RL, Maislin G, McCauley KM, Schwartz JS. Transitional care of older adults hospitalized with heart failure: a randomized, controlled trial. J Am Geriatr Soc. 2004;52:675-684. PubMed
5. Coleman EA, Parry C, Chalmers S, Min SJ. The care transitions intervention: results of a randomized controlled trial. Arch Intern Med. 2006;166:1822-1828. PubMed
6. CMS Office of Information Products and Data Analytics. National Medicare Readmission Findings: Recent Data and Trends. 2012. http://www.academyhealth.org/files/2012/sunday/brennan.pdf. Accessed on September 21, 2015.
7. Centers for Medicare & Medicaid Services, CMS Innovation Center. Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents. https://innovation.cms.gov/initiatives/rahnfr/. Accessed on November 5, 2015.
8. Unroe KT, Nazir A, Holtz LR, et al. The Optimizing Patient Transfers, Impacting Medical Quality and Improving Symptoms: Transforming Institutional Care Approach: Preliminary data from the implementation of a Centers for Medicare and Medicaid Services nursing facility demonstration project. J Am Geriatr Soc. 2015;65:165-169. PubMed
9. Ingber MJ, Feng Z, Khatstsky G, et al. Evaluation of the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents: Final Annual Report Project Year 3. Waltham, MA: RTI International, RTI Project Number 0212790.006, January 2016.
10. Ouslander JG, Lamb G, Tappen R, et al. Interventions to reduce hospitalizations from nursing homes: Evaluation of the INTERACT II collaborative quality improvement project. J Am Geriatr Soc. 2011:59:745-753. PubMed
11. Kim L, Kou L, Hu B, Gorodeski EZ, Rothberg M. Impact of a Connected Care Model on 30-Day Readmission Rates from Skilled Nursing Facilities. J Hosp Med. 2017;12:238-244. PubMed
12. Kansagara D, Englander H, Salanitro A, et al. Risk Prediction Models for Hospital Readmission: A Systematic Review. JAMA. 2011;306(15):1688-1698. PubMed
13. Averill RF, Goldfield N, Hughes JS, et al. All Patient Refined Diagnosis Related Groups (APR-DRGs): Methodology Overview. 3M Health Information Systems Document GRP-041 (2003). https://www.hcup-us.ahrq.gov/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf. Accessed on November 5, 2015.
14. StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.
15. Cebul RD, Rebitzer JB, Taylor LJ, Votruba ME. Organizational fragmentation and care quality in the U.S. healthcare system. J Econ Perspect. 2008;22(4):93-113. PubMed
16. Tjia J, Bonner A, Briesacher BA, McGee S, Terrill E, Miller K. Medication discrepancies upon hospital to skilled nursing facility transitions. J Gen Intern Med. 2009;24:630-635. PubMed
17. Desai R, Williams CE, Greene SB, Pierson S, Hansen RA. Medication errors during patient transitions into nursing homes: characteristics and association with patient harm. Am J Geriatr Pharmacother. 2011;9:413-422. PubMed
18. Chhabra PT, Rattinger GB, Dutcher SK, Hare ME, Parsons KL, Zuckerman IH. Medication reconciliation during the transition to and from long-term care settings: a systematic review. Res Social Adm Pharm. 2012;8(1):60-75. PubMed
19. Rahman M, Foster AD, Grabowski DC, Zinn JS, Mor V. Effect of hospital-SNF referral linkages on rehospitalization. Health Serv Res. 2013;48(6, pt 1):1898-1919. PubMed
20. Schoenfeld AJ, Zhang X, Grabowski DC, Mor V, Weissman JS, Rahman M. Hospital-skilled nursing facility referral linkage reduces readmission rates among Medicare patients receiving major surgery. Surgery. 2016;159(5):1461-1468. PubMed
21. Rahman M, McHugh J, Gozalo P, Ackerly DC, Mor V. The Contribution of Skilled Nursing Facilities to Hospitals’ Readmission Rate. HSR: Health Services Research. 2017;52(2):656-675. PubMed
22. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. New Engl J Med. 2009;360(14):1418-1428. PubMed
23. Hasan O, Meltzer DO, Shaykevich SA, et al. Hospital readmission in general medicine patients: a prediction model. J Hosp Med. 2010;25(3)211-219. PubMed
24. Allaudeen N, Vidyarhi A, Masella J, Auerbach A. Redefining readmission risk factors for general medicine patients. J Hosp Med. 2011;6(2):54-60. PubMed
25. Van Walraven C, Wong J, Forster AJ. LACE+ index: extension of a validated index to predict early death or urgent readmission after discharge using administrative data. Open Med. 2012;6(3):e80-e90. PubMed
26. Protecting Access to Medicare Act of 2014, Pub. L. No. 113-93, 128 Stat. 1040 (April 1, 2014). https://www.congress.gov/113/plaws/publ93/PLAW-113publ93.pdf. Accessed on October 3, 2015.
Public reporting of readmission rates on the Nursing Home Compare website is mandated to begin on October 1, 2017, with skilled nursing facilities (SNFs) set to receive a Medicare bonus or penalty beginning a year later.1 The Centers for Medicare & Medicaid Services (CMS) began public reporting of hospitals’ 30-day readmission rates for selected conditions in 2009, and the Patient Protection and Affordable Care Act of 2010 mandated financial penalties for excess readmissions through the Hospital Readmission Reduction Program.2 In response, most hospitals have focused on patients who return home following discharge. Innovative interventions have proven successful, such as the Transitional Care model developed by Naylor and Coleman’s Care Transitions Intervention.3-5 Approximately 20% of Medicare beneficiaries are discharged from hospitals to SNFs, and these patients have higher readmission rates than those discharged home. CMS reported that in 2010, 23.3% of those with an SNF stay were readmitted within 30 days, compared with 18.8% for those with other discharge dispositions.6
Some work has been undertaken in this arena. In 2012, the Center for Medicare and Medicaid Innovation (CMMI) and the Medicare-Medicaid Coordination Office jointly launched the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents.7 This partnership established 7 Enhanced Care and Coordination Provider organizations and was designed to improve care by reducing hospitalizations among long-stay, dual-eligible nursing facility residents at 143 nursing homes in 7 states.8 At the time of the most recent project report, there were mixed results regarding program effects on hospitalizations and spending, with 2 states showing strongly positive patterns, 3 states with reductions that were consistent though not statistically strong, and mixed results in the remaining states. Quality measures did not show any pattern suggesting a program effect.9 Interventions to Reduce Acute Care Transfers (INTERACT) II was a 6-month, collaborative, quality-improvement project implemented in 2009 at 30 nursing homes in 3 states.10 The project evaluation found a statistically significant, 17% decrease in self-reported hospital admissions among the 25 SNFs that completed the intervention, compared with the same 6 months in the prior year. The Cleveland Clinic recently reported favorable results implementing its Connected Care model, which relied on staff physicians and advanced practice professionals to visit patients 4 to 5 times per week and be on call 24/7 at 7 intervention SNFs.11 Through this intervention, it successfully reduced its 30-day hospital readmission rate from SNFs from 28.1% to 21.7% (P < 0.001), and the authors posed the question as to whether its model and results were reproducible in other healthcare systems.
Herein, we report on the results of a collaborative initiative named the Enhanced Care Program (ECP), which offers the services of clinical providers and administrative staff to assist with the care of patients at 8 partner SNFs. The 3 components of ECP (described below) were specifically designed to address commonly recognized gaps and opportunities in routine SNF care. In contrast to the Cleveland Clinic’s Connected Care model (which involved hospital-employed physicians serving as the SNF attendings and excluded patients followed by their own physicians), ECP was designed to integrate into a pluralistic, community model whereby independent physicians continued to follow their own patients at the SNFs. The Connected Care analysis compared participating versus nonparticipating SNFs; both the Connected Care model and the INTERACT II evaluation relied on pre–post comparisons; the CMMI evaluation used a difference-in-differences model to compare the outcomes of the program SNFs with those of a matched comparison group of nonparticipating SNFs. The evaluation of ECP differs from these other initiatives, using a concurrent comparison group of patients discharged to the same SNFs but who were not enrolled in ECP.
METHODS
Setting
Cedars-Sinai Medical Center (CSMC) is an 850-bed, acute care facility located in an urban area of Los Angeles. Eight SNFs, ranging in size from 49 to 150 beds and located between 0.6 and 2.2 miles from CSMC, were invited to partner with the ECP. The physician community encompasses more than 2000 physicians on the medical staff, including private practitioners, nonteaching hospitalists, full-time faculty hospitalists, and faculty specialists.
Study Design and Patients
This was an observational, retrospective cohort analysis of 30-day same-hospital readmissions among 3951 patients discharged from CSMC to 8 SNFs between January 1, 2014, and June 30, 2015. A total of 2394 patients were enrolled in the ECP, and 1557 patients were not enrolled.
ECP Enrollment Protocol
Every patient discharged from CSMC to 1 of the 8 partner SNFs was eligible to participate in the program. To respect the autonomy of the SNF attending physicians and to facilitate a collaborative relationship, the decision to enroll a patient in the ECP rested with the SNF attending physician. The ECP team maintained a database that tracked whether each SNF attending physician (1) opted to automatically enroll all his or her patients in the ECP, (2) opted to enroll patients on a case-by-case basis (in which case an ECP nurse practitioner [N
Program Description
Patients enrolled in the ECP experienced the standard care provided by the SNF staff and attending physicians plus a clinical care program delivered by 9 full-time NPs, 1 full-time pharmacist, 1 pharmacy technician, 1 full-time nurse educator, a program administrator, and a medical director.
The program included the following 3 major components:
1. Direct patient care and 24/7 NP availability: Program enrollment began with an on-site, bedside evaluation by an ECP NP at the SNF within 24 hours of arrival and continued with weekly NP rounding (or more frequently, if clinically indicated) on the patient. Each encounter included a review of the medical record; a dialogue with the patient’s SNF attending physician to formulate treatment plans and place orders; discussions with nurses, family members, and other caregivers; and documentation in the medical record. The ECP team was on-site at the SNFs 7 days a week and on call 24/7 to address questions and concerns. Patients remained enrolled in the ECP from SNF admission to discharge even if their stay extended beyond 30 days.
2. Medication reconciliation: The ECP pharmacy team completed a review of a patient’s SNF medication administration record (MAR) within 72 hours of SNF admission. This process involved the pharmacy technician gathering medication lists from the SNFs and CSMC and providing this information to the pharmacist for a medication reconciliation and clinical evaluation. Discrepancies and pharmacist recommendations were communicated to the ECP NPs, and all identified issues were resolved.
3. Educational in-services: Building upon the INTERACT II model, the ECP team identified high-yield, clinically relevant topics, which the ECP nurse educator turned into monthly educational sessions for the SNF nursing staff at each of the participating SNFs.10
Primary Outcome Measure
An inpatient readmission to CSMC within 30 days of the hospital discharge date was counted as a readmission, whether the patient returned directly from an SNF or was readmitted from home after an SNF discharge.
Data
ECP patients were identified using a log maintained by the ECP program manager. Non-ECP patients discharged to the same SNFs during the study period were identified from CSMC’s electronic registry of SNF discharges. Covariates known to be associated with increased risk of 30-day readmission were obtained from CSMC’s electronic data warehouse, including demographic information, length of stay (LOS) of index hospitalization, and payer.12 Eleven clinical service lines represented patients’ clinical conditions based on Medicare-Severity Diagnosis-Related groupings. The discharge severity of illness score was calculated using 3M All Patients Refined Diagnosis Related Group software, version 33.13
Analysis
Characteristics of the ECP and non-ECP patients were compared using the χ2 test. A multivariable logistic regression model with fixed effects for SNF was created to determine the program’s impact on 30-day hospital readmission, adjusting for patient characteristics. The Pearson χ2 goodness-of-fit test and the link test for model specification were used to evaluate model specification. The sensitivity of the results to differences in patient characteristics was assessed in 2 ways. First, the ECP and non-ECP populations were stratified based on race and/or ethnicity and payer, and the multivariable regression model was run within the strata associated with the highest readmission rates. Second, a propensity analysis using inverse probability of treatment weighting (IPTW) was performed to control for group differences. Results of all comparisons were considered statistically significant when P < 0.05. Stata version 13 was used to perform the main analyses.14 The propensity analysis was conducted using R version 3.2.3. The CSMC Institutional Review Board (IRB) determined that this study qualified as a quality-improvement activity and did not require IRB approval or exemption.
RESULTS
The average unadjusted 30-day readmission rate for ECP patients over the 18-month study period was 17.2%, compared to 23.0% for patients not enrolled in ECP (P < 0.001) (Figure 1). After adjusting for patient characteristics, ECP patients had 29% lower odds (95% confidence interval [CI], 0.60-0.85) of being readmitted to the medical center within 30 days than non-ECP patients at the same SNFs. The characteristics of the ECP and comparison patient cohorts are shown in Table 1. There were significant differences in sociodemographic characteristics: The ECP group had a higher proportion of non-Hispanic white patients, while the comparison group had a higher proportion of patients who were African American or Hispanic. ECP patients were more likely to prefer speaking English, while Russian, Farsi, and Spanish were preferred more frequently in the comparison group. There were also differences in payer mix, with the ECP group including proportionately more Medicare fee-for-service (52.9% vs 35.0%, P < 0.001), while the comparison group had a correspondingly larger proportion of dual-eligible (Medicare and Medicaid) patients (55.0% vs 35.1%, P < 0.001).
The largest clinical differences observed between the ECP and non-ECP groups were the proportions of patients in the clinical service lines of orthopedic surgery (28.7% vs 21.1%, P < 0.001), medical cardiology (7.4% vs 9.7%, P < 0.001), and surgery other than general surgery (5.8% vs 9.2%, P < 0.001). Despite these differences in case mix, no differences were seen between the 2 groups in discharge severity of illness or LOS of the index hospitalization. The distribution of index hospital LOS by quartile was the same, with the exception that the ECP group had a higher proportion of patients with longer LOS.
Sensitivity Analyses
The results were robust when tested within strata of the study population, including analyses limited to dual-eligible patients, African American patients, patients admitted to all except the highest volume facility, and patients admitted to any service line other than orthopedic surgery. Similar results were obtained when the study population was restricted to patients living within the medical center’s primary service area and to patients living in zip codes in which the proportion of adults living in households with income below 100% of the poverty level was 15% or greater (see Supplementary Material for results).
The effect of the program on readmission was also consistent when the full logistic regression model was run with IPTW using the propensity score. The evaluation of standardized cluster differences between the ECP and non-ECP groups before and after IPTW showed that the differences were reduced to <10% for being African American; speaking Russian or Farsi; having dual-eligible insurance coverage; having orthopedic surgery; being discharged from the clinical service lines of gastroenterology, pulmonary, other surgery, and other services; and having an index hospital LOS of 4 to 5 days or 10 or more days (results are provided in the Supplementary Material).
DISCUSSION
Hospitals continue to experience significant pressure to manage LOS, and SNFs and hospitals are being held accountable for readmission rates. The setting of this study is representative of many large, urban hospitals in the United States whose communities include a heterogeneous mix of hospitalists, primary care physicians who follow their patients in SNFs, and independent SNFs.15 The current regulations have not kept up with the increasing acuity and complexity of SNF patients. Specifically, Medicare guidelines allow the SNF attending physician up to 72 hours to complete a history and physical (or 7 days if he or she was the hospital attending physician for the index hospitalization) and only require monthly follow-up visits. It is the opinion of the ECP designers that these relatively lax requirements present unnecessary risk for vulnerable patients. While the INTERACT II model was focused largely on educational initiatives (with an advanced practice nurse available in a consultative role, as needed), the central tenet of ECP was similar to the Connected Care model in that the focus was on adding an extra layer of direct clinical support. Protocols that provided timely initial assessments by an NP (within 24 hours), weekly NP rounding (at a minimum), and 24/7 on-call availability all contributed to helping patients stay on track. Although the ECP had patients visited less frequently than the Connected Care model, and the Cleveland Clinic started with a higher baseline 30-day readmission rate from SNFs, similar overall reductions in 30-day readmissions were observed. The key point from both initiatives is that an increase in clinical touchpoints and ease of access to clinicians generates myriad opportunities to identify and address small issues before they become clinical emergencies requiring hospital transfers and readmissions.
Correcting medication discrepancies between hospital discharge summaries and SNF admission orders through a systematic medication reconciliation using a clinical pharmacist has previously been shown to improve outcomes.16-18 The ECP pharmacy technician and ECP clinical pharmacist discovered and corrected errors on a daily basis that ranged from incidental to potentially life-threatening. If the SNF staff does not provide the patient’s MAR within 48 hours of arrival, the pharmacy technician contacts the facility to obtain the information. As a result, all patients enrolled in the ECP during the study period received this intervention (unless they were rehospitalized or left the SNF before the process was completed), and 54% of ECP patients required some form of intervention after medication reconciliation was completed (data not shown).
This type of program requires hospital leadership and SNF administrators to be fully committed to developing strong working relationships, and in fact, there is evidence that SNF baseline readmission rates have a greater influence on patients’ risk of rehospitalization than the discharging hospital itself.19-21 Monthly educational in-services are delivered at the partner SNFs to enhance SNF nursing staff knowledge and clinical acumen. High-impact topics identified by the ECP team include the following: fall prevention, hand hygiene, venous thromboembolism, cardiovascular health, how to report change in condition, and advanced care planning, among others. While no formal pre–post assessments of the SNF nurses’ knowledge were conducted, a log of in-services was kept, subjective feedback was collected for performance improvement purposes, and continuing educational units were provided to the SNF nurses who attended.
This study has limitations. As a single-hospital study, generalizability may be limited. While adherence to the program components was closely monitored daily, service gaps may have occurred that were not captured. The program design makes it difficult to quantify the relative impact of the 3 program components on the outcome. Furthermore, the study was observational, so the differences in readmission rates may have been due to unmeasured variables. The decision to enroll patients in the ECP was made by each patient’s SNF attending physician, and those who chose to (or not to) participate in the program may manifest other, unmeasured practice patterns that made readmissions more or less likely. Participating physicians also had the option to enroll their patients on a case-by-case basis, introducing further potential bias in patient selection; however, <5% of physicians exercised this option. Patients may have also been readmitted to hospitals other than CSMC, producing an observed readmission rate for 1 or both groups that underrepresents the true outcome. On this point, while we did not systematically track these other-hospital readmissions for both groups, there is no reason to believe that this occurred preferentially for ECP or non-ECP patients.
Multiple sensitivity analyses were performed to address the observed differences between ECP and non-ECP patients. These included stratified examinations of variables differing between populations, examination of clustering effects between SNFs, and an analysis adjusted for the propensity to be included in the ECP. The calculated effect of the intervention on readmission remained robust, although we acknowledge that differences in the populations may persist and have influenced the outcomes even after controlling for multiple variables.22-25
In conclusion, the results of this intervention are compelling and add to the growing body of literature suggesting that a comprehensive, multipronged effort to enhance clinical oversight and coordination of care for SNF patients can improve outcomes. Given CMS’s plans to report SNF readmission rates in 2017 followed by the application of financial incentives in 2018, a favorable climate currently exists for greater coordination between hospitals and SNFs.26 We are currently undertaking an economic evaluation of the program.
Acknowledgments
The authors would like to thank the following people for their contributions: Mae Saunders, Rita Shane, Dr. Jon Kea, Miranda Li, the ECP NPs, the ECP pharmacy team, CSMC’s performance improvement team, and Alan Matus.
Disclosure
No conflicts of interest or disclosures.
Public reporting of readmission rates on the Nursing Home Compare website is mandated to begin on October 1, 2017, with skilled nursing facilities (SNFs) set to receive a Medicare bonus or penalty beginning a year later.1 The Centers for Medicare & Medicaid Services (CMS) began public reporting of hospitals’ 30-day readmission rates for selected conditions in 2009, and the Patient Protection and Affordable Care Act of 2010 mandated financial penalties for excess readmissions through the Hospital Readmission Reduction Program.2 In response, most hospitals have focused on patients who return home following discharge. Innovative interventions have proven successful, such as the Transitional Care model developed by Naylor and Coleman’s Care Transitions Intervention.3-5 Approximately 20% of Medicare beneficiaries are discharged from hospitals to SNFs, and these patients have higher readmission rates than those discharged home. CMS reported that in 2010, 23.3% of those with an SNF stay were readmitted within 30 days, compared with 18.8% for those with other discharge dispositions.6
Some work has been undertaken in this arena. In 2012, the Center for Medicare and Medicaid Innovation (CMMI) and the Medicare-Medicaid Coordination Office jointly launched the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents.7 This partnership established 7 Enhanced Care and Coordination Provider organizations and was designed to improve care by reducing hospitalizations among long-stay, dual-eligible nursing facility residents at 143 nursing homes in 7 states.8 At the time of the most recent project report, there were mixed results regarding program effects on hospitalizations and spending, with 2 states showing strongly positive patterns, 3 states with reductions that were consistent though not statistically strong, and mixed results in the remaining states. Quality measures did not show any pattern suggesting a program effect.9 Interventions to Reduce Acute Care Transfers (INTERACT) II was a 6-month, collaborative, quality-improvement project implemented in 2009 at 30 nursing homes in 3 states.10 The project evaluation found a statistically significant, 17% decrease in self-reported hospital admissions among the 25 SNFs that completed the intervention, compared with the same 6 months in the prior year. The Cleveland Clinic recently reported favorable results implementing its Connected Care model, which relied on staff physicians and advanced practice professionals to visit patients 4 to 5 times per week and be on call 24/7 at 7 intervention SNFs.11 Through this intervention, it successfully reduced its 30-day hospital readmission rate from SNFs from 28.1% to 21.7% (P < 0.001), and the authors posed the question as to whether its model and results were reproducible in other healthcare systems.
Herein, we report on the results of a collaborative initiative named the Enhanced Care Program (ECP), which offers the services of clinical providers and administrative staff to assist with the care of patients at 8 partner SNFs. The 3 components of ECP (described below) were specifically designed to address commonly recognized gaps and opportunities in routine SNF care. In contrast to the Cleveland Clinic’s Connected Care model (which involved hospital-employed physicians serving as the SNF attendings and excluded patients followed by their own physicians), ECP was designed to integrate into a pluralistic, community model whereby independent physicians continued to follow their own patients at the SNFs. The Connected Care analysis compared participating versus nonparticipating SNFs; both the Connected Care model and the INTERACT II evaluation relied on pre–post comparisons; the CMMI evaluation used a difference-in-differences model to compare the outcomes of the program SNFs with those of a matched comparison group of nonparticipating SNFs. The evaluation of ECP differs from these other initiatives, using a concurrent comparison group of patients discharged to the same SNFs but who were not enrolled in ECP.
METHODS
Setting
Cedars-Sinai Medical Center (CSMC) is an 850-bed, acute care facility located in an urban area of Los Angeles. Eight SNFs, ranging in size from 49 to 150 beds and located between 0.6 and 2.2 miles from CSMC, were invited to partner with the ECP. The physician community encompasses more than 2000 physicians on the medical staff, including private practitioners, nonteaching hospitalists, full-time faculty hospitalists, and faculty specialists.
Study Design and Patients
This was an observational, retrospective cohort analysis of 30-day same-hospital readmissions among 3951 patients discharged from CSMC to 8 SNFs between January 1, 2014, and June 30, 2015. A total of 2394 patients were enrolled in the ECP, and 1557 patients were not enrolled.
ECP Enrollment Protocol
Every patient discharged from CSMC to 1 of the 8 partner SNFs was eligible to participate in the program. To respect the autonomy of the SNF attending physicians and to facilitate a collaborative relationship, the decision to enroll a patient in the ECP rested with the SNF attending physician. The ECP team maintained a database that tracked whether each SNF attending physician (1) opted to automatically enroll all his or her patients in the ECP, (2) opted to enroll patients on a case-by-case basis (in which case an ECP nurse practitioner [N
Program Description
Patients enrolled in the ECP experienced the standard care provided by the SNF staff and attending physicians plus a clinical care program delivered by 9 full-time NPs, 1 full-time pharmacist, 1 pharmacy technician, 1 full-time nurse educator, a program administrator, and a medical director.
The program included the following 3 major components:
1. Direct patient care and 24/7 NP availability: Program enrollment began with an on-site, bedside evaluation by an ECP NP at the SNF within 24 hours of arrival and continued with weekly NP rounding (or more frequently, if clinically indicated) on the patient. Each encounter included a review of the medical record; a dialogue with the patient’s SNF attending physician to formulate treatment plans and place orders; discussions with nurses, family members, and other caregivers; and documentation in the medical record. The ECP team was on-site at the SNFs 7 days a week and on call 24/7 to address questions and concerns. Patients remained enrolled in the ECP from SNF admission to discharge even if their stay extended beyond 30 days.
2. Medication reconciliation: The ECP pharmacy team completed a review of a patient’s SNF medication administration record (MAR) within 72 hours of SNF admission. This process involved the pharmacy technician gathering medication lists from the SNFs and CSMC and providing this information to the pharmacist for a medication reconciliation and clinical evaluation. Discrepancies and pharmacist recommendations were communicated to the ECP NPs, and all identified issues were resolved.
3. Educational in-services: Building upon the INTERACT II model, the ECP team identified high-yield, clinically relevant topics, which the ECP nurse educator turned into monthly educational sessions for the SNF nursing staff at each of the participating SNFs.10
Primary Outcome Measure
An inpatient readmission to CSMC within 30 days of the hospital discharge date was counted as a readmission, whether the patient returned directly from an SNF or was readmitted from home after an SNF discharge.
Data
ECP patients were identified using a log maintained by the ECP program manager. Non-ECP patients discharged to the same SNFs during the study period were identified from CSMC’s electronic registry of SNF discharges. Covariates known to be associated with increased risk of 30-day readmission were obtained from CSMC’s electronic data warehouse, including demographic information, length of stay (LOS) of index hospitalization, and payer.12 Eleven clinical service lines represented patients’ clinical conditions based on Medicare-Severity Diagnosis-Related groupings. The discharge severity of illness score was calculated using 3M All Patients Refined Diagnosis Related Group software, version 33.13
Analysis
Characteristics of the ECP and non-ECP patients were compared using the χ2 test. A multivariable logistic regression model with fixed effects for SNF was created to determine the program’s impact on 30-day hospital readmission, adjusting for patient characteristics. The Pearson χ2 goodness-of-fit test and the link test for model specification were used to evaluate model specification. The sensitivity of the results to differences in patient characteristics was assessed in 2 ways. First, the ECP and non-ECP populations were stratified based on race and/or ethnicity and payer, and the multivariable regression model was run within the strata associated with the highest readmission rates. Second, a propensity analysis using inverse probability of treatment weighting (IPTW) was performed to control for group differences. Results of all comparisons were considered statistically significant when P < 0.05. Stata version 13 was used to perform the main analyses.14 The propensity analysis was conducted using R version 3.2.3. The CSMC Institutional Review Board (IRB) determined that this study qualified as a quality-improvement activity and did not require IRB approval or exemption.
RESULTS
The average unadjusted 30-day readmission rate for ECP patients over the 18-month study period was 17.2%, compared to 23.0% for patients not enrolled in ECP (P < 0.001) (Figure 1). After adjusting for patient characteristics, ECP patients had 29% lower odds (95% confidence interval [CI], 0.60-0.85) of being readmitted to the medical center within 30 days than non-ECP patients at the same SNFs. The characteristics of the ECP and comparison patient cohorts are shown in Table 1. There were significant differences in sociodemographic characteristics: The ECP group had a higher proportion of non-Hispanic white patients, while the comparison group had a higher proportion of patients who were African American or Hispanic. ECP patients were more likely to prefer speaking English, while Russian, Farsi, and Spanish were preferred more frequently in the comparison group. There were also differences in payer mix, with the ECP group including proportionately more Medicare fee-for-service (52.9% vs 35.0%, P < 0.001), while the comparison group had a correspondingly larger proportion of dual-eligible (Medicare and Medicaid) patients (55.0% vs 35.1%, P < 0.001).
The largest clinical differences observed between the ECP and non-ECP groups were the proportions of patients in the clinical service lines of orthopedic surgery (28.7% vs 21.1%, P < 0.001), medical cardiology (7.4% vs 9.7%, P < 0.001), and surgery other than general surgery (5.8% vs 9.2%, P < 0.001). Despite these differences in case mix, no differences were seen between the 2 groups in discharge severity of illness or LOS of the index hospitalization. The distribution of index hospital LOS by quartile was the same, with the exception that the ECP group had a higher proportion of patients with longer LOS.
Sensitivity Analyses
The results were robust when tested within strata of the study population, including analyses limited to dual-eligible patients, African American patients, patients admitted to all except the highest volume facility, and patients admitted to any service line other than orthopedic surgery. Similar results were obtained when the study population was restricted to patients living within the medical center’s primary service area and to patients living in zip codes in which the proportion of adults living in households with income below 100% of the poverty level was 15% or greater (see Supplementary Material for results).
The effect of the program on readmission was also consistent when the full logistic regression model was run with IPTW using the propensity score. The evaluation of standardized cluster differences between the ECP and non-ECP groups before and after IPTW showed that the differences were reduced to <10% for being African American; speaking Russian or Farsi; having dual-eligible insurance coverage; having orthopedic surgery; being discharged from the clinical service lines of gastroenterology, pulmonary, other surgery, and other services; and having an index hospital LOS of 4 to 5 days or 10 or more days (results are provided in the Supplementary Material).
DISCUSSION
Hospitals continue to experience significant pressure to manage LOS, and SNFs and hospitals are being held accountable for readmission rates. The setting of this study is representative of many large, urban hospitals in the United States whose communities include a heterogeneous mix of hospitalists, primary care physicians who follow their patients in SNFs, and independent SNFs.15 The current regulations have not kept up with the increasing acuity and complexity of SNF patients. Specifically, Medicare guidelines allow the SNF attending physician up to 72 hours to complete a history and physical (or 7 days if he or she was the hospital attending physician for the index hospitalization) and only require monthly follow-up visits. It is the opinion of the ECP designers that these relatively lax requirements present unnecessary risk for vulnerable patients. While the INTERACT II model was focused largely on educational initiatives (with an advanced practice nurse available in a consultative role, as needed), the central tenet of ECP was similar to the Connected Care model in that the focus was on adding an extra layer of direct clinical support. Protocols that provided timely initial assessments by an NP (within 24 hours), weekly NP rounding (at a minimum), and 24/7 on-call availability all contributed to helping patients stay on track. Although the ECP had patients visited less frequently than the Connected Care model, and the Cleveland Clinic started with a higher baseline 30-day readmission rate from SNFs, similar overall reductions in 30-day readmissions were observed. The key point from both initiatives is that an increase in clinical touchpoints and ease of access to clinicians generates myriad opportunities to identify and address small issues before they become clinical emergencies requiring hospital transfers and readmissions.
Correcting medication discrepancies between hospital discharge summaries and SNF admission orders through a systematic medication reconciliation using a clinical pharmacist has previously been shown to improve outcomes.16-18 The ECP pharmacy technician and ECP clinical pharmacist discovered and corrected errors on a daily basis that ranged from incidental to potentially life-threatening. If the SNF staff does not provide the patient’s MAR within 48 hours of arrival, the pharmacy technician contacts the facility to obtain the information. As a result, all patients enrolled in the ECP during the study period received this intervention (unless they were rehospitalized or left the SNF before the process was completed), and 54% of ECP patients required some form of intervention after medication reconciliation was completed (data not shown).
This type of program requires hospital leadership and SNF administrators to be fully committed to developing strong working relationships, and in fact, there is evidence that SNF baseline readmission rates have a greater influence on patients’ risk of rehospitalization than the discharging hospital itself.19-21 Monthly educational in-services are delivered at the partner SNFs to enhance SNF nursing staff knowledge and clinical acumen. High-impact topics identified by the ECP team include the following: fall prevention, hand hygiene, venous thromboembolism, cardiovascular health, how to report change in condition, and advanced care planning, among others. While no formal pre–post assessments of the SNF nurses’ knowledge were conducted, a log of in-services was kept, subjective feedback was collected for performance improvement purposes, and continuing educational units were provided to the SNF nurses who attended.
This study has limitations. As a single-hospital study, generalizability may be limited. While adherence to the program components was closely monitored daily, service gaps may have occurred that were not captured. The program design makes it difficult to quantify the relative impact of the 3 program components on the outcome. Furthermore, the study was observational, so the differences in readmission rates may have been due to unmeasured variables. The decision to enroll patients in the ECP was made by each patient’s SNF attending physician, and those who chose to (or not to) participate in the program may manifest other, unmeasured practice patterns that made readmissions more or less likely. Participating physicians also had the option to enroll their patients on a case-by-case basis, introducing further potential bias in patient selection; however, <5% of physicians exercised this option. Patients may have also been readmitted to hospitals other than CSMC, producing an observed readmission rate for 1 or both groups that underrepresents the true outcome. On this point, while we did not systematically track these other-hospital readmissions for both groups, there is no reason to believe that this occurred preferentially for ECP or non-ECP patients.
Multiple sensitivity analyses were performed to address the observed differences between ECP and non-ECP patients. These included stratified examinations of variables differing between populations, examination of clustering effects between SNFs, and an analysis adjusted for the propensity to be included in the ECP. The calculated effect of the intervention on readmission remained robust, although we acknowledge that differences in the populations may persist and have influenced the outcomes even after controlling for multiple variables.22-25
In conclusion, the results of this intervention are compelling and add to the growing body of literature suggesting that a comprehensive, multipronged effort to enhance clinical oversight and coordination of care for SNF patients can improve outcomes. Given CMS’s plans to report SNF readmission rates in 2017 followed by the application of financial incentives in 2018, a favorable climate currently exists for greater coordination between hospitals and SNFs.26 We are currently undertaking an economic evaluation of the program.
Acknowledgments
The authors would like to thank the following people for their contributions: Mae Saunders, Rita Shane, Dr. Jon Kea, Miranda Li, the ECP NPs, the ECP pharmacy team, CSMC’s performance improvement team, and Alan Matus.
Disclosure
No conflicts of interest or disclosures.
1. Centers for Medicare & Medicaid Services (CMS), HHS. Medicare Program; Prospective Payment System and Consolidated Billing for Skilled Nursing Facilities (SNFs) for FY 2016, SNF Value-Based Purchasing Program, SNF Quality Reporting Program, and Staffing Data Collection. Final Rule. Fed Regist. 2015;80(149):46389-46477. PubMed
2. “Readmissions Reduction Program,” Centers for Medicare & Medicaid Services. http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed November 5, 2015.
3. Naylor MD, Brooten D, Campbell R, et al. Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial. JAMA. 1999;281:613-620. PubMed
4. Naylor MD, Brooten DA, Campbell RL, Maislin G, McCauley KM, Schwartz JS. Transitional care of older adults hospitalized with heart failure: a randomized, controlled trial. J Am Geriatr Soc. 2004;52:675-684. PubMed
5. Coleman EA, Parry C, Chalmers S, Min SJ. The care transitions intervention: results of a randomized controlled trial. Arch Intern Med. 2006;166:1822-1828. PubMed
6. CMS Office of Information Products and Data Analytics. National Medicare Readmission Findings: Recent Data and Trends. 2012. http://www.academyhealth.org/files/2012/sunday/brennan.pdf. Accessed on September 21, 2015.
7. Centers for Medicare & Medicaid Services, CMS Innovation Center. Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents. https://innovation.cms.gov/initiatives/rahnfr/. Accessed on November 5, 2015.
8. Unroe KT, Nazir A, Holtz LR, et al. The Optimizing Patient Transfers, Impacting Medical Quality and Improving Symptoms: Transforming Institutional Care Approach: Preliminary data from the implementation of a Centers for Medicare and Medicaid Services nursing facility demonstration project. J Am Geriatr Soc. 2015;65:165-169. PubMed
9. Ingber MJ, Feng Z, Khatstsky G, et al. Evaluation of the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents: Final Annual Report Project Year 3. Waltham, MA: RTI International, RTI Project Number 0212790.006, January 2016.
10. Ouslander JG, Lamb G, Tappen R, et al. Interventions to reduce hospitalizations from nursing homes: Evaluation of the INTERACT II collaborative quality improvement project. J Am Geriatr Soc. 2011:59:745-753. PubMed
11. Kim L, Kou L, Hu B, Gorodeski EZ, Rothberg M. Impact of a Connected Care Model on 30-Day Readmission Rates from Skilled Nursing Facilities. J Hosp Med. 2017;12:238-244. PubMed
12. Kansagara D, Englander H, Salanitro A, et al. Risk Prediction Models for Hospital Readmission: A Systematic Review. JAMA. 2011;306(15):1688-1698. PubMed
13. Averill RF, Goldfield N, Hughes JS, et al. All Patient Refined Diagnosis Related Groups (APR-DRGs): Methodology Overview. 3M Health Information Systems Document GRP-041 (2003). https://www.hcup-us.ahrq.gov/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf. Accessed on November 5, 2015.
14. StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.
15. Cebul RD, Rebitzer JB, Taylor LJ, Votruba ME. Organizational fragmentation and care quality in the U.S. healthcare system. J Econ Perspect. 2008;22(4):93-113. PubMed
16. Tjia J, Bonner A, Briesacher BA, McGee S, Terrill E, Miller K. Medication discrepancies upon hospital to skilled nursing facility transitions. J Gen Intern Med. 2009;24:630-635. PubMed
17. Desai R, Williams CE, Greene SB, Pierson S, Hansen RA. Medication errors during patient transitions into nursing homes: characteristics and association with patient harm. Am J Geriatr Pharmacother. 2011;9:413-422. PubMed
18. Chhabra PT, Rattinger GB, Dutcher SK, Hare ME, Parsons KL, Zuckerman IH. Medication reconciliation during the transition to and from long-term care settings: a systematic review. Res Social Adm Pharm. 2012;8(1):60-75. PubMed
19. Rahman M, Foster AD, Grabowski DC, Zinn JS, Mor V. Effect of hospital-SNF referral linkages on rehospitalization. Health Serv Res. 2013;48(6, pt 1):1898-1919. PubMed
20. Schoenfeld AJ, Zhang X, Grabowski DC, Mor V, Weissman JS, Rahman M. Hospital-skilled nursing facility referral linkage reduces readmission rates among Medicare patients receiving major surgery. Surgery. 2016;159(5):1461-1468. PubMed
21. Rahman M, McHugh J, Gozalo P, Ackerly DC, Mor V. The Contribution of Skilled Nursing Facilities to Hospitals’ Readmission Rate. HSR: Health Services Research. 2017;52(2):656-675. PubMed
22. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. New Engl J Med. 2009;360(14):1418-1428. PubMed
23. Hasan O, Meltzer DO, Shaykevich SA, et al. Hospital readmission in general medicine patients: a prediction model. J Hosp Med. 2010;25(3)211-219. PubMed
24. Allaudeen N, Vidyarhi A, Masella J, Auerbach A. Redefining readmission risk factors for general medicine patients. J Hosp Med. 2011;6(2):54-60. PubMed
25. Van Walraven C, Wong J, Forster AJ. LACE+ index: extension of a validated index to predict early death or urgent readmission after discharge using administrative data. Open Med. 2012;6(3):e80-e90. PubMed
26. Protecting Access to Medicare Act of 2014, Pub. L. No. 113-93, 128 Stat. 1040 (April 1, 2014). https://www.congress.gov/113/plaws/publ93/PLAW-113publ93.pdf. Accessed on October 3, 2015.
1. Centers for Medicare & Medicaid Services (CMS), HHS. Medicare Program; Prospective Payment System and Consolidated Billing for Skilled Nursing Facilities (SNFs) for FY 2016, SNF Value-Based Purchasing Program, SNF Quality Reporting Program, and Staffing Data Collection. Final Rule. Fed Regist. 2015;80(149):46389-46477. PubMed
2. “Readmissions Reduction Program,” Centers for Medicare & Medicaid Services. http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed November 5, 2015.
3. Naylor MD, Brooten D, Campbell R, et al. Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial. JAMA. 1999;281:613-620. PubMed
4. Naylor MD, Brooten DA, Campbell RL, Maislin G, McCauley KM, Schwartz JS. Transitional care of older adults hospitalized with heart failure: a randomized, controlled trial. J Am Geriatr Soc. 2004;52:675-684. PubMed
5. Coleman EA, Parry C, Chalmers S, Min SJ. The care transitions intervention: results of a randomized controlled trial. Arch Intern Med. 2006;166:1822-1828. PubMed
6. CMS Office of Information Products and Data Analytics. National Medicare Readmission Findings: Recent Data and Trends. 2012. http://www.academyhealth.org/files/2012/sunday/brennan.pdf. Accessed on September 21, 2015.
7. Centers for Medicare & Medicaid Services, CMS Innovation Center. Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents. https://innovation.cms.gov/initiatives/rahnfr/. Accessed on November 5, 2015.
8. Unroe KT, Nazir A, Holtz LR, et al. The Optimizing Patient Transfers, Impacting Medical Quality and Improving Symptoms: Transforming Institutional Care Approach: Preliminary data from the implementation of a Centers for Medicare and Medicaid Services nursing facility demonstration project. J Am Geriatr Soc. 2015;65:165-169. PubMed
9. Ingber MJ, Feng Z, Khatstsky G, et al. Evaluation of the Initiative to Reduce Avoidable Hospitalizations among Nursing Facility Residents: Final Annual Report Project Year 3. Waltham, MA: RTI International, RTI Project Number 0212790.006, January 2016.
10. Ouslander JG, Lamb G, Tappen R, et al. Interventions to reduce hospitalizations from nursing homes: Evaluation of the INTERACT II collaborative quality improvement project. J Am Geriatr Soc. 2011:59:745-753. PubMed
11. Kim L, Kou L, Hu B, Gorodeski EZ, Rothberg M. Impact of a Connected Care Model on 30-Day Readmission Rates from Skilled Nursing Facilities. J Hosp Med. 2017;12:238-244. PubMed
12. Kansagara D, Englander H, Salanitro A, et al. Risk Prediction Models for Hospital Readmission: A Systematic Review. JAMA. 2011;306(15):1688-1698. PubMed
13. Averill RF, Goldfield N, Hughes JS, et al. All Patient Refined Diagnosis Related Groups (APR-DRGs): Methodology Overview. 3M Health Information Systems Document GRP-041 (2003). https://www.hcup-us.ahrq.gov/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf. Accessed on November 5, 2015.
14. StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.
15. Cebul RD, Rebitzer JB, Taylor LJ, Votruba ME. Organizational fragmentation and care quality in the U.S. healthcare system. J Econ Perspect. 2008;22(4):93-113. PubMed
16. Tjia J, Bonner A, Briesacher BA, McGee S, Terrill E, Miller K. Medication discrepancies upon hospital to skilled nursing facility transitions. J Gen Intern Med. 2009;24:630-635. PubMed
17. Desai R, Williams CE, Greene SB, Pierson S, Hansen RA. Medication errors during patient transitions into nursing homes: characteristics and association with patient harm. Am J Geriatr Pharmacother. 2011;9:413-422. PubMed
18. Chhabra PT, Rattinger GB, Dutcher SK, Hare ME, Parsons KL, Zuckerman IH. Medication reconciliation during the transition to and from long-term care settings: a systematic review. Res Social Adm Pharm. 2012;8(1):60-75. PubMed
19. Rahman M, Foster AD, Grabowski DC, Zinn JS, Mor V. Effect of hospital-SNF referral linkages on rehospitalization. Health Serv Res. 2013;48(6, pt 1):1898-1919. PubMed
20. Schoenfeld AJ, Zhang X, Grabowski DC, Mor V, Weissman JS, Rahman M. Hospital-skilled nursing facility referral linkage reduces readmission rates among Medicare patients receiving major surgery. Surgery. 2016;159(5):1461-1468. PubMed
21. Rahman M, McHugh J, Gozalo P, Ackerly DC, Mor V. The Contribution of Skilled Nursing Facilities to Hospitals’ Readmission Rate. HSR: Health Services Research. 2017;52(2):656-675. PubMed
22. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. New Engl J Med. 2009;360(14):1418-1428. PubMed
23. Hasan O, Meltzer DO, Shaykevich SA, et al. Hospital readmission in general medicine patients: a prediction model. J Hosp Med. 2010;25(3)211-219. PubMed
24. Allaudeen N, Vidyarhi A, Masella J, Auerbach A. Redefining readmission risk factors for general medicine patients. J Hosp Med. 2011;6(2):54-60. PubMed
25. Van Walraven C, Wong J, Forster AJ. LACE+ index: extension of a validated index to predict early death or urgent readmission after discharge using administrative data. Open Med. 2012;6(3):e80-e90. PubMed
26. Protecting Access to Medicare Act of 2014, Pub. L. No. 113-93, 128 Stat. 1040 (April 1, 2014). https://www.congress.gov/113/plaws/publ93/PLAW-113publ93.pdf. Accessed on October 3, 2015.
© 2018 Society of Hospital Medicine
Personalized Vascular Access Training
The Accreditation Counsel for Graduate Medical Education (ACGME) states in its Program Requirements for Residency Education in Internal Medicine that all residents must develop technical proficiency in several procedures, including central venous line placement.1 Developing competency in common procedural skills has long been a part of medical training. The philosophy of see‐1, do‐1, teach‐1 is still the most common means by which most residents seek to obtain this proficiency, even though serious concerns have been raised about this approach.2 A typical first experience in central line placement usually involves an eager (or terrified) trainee making several clumsy attempts on an actual patient, under the hurried guidance of a senior resident who themselves received an unknown degree of training. In this scenario, rarely does standardized instruction, formal evaluation, or structured follow‐up occur.
A revitalized emphasis is now being placed on patient safety in healthcare, including an industry‐wide commitment to minimizing procedural complications. The most common complications associated to central line placement include vascular damage and catheter‐related bloodstream infections. A number of creative approaches are being developed to improve the quality of instruction on proper procedural techniques, all varying considerably in sophistication, scope, and rigor. Examples include the use of computer‐assisted methods for training ultrasound‐guided needle insertion techniques and ureteroscopy training, hands‐on training with synthetic models for thoracentesis training, video training, and uterine aspiration using papayas.311 Implicit in this trend is recognition that we, as educators, healthcare providers, and patient advocates, must design more cost effective and efficient ways to teach medical and surgical procedural techniques to clinicians.
Our approach was previously described in phase I of the Procedural Patient Safety Initiative (PPSI).12 In PPSI‐I, we introduced a nonhuman tissue model (NHTM; Figure 1) as the basis for teaching physicians a more rigorous curriculum of essential central line placement skills. By way of brief review, the NHTMs were constructed by tunneling 0.2‐mm‐thick rubber tubing (vessels) lengthwise through raw, whole chickens purchased at the grocery store. The vessels were filled with colored water to simulate blood. The NHTM has several unique features, including: (1) realistic‐appearing vessels when viewed under ultrasound, which mimic the appearance of human internal jugular veins and carotid arteries (Figure 2); (2) tissue turgor and vessel composition that produce realistic pops and flashes upon puncture and allow for multiple cannulations; (3) the ability to perform a complete central line placement (including wire advancement, dilation, line insertion, suturing, and sterile dressing placement); (4) cost effectiveness relative to other commercially‐available products (each NHTM costs $120 and can withstand multiple cannulations over 2 days).1317 During the training sessions of Phase I, participants were oriented to the ultrasound machines, shown the contents of our central line kit, and taught the principles of wide sterile barriers (WSBs), sharps safety, and vascular access under real‐time ultrasound guidance. A self‐completed survey tool was filled out by the participants before and after the session that contained questions about their precourse baseline procedural experience, and their subjective comfort level with specific skills after the course. The results of our intervention, as measured by the responses to the survey, were significantly positive. We recognized the limitations of these results based on using subjective criteria to measure efficacy, a lack of follow‐up on participants' skill retention, and with no ultimate evaluation of procedural competency evaluations on actual patients (compared to an untrained control group).
Our ultimate goal is to validate a curriculum that will give trainees the necessary education and skills that enables them to make a smooth, competent, and complication‐free transition to live patient procedures. Phase II of PPSI is our next step toward this goal. In this study, we sought to measure the impact of intensive, 1:1 central line placement training with a proceduralist, objectively validate the efficacy of the NHTM and our training curriculum using a standardized 6‐point scoring scale and skilled evaluators, and to evaluate the degree of skill retention over time (decay). Our hypothesis was that the depth of skills' imprinting from a single, standardized training session would result in a significant improvement in measured procedure skills immediately after the trainee is taught the skills, and that the retention of these skills would be demonstrable when participants were reevaluated at a future date.
Methods
PPSI‐II was an observational, prospective study conducted by The Procedure Center at Cedars‐Sinai Medical Center, a 900+‐bed, community‐based teaching hospital. The Procedure Center is staffed by dedicated Proceduralists who perform a number of common medical procedures on a daily basis and are facile with both real‐time ultrasound guidance and proper procedural techniques.18, 19 Our target population was the incoming Internal Medicine residents. Subjects were recruited by email prior to orientation week and were offered the option of participating in our study. Our only exclusion criterion was the prior observation or placement of 10 or more central lines. The study was approved by our hospital's Institutional Review Board prior to initiating recruitment. Those who chose not to participate underwent the standard orientation training required by our institution, which included a brief overview lecture on the topic of central lines and ultrasound‐guidance, a group viewing of the New England Journal of Medicine (NEJM) video on central line placement,20 and small‐group (4 participants/group) hands‐on practice sessions lasting 45 minutes with NHTMs and a trained Proceduralist.
All of the evaluations for Phase II were done using the Central Line Placement Skill Assessment Tool depicted in Figure 3. This tool, which was developed by Cedars‐Sinai Proceduralists solely for the purposes of this study, is a comprehensive step‐by‐step checklist delineating the specific steps necessary to place a sterile, ultrasound‐guided central venous catheter. It was closely derived from a central line insertion checklist that was created by the Procedure Center 3 years ago to help guide novice clinicians through the procedure, and has since been widely used throughout the institution during the placement of hundreds of central lines. The scoring system, also devised by Cedars‐Sinai Proceduralists, was based on over 15 years of experience supervising and teaching hundreds of residents on proper central line insertion techniques. It consists of clear definitions for each score that were agreed upon via consensus amongst study coordinators. Prior to any evaluations being conducted, we put our 2 evaluators (both senior medicine residents) through identical and simultaneous scoring training with the Proceduralist trainer to standardize procedural knowledge and scoring methodology.
A total of 20 incoming interns (trainees) out of a possible 54 invitations (37%) volunteered to participate. Each trainee was randomly assigned a number from 1 to 20. The study began for each trainee with a brief, 5‐question survey to determine their prior procedural experience (Table 1). Next, each trainee watched the NEJM online training video on central line placement.20 They were then brought into a training room that contained an NHTM sitting on a Mayo stand, an ultrasound machine, and all the materials required to place a central line insertion under ultrasound‐guidance. The trainee's baseline central line insertion skills were evaluated on 22 unique procedure steps, with each score being given by 1 of the 2 evaluators (initial evaluation). The trainee did not receive any guidance or suggestions during this initial evaluation unless the trainee reached an impasse. In these cases, the evaluator completed that single step on the trainee's behalf and then allowed the session to resume. The identity of the evaluator was indicated on each evaluation form, and after each of the evaluations the completed assessment tool was given to our blinded assistant for data recording.
| Number of trainees | 20 |
| How many prior central lines have you inserted independently? (exclusion criteria: >10) | 20 answered 0 |
| How many prior central line insertions have you assisted with? (exclusion criteria: >10) | 13 answered 0; 7 had assisted between 1 and 4 lines |
| How many prior central line insertions have you observed? (exclusion criteria: >10) | 3 answered 0; 17 had observed between 1 and 5 lines |
| Have you had any prior exposure to the use of ultrasound for central line insertion? | 13 no; 7 yes |
| Have you had any prior exposure to the use of wide sterile barriers for procedures? | 11 no; 9 yes |
Each trainee was then given a personalized, hands‐on training session by a proceduralist, using the checklist as a guide to take them through all the steps of a central line insertion. The trainee was allowed to observe and practice each skill for an unlimited period of time with the proceduralist present, until he or she demonstrated competency and felt confident enough with their independent skills (in both trainee's and proceduralist's judgment) to move forward. The entire session ended only when all steps had been taught and practiced to the proceduralist's satisfaction, the trainee felt comfortable independently performing each step (and in proper sequence), and all questions had been answered. At no time was there an imposition of time constraints or external pressure from study coordinators.
Immediately following this training session the proceduralist and trainee left the room, the procedure room was reset by an evaluator (taking approximately 5‐10 minutes), and then the trainee submitted themselves to an immediate posttraining evaluation (immediate evaluation). As with the initial assessment, the evaluator did not interfere or make any comments or suggestions during the evaluation periods, unless the trainee reached an impasse at any step. In that case, the trainee would receive a 0 for that step, the evaluator would assist them to complete that step only, and then the session would continue. No time limits were imposed.
The final part of the study required each trainee to return for follow‐up assessment (delayed evaluation), a process that was identical to the immediate posttraining evaluation. This delayed evaluation was intended to occur between 3 to 4 weeks after the immediate posttraining session, based on trainees' schedules and availability. No refresher or practice time was permitted prior to the delayed evaluation: upon arrival, trainees wrote down on a separate piece of paper (not seen by the evaluator) the number of interim line experiences they had experienced, then they were brought directly into a fully‐prepared room, and instructed to begin. The evaluator was also blind to the trainee's scores from the 2 previous evaluation sessions.
The primary endpoints were the degree of changes in overall average scores (from the 22 steps on the assessment tool) from the initial to the immediate evaluations and from the immediate to delayed evaluations. The secondary endpoints were also based on changes in average scores from the initial to immediate and immediate to delayed evaluations, and looked at 5 essential elements (steps in the assessment tool that we deemed critical to the safe and successful placement of a central line). These essential elements included (1) hand washing; (2) creation of a WSB; (3) ultrasound‐guided vessel cannulation; (4) proper catheter placement; and (5) sharps safety. Of note, the creation of a WSB element consisted of 4 steps, each of which was analyzed separately. The average scores are reported as means standard deviations (SDs).
To determine the type of analysis that would be performed, we started by assessing the changes using paired t tests. The Kolmogorov‐Smirnov and Anderson‐Darling normality tests revealed no evidence of violations of the normality assumption, confirming that using paired t tests was valid.
To address potential contamination from residents' real experiences on the rate of their knowledge decay between the immediate evaluation and delayed follow‐up, each participant completed a brief survey before their delayed evaluation asking about interim experiences. All calculations were performed including and excluding from participants' scores with affirmative answers to control for this contamination. Last, a post‐hoc analysis was performed on participants' scores using a scatterplot and statistical analyses to control for the varying time‐to‐follow‐up.
Results
All 20 individuals completed the study, for a total of 60 evaluations (20 each of initial, immediate, and delayed). The actual training time (not including the viewing of the video) ranged between 45 to 120 minutes, depending on the trainee. Our primary endpoints are depicted in Table 2. The mean overall score on the initial evaluation was 1.0 0.8. The mean overall score for the immediate posttraining evaluation was 4.4 0.3. This improvement of 3.4 points was significant (P < 0.001; 95% CI, 3.0‐3.7). The delayed evaluations took place an average of 22 days after the training session (range, 5‐101 days), and produced an overall mean score of 4.2 0.3. This decay of 0.2 was not significant (P = 0.14; 95% CI, 0.31 to 0.05). With regard to the amount of skills decay, additional calculations were performed from the scatterplot that depicted scores and the variability in time‐to‐follow‐up. We found that even after controlling this variable, the amount of decay for the overall score remained insignificant.
| ||
| Mean (SD) score of initial (baseline) evaluation | 1.0 (0.80) | |
| Mean (SD) score of immediate posttraining (baseline) evaluation | 4.4 (0.30) | |
| Average change between initial and immediate posttraining scores | +3.4 | P < 0.001; CI, 3.0‐3.7 |
| Mean (SD) score of delayed posttraining evaluation | 4.2 (0.32) | |
| Change between immediate and delayed posttraining scores | 0.2 | P = 0.144; CI, 0.31‐0.05 |
The results of the secondary endpoint calculations (essential elements) are depicted in Table 3. Ultrasound‐guided vessel cannulations improved from an initial average score of 0.9 1.0 to an immediate average score of 4.2 0.5 (P < 0.001; 95% CI, 3.0‐3.7); the delayed score of 4.3 0.6 was statistically unchanged from immediate (P = 0.77; 95% CI, 0.4 to 0.3). Catheter placement skills improved from 1.1 1.1 to 4.2 0.5 (P < 0.001; 95% CI, 2.6‐3.7), and the delayed score of 4.3 0.7 was unchanged from immediate (P < 0.58; 95% CI, 0.5 to 0.3). Sharps safety also improved significantly from initial (2.0 2.3) to immediate (4.9 0.5) (P < 0.0001; 95% CI, 1.9‐3.9), and the delayed scores dropped insignificantly to 4.6 0.8 (P = 0.08; 95% CI, 0.0‐0.6). Hand washing improved significantly from an initial score of 0.9 1.9 to an immediate score of 3.5 2.2 (P < 0.001; 95% CI, 1.4‐3.7), and decayed insignificantly on the delayed evaluation to 3.0 2.3 (P = 0.53; 95% CI, 0.9 to 1.7). WSB skills consisted of 4 individual steps, all of which all improved significantly from initial to immediate scores, and had insignificant decays on the delayed evaluations (see Table 3 WSB for details).
| Initial Evaluation | Immediate Follow‐Up | P Value (Initial to Immediate) | Delayed Follow‐Up | P Value (Immediate to Delayed) | |
|---|---|---|---|---|---|
| |||||
| Ultrasound‐guided insertion of needle into vein (step 15) | 0.9 (1.0) | 4.2 (0.5) | P < 0.001; CI, 3.0‐3.7 | 4.3 ( 0.6) | P = 0.77; CI, 0.4 to 0.3 |
| Catheter placement (step 18) | 1.1 (1.1) | 4.2 (0.5) | P < 0.0001; CI, 2.6‐3.7 | 4.3 ( 0.7) | P = 0.58; CI, 0.5 to 0.3 |
| Sharps safety (step 20) | 2.0 (2.3) | 4.9 (0.5) | P < 0.0001; CI, 1.9‐3.9 | 4.6 ( 0.8) | P = 0.08; CI = 0 to 0.6 |
| Hand washing (step 2) | 0.9 (1.9) | 3.5 (2.2) | P < 0.001; CI, 1.4‐3.7 | 3.0 ( 2.3) | P = 0.53; CI, 0.9 to 1.7 |
| WSBs | |||||
| MD prep (step 3) | 1.8 (1.5) | 4.3 (0.7) | P < 0.0001; CI, 1.7‐3.3 | 4.2 ( 0.6) | P = 0.30; CI, 0.2 to 0.6 |
| Site sterilization (step 7) | 1.1 (1.1) | 4.3 (0.9) | P < 0.0001; CI, 2.7‐3.7 | 4.5 ( 0.5) | P = 0.45; CI, 0.6 to 0.3 |
| WSB creation (step 8) | 0.6 ( 0.6) | 4.1 ( 0.9) | P < 0.0001; CI, 3.0‐4.0 | 4.4 ( 0.6) | P = 0.26; CI, 0.7 to 0.2 |
| Ultrasound probe cover application (step 9) | 0.4 ( 0.9) | 4.1 ( 0.8) | P < 0.0001; CI, 3.2‐4.1 | 4.4 ( 0.8) | P = 0.23; CI, 0.8 to 0.2 |
We performed validation exercises to determine the degree of interrater agreement. Of the 60 total evaluations that were eventually performed, 11 evaluations had been performed simultaneously and independently by evaluators A and B. An analysis of the scores assigned by each evaluator to these 11 trainees revealed a high level of interrater agreement (96%). Further, we performed independent analyses of the trainees' scores as assessed by evaluator A (22 sessions) or evaluator B (27 sessions) across the initial, immediate, and delayed sessions, and we detected no statistical differences in the changes in scores (which mirrored the overall results above).
With regards to real‐life contamination between immediate scores and delayed scores, we identified 3 trainees who had placed central lines on actual patients during the interim period (2 trainees placed 1 line each, and 1 trainee placed 2 lines). We repeated all of the calculations without these participants' delayed scores and determined that the removal of their scores did not change the statistical significance of any of the study results. With regard to knowledge decay, the scatterplot comparing delayed scores to varying time‐to‐follow up revealed no correlation.
Discussion
Our study was designed to determine whether novice trainees could learn and retain proper central line placement skills on the NHTM by receiving personalized training in a relaxed, 1‐on‐1 learning environment. Success was measured by trained evaluators using a detailed evaluation tool with a 6‐point scoring scale. The results of our primary endpoints (changes in overall average scores across the 3 evaluation periods) confirmed that this type of training could quickly improve novice practitioners' skill levels from very low (initial evaluation) to significantly higher (immediate posttraining). The dropoff (decay) in skill levels was found to not be statistically significant over a period of several weeks, although we recognize that further study should be performed to establish the degree of skill decay over a longer period of time.
Because some steps in a central line insertion are more critical to the procedure's success than others (ie, a skin nick with a scalpel is less critical than vessel cannulation under ultrasound‐guidance), we analyzed 5 essential elements individually as secondary endpoints. This secondary analysis was designed to unmask any critical skill deficiencies that might otherwise have been lost in the overall analysis. For each individual essential elements step, this subanalysis similarly revealed a significant improvement from initial to immediate posttraining, and an insignificant score decay on the delayed evaluation.
We recognize a number of limitations to this study. First, the n is relatively small. A larger sample size would have allowed for greater statistical power. In addition, the scoring system used for this study was created by our Procedure Center staff and had never been truly validated elsewhere. The scoring system was transparent and logical, but we recognize that any attempt to use an interval scoring system to quantify procedural skills will be inherently imperfect; the difference between 1 and 2 is not necessarily the same as a difference between 4 and 5. Great efforts were taken to mitigate the impact of this limitation: explicit definitions were established for each score, and we put our evaluators through a rigorous scoring orientation at the outset to standardize their interpretation and use of the scoring system and assessment tool.
The variability in the amount of training time spent in each session could be considered to be a confounder. Our prior experiences training interns in small groups, however, suggested that individuals learn these skills at different paces and in different ways, and so we consider our customized approach to be an essential part of this training experience. We do recognize the practical limitations inherent in rolling out such an open‐ended approach, and program directors may face time and/or resource limitations if attempting to replicate this training strategy.
We were also aware of potential interrater variability between the evaluators. Our approach to addressing this was multifactorial: we went to great lengths to standardize evaluators' understanding of the intended scoring methodology prior to the initiation of the study. We also assessed the degree of interrater reliability once all data was collected. This analysis reinforced that both evaluators were scoring trainees in a virtually identical fashion. We attribute this consistency to the quality of the scoring system, the effectiveness of the prestudy evaluator orientation with a proceduralist, and the high degree of teamwork between the 2 evaluators that kept them closely in sync with one another throughout the study.
Evaluator bias was also a concern. While each evaluator was blinded to the trainees' prior scores, the setup associated with the different training sessions, as well as the obvious differences in performance between the trainees' initial and immediate/delayed performances, made full blinding of the evaluators difficult. The theoretical risk of evaluator bias in this study would have led to evaluators rating trainees higher in the immediate and delayed performances in order to demonstrate more dramatic results. We believe that, since the evaluators themselves did not perform the actual training, and since they did not know the previous scores for the trainee, they were less inclined to skew the scores. Video recording each performance and submitting this recording to a fully‐blinded, third‐party evaluator would have more rigorously ensured blinding than we were able to accomplish. This approach could be considered in future studies of this type.
An addition limitation involved the time‐to‐follow‐up. While a longer time interval between the immediate and delayed evaluations may have better evaluated the impact of the training and potential decay, we sought to balance this with the growing risk of contamination from real central line placement experiences as more time passed. With this issue in mind, the removal of the delayed scores from the 3 trainees who had placed central lines on actual patients in between the immediate and delayed evaluations (2 trainees placed 1 line each, and 1 trainee placed 2 lines) did not change the statistical significance of any of the study results.
One practical concern has to do with the reproducibility of this approach at other institutions. Each trainee received up to 2 hours of individualized attention, and each session consumed fresh supplies and required a proceduralist's and an evaluator's time. This represents a significant commitment of materials and manpower. A careful cost/benefit analysis is therefore warranted before implementing this kind of rigorous training program. As mentioned, the cost of the NHTM is approximately $120 and can withstand several cannulations over a 2‐day period; the sterile supplies and central line add up to approximately another $75/evaluation. Depending on the number of interns and residents at a given institution, these costs could prove prohibitive to cash‐poor residency training programs. In the larger picture, however, catheter‐related bloodstream infections have been estimated to result in a mortality rate of 4% to 20%, and a single catheter‐related bloodstream infection can cost up to $45,000.2124 In addition, new Medicare reimbursement policies are now beginning to limit hospital reimbursement for these types of iatrogenic events; hence, narrowing the margin of error and putting even greater financial pressures on hospitals.25 It is our belief, therefore, that an up‐front investment in NHTMs (or an alternative simulator), basic supplies, and the necessary trainer time will prove to be cost‐effective and enlightened investments from forward‐thinking leadership.
Last, we are also aware that our study did not look at whether our trainees' improved performance on the NHTM actually translated into better patient outcomes. Since patient safety is our ultimate goal, and this phase of PPSI limited all of our training and evaluations to the NHTMs, future studies must ultimately evaluate how well these learned skills translate into procedure performance on actual patients. This controlled study (possibly with a see‐1 do‐1 teach‐1 control group) will be logistically challenging, but will be the most definitive manner with which to demonstrate the true value of personalized training sessions using the NHTM (or another nonhuman simulator).
PPSI‐II demonstrated that using the NHTM as the basis for training novice practitioners in a personalized, 1‐on‐1 training session led to significant improvements in measured procedural skills. Further, these skills were retained over time. This positive study contributes to the growing body of literature pointing towards the role of intensive 1‐on‐1 training with simulators to advance procedural education for clinicians. Ultimately, we aim to demonstrate that providing trainees this type of training prior to having them perform procedures on actual patients will translate into superior patient care, greater success rates, fewer minor and major complications, and lower overall patient care costs. Rather than clinging to the classic but never‐validated see‐1, do‐1, teach‐1 approach, we believe that procedural training must adapt new curricula and technologies that will help us achieve the goals of maximizing the safety and quality of care for our patients.
Acknowledgements
The authors recognize and appreciate the entire staff of The Procedure Center at Cedars‐Sinai Medical Center for their support of this research project. The authors give special thanks to Obed Martinez for his tireless assistance with the scheduling and coordination of training activities, and to Jim Mirocha for statistical analysis and editorial contributions.
- Accreditation Council for Graduate Medical Education (ACGME). Home page. Available at: http://www.acgme.org. Accessed June2009.
- ,,.Procedural training at a crossroads: striking a balance between education, patient safety and quality.J Hosp Med.2007;2(3):123–125.
- ,,,,,.Firm‐based trial to improve central venous catheter insertion practices.J Hosp Med.2007;2(3):135–142.
- ,,,,,Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice.J Hosp Med.2008;3(1):48–54.
- ,,.Central line simulation: a new training algorithm.Am Surg.2007;73:680–682.
- ,,, et al.Developing technical expertise in emergency medicine—the role of simulation in procedural skill acquisition.Acad Emerg Med.2008;15:1046–1057.
- ,,,.A training system for ultrasound‐guided needle insertion procedures.Med Image Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv.2007;10(1):566–574.
- ,,, et al.Video‐based training increases sterile‐technique compliance during central venous catheter insertion.Crit Care Med.2007;35:1302–1306.
- ,,,,Comparison of results of virtual‐reality simulator and training model for basic ureteroscopy training.J Endourol.2006;20(4):266–271.
- ,.Papaya: a simulation model for training in uterine aspiration.Fam Med.2005;37(4):242–244.
- ,,,,.An intervention to improve procedure education for internal medicine residents.J Gen Intern Med.2008;23(3):288–293.
- ,,.The use of tissue models for vascular access training: phase 1 of the procedural patient safety initiative.J Gen Intern Med.2006;21(5):514–517.
- Blue phantom: CVC hands‐on trainer, items # BPH600f, BPH604HP, BPH600AP. Available at: http://www.bluephantom.com/desktopdefault.aspx?tabid=232. Accessed June2009.
- Simulab Corporation: Central Line Man System. Available at: http://www.simulab.com/product/surgery/open/centralineman‐system. Accessed June2009.
- KyotoKagaku Co., Ltd.: CVC Insertion Simulator. Available at: http://www.kyotokagaku.com/products/detail01/m93u.html. Accessed June2009.
- First Aid Manufacturer CVC Simulator. Available at: http://www.first‐aid‐manufacturer.com/CVC‐Simulator.aspx. Accessed June2009.
- Limbs and Things: Central Venous Catheter Insertion Simulator, part #KKM93UB. Available at: http://www.golimbs.com/products/products.php?sectid=5356(17):1789–1790.
- ,,.Practice #20: proceduralists. The Advisory Board Annual Report.2007:162–169.
- NEJM video. Available at: http://content.nejm.org/cgi/content/short/356/21/e21. Accessed June2009.
- .Prevention of intravascular catheter‐related infections.Ann Intern Med.2000;132(5):391–402.
- ,,, et al.Guidelines for the prevention of intravascular catheter‐related infections.MMWR Recomm Rep.2002;51(RR‐10):1–29.
- ,,, et al.Clinical and economic outcomes in critically ill patients with nosocomial catheter‐related bloodstream infections.Clin Infect Dis.2005;41:1591–1598.
- ,,,,,.Attributable morbidity and mortality of catheter‐related septicemia in critically ill patients: a matched, risk‐adjusted, cohort study.Infect Control Hosp Epidemiol.1999;20(6):396–401.
- Centers for Medicaid and Medicare Services. U.S. Department of Health and Human Services. Hospital‐Acquired Conditions. Available at: http://www.cms.hhs.gov/HospitalAcqCond/06_Hospital‐Acquired_Conditions.asp#TopOfPage. Accessed June2009.
The Accreditation Counsel for Graduate Medical Education (ACGME) states in its Program Requirements for Residency Education in Internal Medicine that all residents must develop technical proficiency in several procedures, including central venous line placement.1 Developing competency in common procedural skills has long been a part of medical training. The philosophy of see‐1, do‐1, teach‐1 is still the most common means by which most residents seek to obtain this proficiency, even though serious concerns have been raised about this approach.2 A typical first experience in central line placement usually involves an eager (or terrified) trainee making several clumsy attempts on an actual patient, under the hurried guidance of a senior resident who themselves received an unknown degree of training. In this scenario, rarely does standardized instruction, formal evaluation, or structured follow‐up occur.
A revitalized emphasis is now being placed on patient safety in healthcare, including an industry‐wide commitment to minimizing procedural complications. The most common complications associated to central line placement include vascular damage and catheter‐related bloodstream infections. A number of creative approaches are being developed to improve the quality of instruction on proper procedural techniques, all varying considerably in sophistication, scope, and rigor. Examples include the use of computer‐assisted methods for training ultrasound‐guided needle insertion techniques and ureteroscopy training, hands‐on training with synthetic models for thoracentesis training, video training, and uterine aspiration using papayas.311 Implicit in this trend is recognition that we, as educators, healthcare providers, and patient advocates, must design more cost effective and efficient ways to teach medical and surgical procedural techniques to clinicians.
Our approach was previously described in phase I of the Procedural Patient Safety Initiative (PPSI).12 In PPSI‐I, we introduced a nonhuman tissue model (NHTM; Figure 1) as the basis for teaching physicians a more rigorous curriculum of essential central line placement skills. By way of brief review, the NHTMs were constructed by tunneling 0.2‐mm‐thick rubber tubing (vessels) lengthwise through raw, whole chickens purchased at the grocery store. The vessels were filled with colored water to simulate blood. The NHTM has several unique features, including: (1) realistic‐appearing vessels when viewed under ultrasound, which mimic the appearance of human internal jugular veins and carotid arteries (Figure 2); (2) tissue turgor and vessel composition that produce realistic pops and flashes upon puncture and allow for multiple cannulations; (3) the ability to perform a complete central line placement (including wire advancement, dilation, line insertion, suturing, and sterile dressing placement); (4) cost effectiveness relative to other commercially‐available products (each NHTM costs $120 and can withstand multiple cannulations over 2 days).1317 During the training sessions of Phase I, participants were oriented to the ultrasound machines, shown the contents of our central line kit, and taught the principles of wide sterile barriers (WSBs), sharps safety, and vascular access under real‐time ultrasound guidance. A self‐completed survey tool was filled out by the participants before and after the session that contained questions about their precourse baseline procedural experience, and their subjective comfort level with specific skills after the course. The results of our intervention, as measured by the responses to the survey, were significantly positive. We recognized the limitations of these results based on using subjective criteria to measure efficacy, a lack of follow‐up on participants' skill retention, and with no ultimate evaluation of procedural competency evaluations on actual patients (compared to an untrained control group).
Our ultimate goal is to validate a curriculum that will give trainees the necessary education and skills that enables them to make a smooth, competent, and complication‐free transition to live patient procedures. Phase II of PPSI is our next step toward this goal. In this study, we sought to measure the impact of intensive, 1:1 central line placement training with a proceduralist, objectively validate the efficacy of the NHTM and our training curriculum using a standardized 6‐point scoring scale and skilled evaluators, and to evaluate the degree of skill retention over time (decay). Our hypothesis was that the depth of skills' imprinting from a single, standardized training session would result in a significant improvement in measured procedure skills immediately after the trainee is taught the skills, and that the retention of these skills would be demonstrable when participants were reevaluated at a future date.
Methods
PPSI‐II was an observational, prospective study conducted by The Procedure Center at Cedars‐Sinai Medical Center, a 900+‐bed, community‐based teaching hospital. The Procedure Center is staffed by dedicated Proceduralists who perform a number of common medical procedures on a daily basis and are facile with both real‐time ultrasound guidance and proper procedural techniques.18, 19 Our target population was the incoming Internal Medicine residents. Subjects were recruited by email prior to orientation week and were offered the option of participating in our study. Our only exclusion criterion was the prior observation or placement of 10 or more central lines. The study was approved by our hospital's Institutional Review Board prior to initiating recruitment. Those who chose not to participate underwent the standard orientation training required by our institution, which included a brief overview lecture on the topic of central lines and ultrasound‐guidance, a group viewing of the New England Journal of Medicine (NEJM) video on central line placement,20 and small‐group (4 participants/group) hands‐on practice sessions lasting 45 minutes with NHTMs and a trained Proceduralist.
All of the evaluations for Phase II were done using the Central Line Placement Skill Assessment Tool depicted in Figure 3. This tool, which was developed by Cedars‐Sinai Proceduralists solely for the purposes of this study, is a comprehensive step‐by‐step checklist delineating the specific steps necessary to place a sterile, ultrasound‐guided central venous catheter. It was closely derived from a central line insertion checklist that was created by the Procedure Center 3 years ago to help guide novice clinicians through the procedure, and has since been widely used throughout the institution during the placement of hundreds of central lines. The scoring system, also devised by Cedars‐Sinai Proceduralists, was based on over 15 years of experience supervising and teaching hundreds of residents on proper central line insertion techniques. It consists of clear definitions for each score that were agreed upon via consensus amongst study coordinators. Prior to any evaluations being conducted, we put our 2 evaluators (both senior medicine residents) through identical and simultaneous scoring training with the Proceduralist trainer to standardize procedural knowledge and scoring methodology.
A total of 20 incoming interns (trainees) out of a possible 54 invitations (37%) volunteered to participate. Each trainee was randomly assigned a number from 1 to 20. The study began for each trainee with a brief, 5‐question survey to determine their prior procedural experience (Table 1). Next, each trainee watched the NEJM online training video on central line placement.20 They were then brought into a training room that contained an NHTM sitting on a Mayo stand, an ultrasound machine, and all the materials required to place a central line insertion under ultrasound‐guidance. The trainee's baseline central line insertion skills were evaluated on 22 unique procedure steps, with each score being given by 1 of the 2 evaluators (initial evaluation). The trainee did not receive any guidance or suggestions during this initial evaluation unless the trainee reached an impasse. In these cases, the evaluator completed that single step on the trainee's behalf and then allowed the session to resume. The identity of the evaluator was indicated on each evaluation form, and after each of the evaluations the completed assessment tool was given to our blinded assistant for data recording.
| Number of trainees | 20 |
| How many prior central lines have you inserted independently? (exclusion criteria: >10) | 20 answered 0 |
| How many prior central line insertions have you assisted with? (exclusion criteria: >10) | 13 answered 0; 7 had assisted between 1 and 4 lines |
| How many prior central line insertions have you observed? (exclusion criteria: >10) | 3 answered 0; 17 had observed between 1 and 5 lines |
| Have you had any prior exposure to the use of ultrasound for central line insertion? | 13 no; 7 yes |
| Have you had any prior exposure to the use of wide sterile barriers for procedures? | 11 no; 9 yes |
Each trainee was then given a personalized, hands‐on training session by a proceduralist, using the checklist as a guide to take them through all the steps of a central line insertion. The trainee was allowed to observe and practice each skill for an unlimited period of time with the proceduralist present, until he or she demonstrated competency and felt confident enough with their independent skills (in both trainee's and proceduralist's judgment) to move forward. The entire session ended only when all steps had been taught and practiced to the proceduralist's satisfaction, the trainee felt comfortable independently performing each step (and in proper sequence), and all questions had been answered. At no time was there an imposition of time constraints or external pressure from study coordinators.
Immediately following this training session the proceduralist and trainee left the room, the procedure room was reset by an evaluator (taking approximately 5‐10 minutes), and then the trainee submitted themselves to an immediate posttraining evaluation (immediate evaluation). As with the initial assessment, the evaluator did not interfere or make any comments or suggestions during the evaluation periods, unless the trainee reached an impasse at any step. In that case, the trainee would receive a 0 for that step, the evaluator would assist them to complete that step only, and then the session would continue. No time limits were imposed.
The final part of the study required each trainee to return for follow‐up assessment (delayed evaluation), a process that was identical to the immediate posttraining evaluation. This delayed evaluation was intended to occur between 3 to 4 weeks after the immediate posttraining session, based on trainees' schedules and availability. No refresher or practice time was permitted prior to the delayed evaluation: upon arrival, trainees wrote down on a separate piece of paper (not seen by the evaluator) the number of interim line experiences they had experienced, then they were brought directly into a fully‐prepared room, and instructed to begin. The evaluator was also blind to the trainee's scores from the 2 previous evaluation sessions.
The primary endpoints were the degree of changes in overall average scores (from the 22 steps on the assessment tool) from the initial to the immediate evaluations and from the immediate to delayed evaluations. The secondary endpoints were also based on changes in average scores from the initial to immediate and immediate to delayed evaluations, and looked at 5 essential elements (steps in the assessment tool that we deemed critical to the safe and successful placement of a central line). These essential elements included (1) hand washing; (2) creation of a WSB; (3) ultrasound‐guided vessel cannulation; (4) proper catheter placement; and (5) sharps safety. Of note, the creation of a WSB element consisted of 4 steps, each of which was analyzed separately. The average scores are reported as means standard deviations (SDs).
To determine the type of analysis that would be performed, we started by assessing the changes using paired t tests. The Kolmogorov‐Smirnov and Anderson‐Darling normality tests revealed no evidence of violations of the normality assumption, confirming that using paired t tests was valid.
To address potential contamination from residents' real experiences on the rate of their knowledge decay between the immediate evaluation and delayed follow‐up, each participant completed a brief survey before their delayed evaluation asking about interim experiences. All calculations were performed including and excluding from participants' scores with affirmative answers to control for this contamination. Last, a post‐hoc analysis was performed on participants' scores using a scatterplot and statistical analyses to control for the varying time‐to‐follow‐up.
Results
All 20 individuals completed the study, for a total of 60 evaluations (20 each of initial, immediate, and delayed). The actual training time (not including the viewing of the video) ranged between 45 to 120 minutes, depending on the trainee. Our primary endpoints are depicted in Table 2. The mean overall score on the initial evaluation was 1.0 0.8. The mean overall score for the immediate posttraining evaluation was 4.4 0.3. This improvement of 3.4 points was significant (P < 0.001; 95% CI, 3.0‐3.7). The delayed evaluations took place an average of 22 days after the training session (range, 5‐101 days), and produced an overall mean score of 4.2 0.3. This decay of 0.2 was not significant (P = 0.14; 95% CI, 0.31 to 0.05). With regard to the amount of skills decay, additional calculations were performed from the scatterplot that depicted scores and the variability in time‐to‐follow‐up. We found that even after controlling this variable, the amount of decay for the overall score remained insignificant.
| ||
| Mean (SD) score of initial (baseline) evaluation | 1.0 (0.80) | |
| Mean (SD) score of immediate posttraining (baseline) evaluation | 4.4 (0.30) | |
| Average change between initial and immediate posttraining scores | +3.4 | P < 0.001; CI, 3.0‐3.7 |
| Mean (SD) score of delayed posttraining evaluation | 4.2 (0.32) | |
| Change between immediate and delayed posttraining scores | 0.2 | P = 0.144; CI, 0.31‐0.05 |
The results of the secondary endpoint calculations (essential elements) are depicted in Table 3. Ultrasound‐guided vessel cannulations improved from an initial average score of 0.9 1.0 to an immediate average score of 4.2 0.5 (P < 0.001; 95% CI, 3.0‐3.7); the delayed score of 4.3 0.6 was statistically unchanged from immediate (P = 0.77; 95% CI, 0.4 to 0.3). Catheter placement skills improved from 1.1 1.1 to 4.2 0.5 (P < 0.001; 95% CI, 2.6‐3.7), and the delayed score of 4.3 0.7 was unchanged from immediate (P < 0.58; 95% CI, 0.5 to 0.3). Sharps safety also improved significantly from initial (2.0 2.3) to immediate (4.9 0.5) (P < 0.0001; 95% CI, 1.9‐3.9), and the delayed scores dropped insignificantly to 4.6 0.8 (P = 0.08; 95% CI, 0.0‐0.6). Hand washing improved significantly from an initial score of 0.9 1.9 to an immediate score of 3.5 2.2 (P < 0.001; 95% CI, 1.4‐3.7), and decayed insignificantly on the delayed evaluation to 3.0 2.3 (P = 0.53; 95% CI, 0.9 to 1.7). WSB skills consisted of 4 individual steps, all of which all improved significantly from initial to immediate scores, and had insignificant decays on the delayed evaluations (see Table 3 WSB for details).
| Initial Evaluation | Immediate Follow‐Up | P Value (Initial to Immediate) | Delayed Follow‐Up | P Value (Immediate to Delayed) | |
|---|---|---|---|---|---|
| |||||
| Ultrasound‐guided insertion of needle into vein (step 15) | 0.9 (1.0) | 4.2 (0.5) | P < 0.001; CI, 3.0‐3.7 | 4.3 ( 0.6) | P = 0.77; CI, 0.4 to 0.3 |
| Catheter placement (step 18) | 1.1 (1.1) | 4.2 (0.5) | P < 0.0001; CI, 2.6‐3.7 | 4.3 ( 0.7) | P = 0.58; CI, 0.5 to 0.3 |
| Sharps safety (step 20) | 2.0 (2.3) | 4.9 (0.5) | P < 0.0001; CI, 1.9‐3.9 | 4.6 ( 0.8) | P = 0.08; CI = 0 to 0.6 |
| Hand washing (step 2) | 0.9 (1.9) | 3.5 (2.2) | P < 0.001; CI, 1.4‐3.7 | 3.0 ( 2.3) | P = 0.53; CI, 0.9 to 1.7 |
| WSBs | |||||
| MD prep (step 3) | 1.8 (1.5) | 4.3 (0.7) | P < 0.0001; CI, 1.7‐3.3 | 4.2 ( 0.6) | P = 0.30; CI, 0.2 to 0.6 |
| Site sterilization (step 7) | 1.1 (1.1) | 4.3 (0.9) | P < 0.0001; CI, 2.7‐3.7 | 4.5 ( 0.5) | P = 0.45; CI, 0.6 to 0.3 |
| WSB creation (step 8) | 0.6 ( 0.6) | 4.1 ( 0.9) | P < 0.0001; CI, 3.0‐4.0 | 4.4 ( 0.6) | P = 0.26; CI, 0.7 to 0.2 |
| Ultrasound probe cover application (step 9) | 0.4 ( 0.9) | 4.1 ( 0.8) | P < 0.0001; CI, 3.2‐4.1 | 4.4 ( 0.8) | P = 0.23; CI, 0.8 to 0.2 |
We performed validation exercises to determine the degree of interrater agreement. Of the 60 total evaluations that were eventually performed, 11 evaluations had been performed simultaneously and independently by evaluators A and B. An analysis of the scores assigned by each evaluator to these 11 trainees revealed a high level of interrater agreement (96%). Further, we performed independent analyses of the trainees' scores as assessed by evaluator A (22 sessions) or evaluator B (27 sessions) across the initial, immediate, and delayed sessions, and we detected no statistical differences in the changes in scores (which mirrored the overall results above).
With regards to real‐life contamination between immediate scores and delayed scores, we identified 3 trainees who had placed central lines on actual patients during the interim period (2 trainees placed 1 line each, and 1 trainee placed 2 lines). We repeated all of the calculations without these participants' delayed scores and determined that the removal of their scores did not change the statistical significance of any of the study results. With regard to knowledge decay, the scatterplot comparing delayed scores to varying time‐to‐follow up revealed no correlation.
Discussion
Our study was designed to determine whether novice trainees could learn and retain proper central line placement skills on the NHTM by receiving personalized training in a relaxed, 1‐on‐1 learning environment. Success was measured by trained evaluators using a detailed evaluation tool with a 6‐point scoring scale. The results of our primary endpoints (changes in overall average scores across the 3 evaluation periods) confirmed that this type of training could quickly improve novice practitioners' skill levels from very low (initial evaluation) to significantly higher (immediate posttraining). The dropoff (decay) in skill levels was found to not be statistically significant over a period of several weeks, although we recognize that further study should be performed to establish the degree of skill decay over a longer period of time.
Because some steps in a central line insertion are more critical to the procedure's success than others (ie, a skin nick with a scalpel is less critical than vessel cannulation under ultrasound‐guidance), we analyzed 5 essential elements individually as secondary endpoints. This secondary analysis was designed to unmask any critical skill deficiencies that might otherwise have been lost in the overall analysis. For each individual essential elements step, this subanalysis similarly revealed a significant improvement from initial to immediate posttraining, and an insignificant score decay on the delayed evaluation.
We recognize a number of limitations to this study. First, the n is relatively small. A larger sample size would have allowed for greater statistical power. In addition, the scoring system used for this study was created by our Procedure Center staff and had never been truly validated elsewhere. The scoring system was transparent and logical, but we recognize that any attempt to use an interval scoring system to quantify procedural skills will be inherently imperfect; the difference between 1 and 2 is not necessarily the same as a difference between 4 and 5. Great efforts were taken to mitigate the impact of this limitation: explicit definitions were established for each score, and we put our evaluators through a rigorous scoring orientation at the outset to standardize their interpretation and use of the scoring system and assessment tool.
The variability in the amount of training time spent in each session could be considered to be a confounder. Our prior experiences training interns in small groups, however, suggested that individuals learn these skills at different paces and in different ways, and so we consider our customized approach to be an essential part of this training experience. We do recognize the practical limitations inherent in rolling out such an open‐ended approach, and program directors may face time and/or resource limitations if attempting to replicate this training strategy.
We were also aware of potential interrater variability between the evaluators. Our approach to addressing this was multifactorial: we went to great lengths to standardize evaluators' understanding of the intended scoring methodology prior to the initiation of the study. We also assessed the degree of interrater reliability once all data was collected. This analysis reinforced that both evaluators were scoring trainees in a virtually identical fashion. We attribute this consistency to the quality of the scoring system, the effectiveness of the prestudy evaluator orientation with a proceduralist, and the high degree of teamwork between the 2 evaluators that kept them closely in sync with one another throughout the study.
Evaluator bias was also a concern. While each evaluator was blinded to the trainees' prior scores, the setup associated with the different training sessions, as well as the obvious differences in performance between the trainees' initial and immediate/delayed performances, made full blinding of the evaluators difficult. The theoretical risk of evaluator bias in this study would have led to evaluators rating trainees higher in the immediate and delayed performances in order to demonstrate more dramatic results. We believe that, since the evaluators themselves did not perform the actual training, and since they did not know the previous scores for the trainee, they were less inclined to skew the scores. Video recording each performance and submitting this recording to a fully‐blinded, third‐party evaluator would have more rigorously ensured blinding than we were able to accomplish. This approach could be considered in future studies of this type.
An addition limitation involved the time‐to‐follow‐up. While a longer time interval between the immediate and delayed evaluations may have better evaluated the impact of the training and potential decay, we sought to balance this with the growing risk of contamination from real central line placement experiences as more time passed. With this issue in mind, the removal of the delayed scores from the 3 trainees who had placed central lines on actual patients in between the immediate and delayed evaluations (2 trainees placed 1 line each, and 1 trainee placed 2 lines) did not change the statistical significance of any of the study results.
One practical concern has to do with the reproducibility of this approach at other institutions. Each trainee received up to 2 hours of individualized attention, and each session consumed fresh supplies and required a proceduralist's and an evaluator's time. This represents a significant commitment of materials and manpower. A careful cost/benefit analysis is therefore warranted before implementing this kind of rigorous training program. As mentioned, the cost of the NHTM is approximately $120 and can withstand several cannulations over a 2‐day period; the sterile supplies and central line add up to approximately another $75/evaluation. Depending on the number of interns and residents at a given institution, these costs could prove prohibitive to cash‐poor residency training programs. In the larger picture, however, catheter‐related bloodstream infections have been estimated to result in a mortality rate of 4% to 20%, and a single catheter‐related bloodstream infection can cost up to $45,000.2124 In addition, new Medicare reimbursement policies are now beginning to limit hospital reimbursement for these types of iatrogenic events; hence, narrowing the margin of error and putting even greater financial pressures on hospitals.25 It is our belief, therefore, that an up‐front investment in NHTMs (or an alternative simulator), basic supplies, and the necessary trainer time will prove to be cost‐effective and enlightened investments from forward‐thinking leadership.
Last, we are also aware that our study did not look at whether our trainees' improved performance on the NHTM actually translated into better patient outcomes. Since patient safety is our ultimate goal, and this phase of PPSI limited all of our training and evaluations to the NHTMs, future studies must ultimately evaluate how well these learned skills translate into procedure performance on actual patients. This controlled study (possibly with a see‐1 do‐1 teach‐1 control group) will be logistically challenging, but will be the most definitive manner with which to demonstrate the true value of personalized training sessions using the NHTM (or another nonhuman simulator).
PPSI‐II demonstrated that using the NHTM as the basis for training novice practitioners in a personalized, 1‐on‐1 training session led to significant improvements in measured procedural skills. Further, these skills were retained over time. This positive study contributes to the growing body of literature pointing towards the role of intensive 1‐on‐1 training with simulators to advance procedural education for clinicians. Ultimately, we aim to demonstrate that providing trainees this type of training prior to having them perform procedures on actual patients will translate into superior patient care, greater success rates, fewer minor and major complications, and lower overall patient care costs. Rather than clinging to the classic but never‐validated see‐1, do‐1, teach‐1 approach, we believe that procedural training must adapt new curricula and technologies that will help us achieve the goals of maximizing the safety and quality of care for our patients.
Acknowledgements
The authors recognize and appreciate the entire staff of The Procedure Center at Cedars‐Sinai Medical Center for their support of this research project. The authors give special thanks to Obed Martinez for his tireless assistance with the scheduling and coordination of training activities, and to Jim Mirocha for statistical analysis and editorial contributions.
The Accreditation Counsel for Graduate Medical Education (ACGME) states in its Program Requirements for Residency Education in Internal Medicine that all residents must develop technical proficiency in several procedures, including central venous line placement.1 Developing competency in common procedural skills has long been a part of medical training. The philosophy of see‐1, do‐1, teach‐1 is still the most common means by which most residents seek to obtain this proficiency, even though serious concerns have been raised about this approach.2 A typical first experience in central line placement usually involves an eager (or terrified) trainee making several clumsy attempts on an actual patient, under the hurried guidance of a senior resident who themselves received an unknown degree of training. In this scenario, rarely does standardized instruction, formal evaluation, or structured follow‐up occur.
A revitalized emphasis is now being placed on patient safety in healthcare, including an industry‐wide commitment to minimizing procedural complications. The most common complications associated to central line placement include vascular damage and catheter‐related bloodstream infections. A number of creative approaches are being developed to improve the quality of instruction on proper procedural techniques, all varying considerably in sophistication, scope, and rigor. Examples include the use of computer‐assisted methods for training ultrasound‐guided needle insertion techniques and ureteroscopy training, hands‐on training with synthetic models for thoracentesis training, video training, and uterine aspiration using papayas.311 Implicit in this trend is recognition that we, as educators, healthcare providers, and patient advocates, must design more cost effective and efficient ways to teach medical and surgical procedural techniques to clinicians.
Our approach was previously described in phase I of the Procedural Patient Safety Initiative (PPSI).12 In PPSI‐I, we introduced a nonhuman tissue model (NHTM; Figure 1) as the basis for teaching physicians a more rigorous curriculum of essential central line placement skills. By way of brief review, the NHTMs were constructed by tunneling 0.2‐mm‐thick rubber tubing (vessels) lengthwise through raw, whole chickens purchased at the grocery store. The vessels were filled with colored water to simulate blood. The NHTM has several unique features, including: (1) realistic‐appearing vessels when viewed under ultrasound, which mimic the appearance of human internal jugular veins and carotid arteries (Figure 2); (2) tissue turgor and vessel composition that produce realistic pops and flashes upon puncture and allow for multiple cannulations; (3) the ability to perform a complete central line placement (including wire advancement, dilation, line insertion, suturing, and sterile dressing placement); (4) cost effectiveness relative to other commercially‐available products (each NHTM costs $120 and can withstand multiple cannulations over 2 days).1317 During the training sessions of Phase I, participants were oriented to the ultrasound machines, shown the contents of our central line kit, and taught the principles of wide sterile barriers (WSBs), sharps safety, and vascular access under real‐time ultrasound guidance. A self‐completed survey tool was filled out by the participants before and after the session that contained questions about their precourse baseline procedural experience, and their subjective comfort level with specific skills after the course. The results of our intervention, as measured by the responses to the survey, were significantly positive. We recognized the limitations of these results based on using subjective criteria to measure efficacy, a lack of follow‐up on participants' skill retention, and with no ultimate evaluation of procedural competency evaluations on actual patients (compared to an untrained control group).
Our ultimate goal is to validate a curriculum that will give trainees the necessary education and skills that enables them to make a smooth, competent, and complication‐free transition to live patient procedures. Phase II of PPSI is our next step toward this goal. In this study, we sought to measure the impact of intensive, 1:1 central line placement training with a proceduralist, objectively validate the efficacy of the NHTM and our training curriculum using a standardized 6‐point scoring scale and skilled evaluators, and to evaluate the degree of skill retention over time (decay). Our hypothesis was that the depth of skills' imprinting from a single, standardized training session would result in a significant improvement in measured procedure skills immediately after the trainee is taught the skills, and that the retention of these skills would be demonstrable when participants were reevaluated at a future date.
Methods
PPSI‐II was an observational, prospective study conducted by The Procedure Center at Cedars‐Sinai Medical Center, a 900+‐bed, community‐based teaching hospital. The Procedure Center is staffed by dedicated Proceduralists who perform a number of common medical procedures on a daily basis and are facile with both real‐time ultrasound guidance and proper procedural techniques.18, 19 Our target population was the incoming Internal Medicine residents. Subjects were recruited by email prior to orientation week and were offered the option of participating in our study. Our only exclusion criterion was the prior observation or placement of 10 or more central lines. The study was approved by our hospital's Institutional Review Board prior to initiating recruitment. Those who chose not to participate underwent the standard orientation training required by our institution, which included a brief overview lecture on the topic of central lines and ultrasound‐guidance, a group viewing of the New England Journal of Medicine (NEJM) video on central line placement,20 and small‐group (4 participants/group) hands‐on practice sessions lasting 45 minutes with NHTMs and a trained Proceduralist.
All of the evaluations for Phase II were done using the Central Line Placement Skill Assessment Tool depicted in Figure 3. This tool, which was developed by Cedars‐Sinai Proceduralists solely for the purposes of this study, is a comprehensive step‐by‐step checklist delineating the specific steps necessary to place a sterile, ultrasound‐guided central venous catheter. It was closely derived from a central line insertion checklist that was created by the Procedure Center 3 years ago to help guide novice clinicians through the procedure, and has since been widely used throughout the institution during the placement of hundreds of central lines. The scoring system, also devised by Cedars‐Sinai Proceduralists, was based on over 15 years of experience supervising and teaching hundreds of residents on proper central line insertion techniques. It consists of clear definitions for each score that were agreed upon via consensus amongst study coordinators. Prior to any evaluations being conducted, we put our 2 evaluators (both senior medicine residents) through identical and simultaneous scoring training with the Proceduralist trainer to standardize procedural knowledge and scoring methodology.
A total of 20 incoming interns (trainees) out of a possible 54 invitations (37%) volunteered to participate. Each trainee was randomly assigned a number from 1 to 20. The study began for each trainee with a brief, 5‐question survey to determine their prior procedural experience (Table 1). Next, each trainee watched the NEJM online training video on central line placement.20 They were then brought into a training room that contained an NHTM sitting on a Mayo stand, an ultrasound machine, and all the materials required to place a central line insertion under ultrasound‐guidance. The trainee's baseline central line insertion skills were evaluated on 22 unique procedure steps, with each score being given by 1 of the 2 evaluators (initial evaluation). The trainee did not receive any guidance or suggestions during this initial evaluation unless the trainee reached an impasse. In these cases, the evaluator completed that single step on the trainee's behalf and then allowed the session to resume. The identity of the evaluator was indicated on each evaluation form, and after each of the evaluations the completed assessment tool was given to our blinded assistant for data recording.
| Number of trainees | 20 |
| How many prior central lines have you inserted independently? (exclusion criteria: >10) | 20 answered 0 |
| How many prior central line insertions have you assisted with? (exclusion criteria: >10) | 13 answered 0; 7 had assisted between 1 and 4 lines |
| How many prior central line insertions have you observed? (exclusion criteria: >10) | 3 answered 0; 17 had observed between 1 and 5 lines |
| Have you had any prior exposure to the use of ultrasound for central line insertion? | 13 no; 7 yes |
| Have you had any prior exposure to the use of wide sterile barriers for procedures? | 11 no; 9 yes |
Each trainee was then given a personalized, hands‐on training session by a proceduralist, using the checklist as a guide to take them through all the steps of a central line insertion. The trainee was allowed to observe and practice each skill for an unlimited period of time with the proceduralist present, until he or she demonstrated competency and felt confident enough with their independent skills (in both trainee's and proceduralist's judgment) to move forward. The entire session ended only when all steps had been taught and practiced to the proceduralist's satisfaction, the trainee felt comfortable independently performing each step (and in proper sequence), and all questions had been answered. At no time was there an imposition of time constraints or external pressure from study coordinators.
Immediately following this training session the proceduralist and trainee left the room, the procedure room was reset by an evaluator (taking approximately 5‐10 minutes), and then the trainee submitted themselves to an immediate posttraining evaluation (immediate evaluation). As with the initial assessment, the evaluator did not interfere or make any comments or suggestions during the evaluation periods, unless the trainee reached an impasse at any step. In that case, the trainee would receive a 0 for that step, the evaluator would assist them to complete that step only, and then the session would continue. No time limits were imposed.
The final part of the study required each trainee to return for follow‐up assessment (delayed evaluation), a process that was identical to the immediate posttraining evaluation. This delayed evaluation was intended to occur between 3 to 4 weeks after the immediate posttraining session, based on trainees' schedules and availability. No refresher or practice time was permitted prior to the delayed evaluation: upon arrival, trainees wrote down on a separate piece of paper (not seen by the evaluator) the number of interim line experiences they had experienced, then they were brought directly into a fully‐prepared room, and instructed to begin. The evaluator was also blind to the trainee's scores from the 2 previous evaluation sessions.
The primary endpoints were the degree of changes in overall average scores (from the 22 steps on the assessment tool) from the initial to the immediate evaluations and from the immediate to delayed evaluations. The secondary endpoints were also based on changes in average scores from the initial to immediate and immediate to delayed evaluations, and looked at 5 essential elements (steps in the assessment tool that we deemed critical to the safe and successful placement of a central line). These essential elements included (1) hand washing; (2) creation of a WSB; (3) ultrasound‐guided vessel cannulation; (4) proper catheter placement; and (5) sharps safety. Of note, the creation of a WSB element consisted of 4 steps, each of which was analyzed separately. The average scores are reported as means standard deviations (SDs).
To determine the type of analysis that would be performed, we started by assessing the changes using paired t tests. The Kolmogorov‐Smirnov and Anderson‐Darling normality tests revealed no evidence of violations of the normality assumption, confirming that using paired t tests was valid.
To address potential contamination from residents' real experiences on the rate of their knowledge decay between the immediate evaluation and delayed follow‐up, each participant completed a brief survey before their delayed evaluation asking about interim experiences. All calculations were performed including and excluding from participants' scores with affirmative answers to control for this contamination. Last, a post‐hoc analysis was performed on participants' scores using a scatterplot and statistical analyses to control for the varying time‐to‐follow‐up.
Results
All 20 individuals completed the study, for a total of 60 evaluations (20 each of initial, immediate, and delayed). The actual training time (not including the viewing of the video) ranged between 45 to 120 minutes, depending on the trainee. Our primary endpoints are depicted in Table 2. The mean overall score on the initial evaluation was 1.0 0.8. The mean overall score for the immediate posttraining evaluation was 4.4 0.3. This improvement of 3.4 points was significant (P < 0.001; 95% CI, 3.0‐3.7). The delayed evaluations took place an average of 22 days after the training session (range, 5‐101 days), and produced an overall mean score of 4.2 0.3. This decay of 0.2 was not significant (P = 0.14; 95% CI, 0.31 to 0.05). With regard to the amount of skills decay, additional calculations were performed from the scatterplot that depicted scores and the variability in time‐to‐follow‐up. We found that even after controlling this variable, the amount of decay for the overall score remained insignificant.
| ||
| Mean (SD) score of initial (baseline) evaluation | 1.0 (0.80) | |
| Mean (SD) score of immediate posttraining (baseline) evaluation | 4.4 (0.30) | |
| Average change between initial and immediate posttraining scores | +3.4 | P < 0.001; CI, 3.0‐3.7 |
| Mean (SD) score of delayed posttraining evaluation | 4.2 (0.32) | |
| Change between immediate and delayed posttraining scores | 0.2 | P = 0.144; CI, 0.31‐0.05 |
The results of the secondary endpoint calculations (essential elements) are depicted in Table 3. Ultrasound‐guided vessel cannulations improved from an initial average score of 0.9 1.0 to an immediate average score of 4.2 0.5 (P < 0.001; 95% CI, 3.0‐3.7); the delayed score of 4.3 0.6 was statistically unchanged from immediate (P = 0.77; 95% CI, 0.4 to 0.3). Catheter placement skills improved from 1.1 1.1 to 4.2 0.5 (P < 0.001; 95% CI, 2.6‐3.7), and the delayed score of 4.3 0.7 was unchanged from immediate (P < 0.58; 95% CI, 0.5 to 0.3). Sharps safety also improved significantly from initial (2.0 2.3) to immediate (4.9 0.5) (P < 0.0001; 95% CI, 1.9‐3.9), and the delayed scores dropped insignificantly to 4.6 0.8 (P = 0.08; 95% CI, 0.0‐0.6). Hand washing improved significantly from an initial score of 0.9 1.9 to an immediate score of 3.5 2.2 (P < 0.001; 95% CI, 1.4‐3.7), and decayed insignificantly on the delayed evaluation to 3.0 2.3 (P = 0.53; 95% CI, 0.9 to 1.7). WSB skills consisted of 4 individual steps, all of which all improved significantly from initial to immediate scores, and had insignificant decays on the delayed evaluations (see Table 3 WSB for details).
| Initial Evaluation | Immediate Follow‐Up | P Value (Initial to Immediate) | Delayed Follow‐Up | P Value (Immediate to Delayed) | |
|---|---|---|---|---|---|
| |||||
| Ultrasound‐guided insertion of needle into vein (step 15) | 0.9 (1.0) | 4.2 (0.5) | P < 0.001; CI, 3.0‐3.7 | 4.3 ( 0.6) | P = 0.77; CI, 0.4 to 0.3 |
| Catheter placement (step 18) | 1.1 (1.1) | 4.2 (0.5) | P < 0.0001; CI, 2.6‐3.7 | 4.3 ( 0.7) | P = 0.58; CI, 0.5 to 0.3 |
| Sharps safety (step 20) | 2.0 (2.3) | 4.9 (0.5) | P < 0.0001; CI, 1.9‐3.9 | 4.6 ( 0.8) | P = 0.08; CI = 0 to 0.6 |
| Hand washing (step 2) | 0.9 (1.9) | 3.5 (2.2) | P < 0.001; CI, 1.4‐3.7 | 3.0 ( 2.3) | P = 0.53; CI, 0.9 to 1.7 |
| WSBs | |||||
| MD prep (step 3) | 1.8 (1.5) | 4.3 (0.7) | P < 0.0001; CI, 1.7‐3.3 | 4.2 ( 0.6) | P = 0.30; CI, 0.2 to 0.6 |
| Site sterilization (step 7) | 1.1 (1.1) | 4.3 (0.9) | P < 0.0001; CI, 2.7‐3.7 | 4.5 ( 0.5) | P = 0.45; CI, 0.6 to 0.3 |
| WSB creation (step 8) | 0.6 ( 0.6) | 4.1 ( 0.9) | P < 0.0001; CI, 3.0‐4.0 | 4.4 ( 0.6) | P = 0.26; CI, 0.7 to 0.2 |
| Ultrasound probe cover application (step 9) | 0.4 ( 0.9) | 4.1 ( 0.8) | P < 0.0001; CI, 3.2‐4.1 | 4.4 ( 0.8) | P = 0.23; CI, 0.8 to 0.2 |
We performed validation exercises to determine the degree of interrater agreement. Of the 60 total evaluations that were eventually performed, 11 evaluations had been performed simultaneously and independently by evaluators A and B. An analysis of the scores assigned by each evaluator to these 11 trainees revealed a high level of interrater agreement (96%). Further, we performed independent analyses of the trainees' scores as assessed by evaluator A (22 sessions) or evaluator B (27 sessions) across the initial, immediate, and delayed sessions, and we detected no statistical differences in the changes in scores (which mirrored the overall results above).
With regards to real‐life contamination between immediate scores and delayed scores, we identified 3 trainees who had placed central lines on actual patients during the interim period (2 trainees placed 1 line each, and 1 trainee placed 2 lines). We repeated all of the calculations without these participants' delayed scores and determined that the removal of their scores did not change the statistical significance of any of the study results. With regard to knowledge decay, the scatterplot comparing delayed scores to varying time‐to‐follow up revealed no correlation.
Discussion
Our study was designed to determine whether novice trainees could learn and retain proper central line placement skills on the NHTM by receiving personalized training in a relaxed, 1‐on‐1 learning environment. Success was measured by trained evaluators using a detailed evaluation tool with a 6‐point scoring scale. The results of our primary endpoints (changes in overall average scores across the 3 evaluation periods) confirmed that this type of training could quickly improve novice practitioners' skill levels from very low (initial evaluation) to significantly higher (immediate posttraining). The dropoff (decay) in skill levels was found to not be statistically significant over a period of several weeks, although we recognize that further study should be performed to establish the degree of skill decay over a longer period of time.
Because some steps in a central line insertion are more critical to the procedure's success than others (ie, a skin nick with a scalpel is less critical than vessel cannulation under ultrasound‐guidance), we analyzed 5 essential elements individually as secondary endpoints. This secondary analysis was designed to unmask any critical skill deficiencies that might otherwise have been lost in the overall analysis. For each individual essential elements step, this subanalysis similarly revealed a significant improvement from initial to immediate posttraining, and an insignificant score decay on the delayed evaluation.
We recognize a number of limitations to this study. First, the n is relatively small. A larger sample size would have allowed for greater statistical power. In addition, the scoring system used for this study was created by our Procedure Center staff and had never been truly validated elsewhere. The scoring system was transparent and logical, but we recognize that any attempt to use an interval scoring system to quantify procedural skills will be inherently imperfect; the difference between 1 and 2 is not necessarily the same as a difference between 4 and 5. Great efforts were taken to mitigate the impact of this limitation: explicit definitions were established for each score, and we put our evaluators through a rigorous scoring orientation at the outset to standardize their interpretation and use of the scoring system and assessment tool.
The variability in the amount of training time spent in each session could be considered to be a confounder. Our prior experiences training interns in small groups, however, suggested that individuals learn these skills at different paces and in different ways, and so we consider our customized approach to be an essential part of this training experience. We do recognize the practical limitations inherent in rolling out such an open‐ended approach, and program directors may face time and/or resource limitations if attempting to replicate this training strategy.
We were also aware of potential interrater variability between the evaluators. Our approach to addressing this was multifactorial: we went to great lengths to standardize evaluators' understanding of the intended scoring methodology prior to the initiation of the study. We also assessed the degree of interrater reliability once all data was collected. This analysis reinforced that both evaluators were scoring trainees in a virtually identical fashion. We attribute this consistency to the quality of the scoring system, the effectiveness of the prestudy evaluator orientation with a proceduralist, and the high degree of teamwork between the 2 evaluators that kept them closely in sync with one another throughout the study.
Evaluator bias was also a concern. While each evaluator was blinded to the trainees' prior scores, the setup associated with the different training sessions, as well as the obvious differences in performance between the trainees' initial and immediate/delayed performances, made full blinding of the evaluators difficult. The theoretical risk of evaluator bias in this study would have led to evaluators rating trainees higher in the immediate and delayed performances in order to demonstrate more dramatic results. We believe that, since the evaluators themselves did not perform the actual training, and since they did not know the previous scores for the trainee, they were less inclined to skew the scores. Video recording each performance and submitting this recording to a fully‐blinded, third‐party evaluator would have more rigorously ensured blinding than we were able to accomplish. This approach could be considered in future studies of this type.
An addition limitation involved the time‐to‐follow‐up. While a longer time interval between the immediate and delayed evaluations may have better evaluated the impact of the training and potential decay, we sought to balance this with the growing risk of contamination from real central line placement experiences as more time passed. With this issue in mind, the removal of the delayed scores from the 3 trainees who had placed central lines on actual patients in between the immediate and delayed evaluations (2 trainees placed 1 line each, and 1 trainee placed 2 lines) did not change the statistical significance of any of the study results.
One practical concern has to do with the reproducibility of this approach at other institutions. Each trainee received up to 2 hours of individualized attention, and each session consumed fresh supplies and required a proceduralist's and an evaluator's time. This represents a significant commitment of materials and manpower. A careful cost/benefit analysis is therefore warranted before implementing this kind of rigorous training program. As mentioned, the cost of the NHTM is approximately $120 and can withstand several cannulations over a 2‐day period; the sterile supplies and central line add up to approximately another $75/evaluation. Depending on the number of interns and residents at a given institution, these costs could prove prohibitive to cash‐poor residency training programs. In the larger picture, however, catheter‐related bloodstream infections have been estimated to result in a mortality rate of 4% to 20%, and a single catheter‐related bloodstream infection can cost up to $45,000.2124 In addition, new Medicare reimbursement policies are now beginning to limit hospital reimbursement for these types of iatrogenic events; hence, narrowing the margin of error and putting even greater financial pressures on hospitals.25 It is our belief, therefore, that an up‐front investment in NHTMs (or an alternative simulator), basic supplies, and the necessary trainer time will prove to be cost‐effective and enlightened investments from forward‐thinking leadership.
Last, we are also aware that our study did not look at whether our trainees' improved performance on the NHTM actually translated into better patient outcomes. Since patient safety is our ultimate goal, and this phase of PPSI limited all of our training and evaluations to the NHTMs, future studies must ultimately evaluate how well these learned skills translate into procedure performance on actual patients. This controlled study (possibly with a see‐1 do‐1 teach‐1 control group) will be logistically challenging, but will be the most definitive manner with which to demonstrate the true value of personalized training sessions using the NHTM (or another nonhuman simulator).
PPSI‐II demonstrated that using the NHTM as the basis for training novice practitioners in a personalized, 1‐on‐1 training session led to significant improvements in measured procedural skills. Further, these skills were retained over time. This positive study contributes to the growing body of literature pointing towards the role of intensive 1‐on‐1 training with simulators to advance procedural education for clinicians. Ultimately, we aim to demonstrate that providing trainees this type of training prior to having them perform procedures on actual patients will translate into superior patient care, greater success rates, fewer minor and major complications, and lower overall patient care costs. Rather than clinging to the classic but never‐validated see‐1, do‐1, teach‐1 approach, we believe that procedural training must adapt new curricula and technologies that will help us achieve the goals of maximizing the safety and quality of care for our patients.
Acknowledgements
The authors recognize and appreciate the entire staff of The Procedure Center at Cedars‐Sinai Medical Center for their support of this research project. The authors give special thanks to Obed Martinez for his tireless assistance with the scheduling and coordination of training activities, and to Jim Mirocha for statistical analysis and editorial contributions.
- Accreditation Council for Graduate Medical Education (ACGME). Home page. Available at: http://www.acgme.org. Accessed June2009.
- ,,.Procedural training at a crossroads: striking a balance between education, patient safety and quality.J Hosp Med.2007;2(3):123–125.
- ,,,,,.Firm‐based trial to improve central venous catheter insertion practices.J Hosp Med.2007;2(3):135–142.
- ,,,,,Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice.J Hosp Med.2008;3(1):48–54.
- ,,.Central line simulation: a new training algorithm.Am Surg.2007;73:680–682.
- ,,, et al.Developing technical expertise in emergency medicine—the role of simulation in procedural skill acquisition.Acad Emerg Med.2008;15:1046–1057.
- ,,,.A training system for ultrasound‐guided needle insertion procedures.Med Image Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv.2007;10(1):566–574.
- ,,, et al.Video‐based training increases sterile‐technique compliance during central venous catheter insertion.Crit Care Med.2007;35:1302–1306.
- ,,,,Comparison of results of virtual‐reality simulator and training model for basic ureteroscopy training.J Endourol.2006;20(4):266–271.
- ,.Papaya: a simulation model for training in uterine aspiration.Fam Med.2005;37(4):242–244.
- ,,,,.An intervention to improve procedure education for internal medicine residents.J Gen Intern Med.2008;23(3):288–293.
- ,,.The use of tissue models for vascular access training: phase 1 of the procedural patient safety initiative.J Gen Intern Med.2006;21(5):514–517.
- Blue phantom: CVC hands‐on trainer, items # BPH600f, BPH604HP, BPH600AP. Available at: http://www.bluephantom.com/desktopdefault.aspx?tabid=232. Accessed June2009.
- Simulab Corporation: Central Line Man System. Available at: http://www.simulab.com/product/surgery/open/centralineman‐system. Accessed June2009.
- KyotoKagaku Co., Ltd.: CVC Insertion Simulator. Available at: http://www.kyotokagaku.com/products/detail01/m93u.html. Accessed June2009.
- First Aid Manufacturer CVC Simulator. Available at: http://www.first‐aid‐manufacturer.com/CVC‐Simulator.aspx. Accessed June2009.
- Limbs and Things: Central Venous Catheter Insertion Simulator, part #KKM93UB. Available at: http://www.golimbs.com/products/products.php?sectid=5356(17):1789–1790.
- ,,.Practice #20: proceduralists. The Advisory Board Annual Report.2007:162–169.
- NEJM video. Available at: http://content.nejm.org/cgi/content/short/356/21/e21. Accessed June2009.
- .Prevention of intravascular catheter‐related infections.Ann Intern Med.2000;132(5):391–402.
- ,,, et al.Guidelines for the prevention of intravascular catheter‐related infections.MMWR Recomm Rep.2002;51(RR‐10):1–29.
- ,,, et al.Clinical and economic outcomes in critically ill patients with nosocomial catheter‐related bloodstream infections.Clin Infect Dis.2005;41:1591–1598.
- ,,,,,.Attributable morbidity and mortality of catheter‐related septicemia in critically ill patients: a matched, risk‐adjusted, cohort study.Infect Control Hosp Epidemiol.1999;20(6):396–401.
- Centers for Medicaid and Medicare Services. U.S. Department of Health and Human Services. Hospital‐Acquired Conditions. Available at: http://www.cms.hhs.gov/HospitalAcqCond/06_Hospital‐Acquired_Conditions.asp#TopOfPage. Accessed June2009.
- Accreditation Council for Graduate Medical Education (ACGME). Home page. Available at: http://www.acgme.org. Accessed June2009.
- ,,.Procedural training at a crossroads: striking a balance between education, patient safety and quality.J Hosp Med.2007;2(3):123–125.
- ,,,,,.Firm‐based trial to improve central venous catheter insertion practices.J Hosp Med.2007;2(3):135–142.
- ,,,,,Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice.J Hosp Med.2008;3(1):48–54.
- ,,.Central line simulation: a new training algorithm.Am Surg.2007;73:680–682.
- ,,, et al.Developing technical expertise in emergency medicine—the role of simulation in procedural skill acquisition.Acad Emerg Med.2008;15:1046–1057.
- ,,,.A training system for ultrasound‐guided needle insertion procedures.Med Image Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv.2007;10(1):566–574.
- ,,, et al.Video‐based training increases sterile‐technique compliance during central venous catheter insertion.Crit Care Med.2007;35:1302–1306.
- ,,,,Comparison of results of virtual‐reality simulator and training model for basic ureteroscopy training.J Endourol.2006;20(4):266–271.
- ,.Papaya: a simulation model for training in uterine aspiration.Fam Med.2005;37(4):242–244.
- ,,,,.An intervention to improve procedure education for internal medicine residents.J Gen Intern Med.2008;23(3):288–293.
- ,,.The use of tissue models for vascular access training: phase 1 of the procedural patient safety initiative.J Gen Intern Med.2006;21(5):514–517.
- Blue phantom: CVC hands‐on trainer, items # BPH600f, BPH604HP, BPH600AP. Available at: http://www.bluephantom.com/desktopdefault.aspx?tabid=232. Accessed June2009.
- Simulab Corporation: Central Line Man System. Available at: http://www.simulab.com/product/surgery/open/centralineman‐system. Accessed June2009.
- KyotoKagaku Co., Ltd.: CVC Insertion Simulator. Available at: http://www.kyotokagaku.com/products/detail01/m93u.html. Accessed June2009.
- First Aid Manufacturer CVC Simulator. Available at: http://www.first‐aid‐manufacturer.com/CVC‐Simulator.aspx. Accessed June2009.
- Limbs and Things: Central Venous Catheter Insertion Simulator, part #KKM93UB. Available at: http://www.golimbs.com/products/products.php?sectid=5356(17):1789–1790.
- ,,.Practice #20: proceduralists. The Advisory Board Annual Report.2007:162–169.
- NEJM video. Available at: http://content.nejm.org/cgi/content/short/356/21/e21. Accessed June2009.
- .Prevention of intravascular catheter‐related infections.Ann Intern Med.2000;132(5):391–402.
- ,,, et al.Guidelines for the prevention of intravascular catheter‐related infections.MMWR Recomm Rep.2002;51(RR‐10):1–29.
- ,,, et al.Clinical and economic outcomes in critically ill patients with nosocomial catheter‐related bloodstream infections.Clin Infect Dis.2005;41:1591–1598.
- ,,,,,.Attributable morbidity and mortality of catheter‐related septicemia in critically ill patients: a matched, risk‐adjusted, cohort study.Infect Control Hosp Epidemiol.1999;20(6):396–401.
- Centers for Medicaid and Medicare Services. U.S. Department of Health and Human Services. Hospital‐Acquired Conditions. Available at: http://www.cms.hhs.gov/HospitalAcqCond/06_Hospital‐Acquired_Conditions.asp#TopOfPage. Accessed June2009.
Copyright © 2009 Society of Hospital Medicine
