User login
The Impact of Checklists on Inpatient Safety Outcomes: A Systematic Review of Randomized Controlled Trials
In response to widely publicized reports highlighting the challenges of suboptimal quality of healthcare, improving patient safety has been a leading healthcare initiative for more than 10 years.1-4 Numerous strategies to improve patient safety have been proposed,5-9 but improvements have been limited, which raises questions about whether the right approaches are being employed.10,11
Checklists have served as a foundation for the standardization and safety of aviation and nuclear power12,13 and are advocated as simple and effective instruments for ensuring safe care.7,14,15 Systematic reviews of observational studies suggest that checklists can reduce medical errors and adverse events,15-19 but these reviews are at risk of bias due to the limitations of observational methods. Furthermore, discordant results of recent high-profile evaluations of the World Health Organization (WHO) Surgical Safety Checklist highlight the need for checklist evaluations using rigorous study designs.20-22 Therefore, we sought to conduct a systematic review of RCTs (randomized controlled trials) to determine whether checklists, as a type of decision-support tool, are effective at improving patient safety outcomes in hospitalized patients.
METHODS
The study protocol was registered with the PROSPERO Register of Systematic Reviews (registration number: CRD42016037441) and developed according to the Preferred Reporting Items in Systematic Reviews and Meta-analyses (PRISMA) statement.23
Search Strategy
On December 8, 2016, we systematically searched Ovid MEDLINE, Ovid EMBASE, PubMed, and the Cochrane Central Register of Controlled Trials. The search was performed using no language or publication date restrictions and included 2 groups of terms (key words with similar characteristics): ‘checklists’ and ‘patient outcomes assessment’. We restricted our search to patient outcomes because these are more patient-oriented than the proximal processes of care that may not translate into outcomes. The search was restricted to RCTs using the Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials from the Cochrane Collaborative.24 The MEDLINE search strategy is depicted in Appendix I (Supplementary File 1). Reference lists of included articles were manually searched for additional publications. The search strategy was designed with the help of an information scientist (DL). EndNote X7 (Thomas Reuters, Philadelphia, PA, USA) was the reference software used for the management of citations.
Eligibility Criteria
We selected all studies reporting patient safety outcomes of a checklist intervention, using the following inclusion criteria: 1) acute care hospital inpatient population, 2) checklist intervention, 3) contain a control group (ie, no checklist), 4) report one or more patient safety outcome, as defined by the authors (eg, medical errors, adverse events, mortality), and 5) RCT design. We restricted our focus to inpatient populations given the heterogeneity of illness and patient care between acute and community settings. We defined a checklist as a tool that details the essential steps of a task, requiring the target provider to indicate whether an item was completed or not.1,7 Tools that included only 1 item (eg, electronic prompts) or did not require acknowledgement of the items (eg, guidelines) were excluded. We defined patient safety outcomes as the authors’ definition of patient safety (eg, medical error, adverse event, provider compliance with safety regulations).
Study Selection
Two reviewers (JMB, GW) independently, and in duplicate, reviewed the titles and abstracts of the retrieved citations against the eligibility criteria. The same 2 reviewers subsequently reviewed the full text of relevant articles for inclusion. Eligibility disagreements were resolved by consensus. A Kappa statistic was calculated for reviewer agreement of full-text screening.25 Reviewers were not blinded to author or journal names.26
Data Extraction
The structured data extraction form was calibrated using the first 2 articles. The 2 reviewers (JMB, GW) independently, and in duplicate, extracted data from included studies on the study characteristics, setting, study population, sample size, intervention used, outcomes examined, analytic method, and study quality. The data extraction form is depicted in Appendix II (Supplementary File 2). Coding discrepancies were resolved by consensus.
Quality Assessment
The 2 reviewers (JMB, GW) extracted data on study quality independently and in duplicate using 2 approaches. First, reviewers assessed study quality using a component method derived from the Cochrane Collaboration criteria.24 For each included study, the reviewers documented if the authors had adequately described inclusion/exclusion criteria, randomization, allocation concealment, blinding of participants/outcome assessors, attrition, cross over, baseline characteristics, and power calculation. Second, the reviewers calculated and reported the Jadad score for each included study, a validated assessment scale that assigns points (1 to 5) based on randomization, blinding, and attrition.27
Analysis
Owing to the heterogeneity of the data and the small number of studies that satisfied the inclusion criteria, the data were analyzed using guidelines for the narrative synthesis of a systematic review.28 Descriptive statistical findings from each included study were reported. The DerSimonian and Laird method for random-effects models was used to calculate a pooled estimate of 30-day all-cause mortality from the raw data available from a subset of studies (number of events, study population).29 Stata SE version 13.1 (Stata Corp, LP, College Station, TX) was used to perform the statistical analyses.
RESULTS
The literature search identified 11,225 unique citations from which 83 abstracts were eligible for full-text review. We identified 9 full-text articles for inclusion in the review (Figure 1 [Supplementary File 3]). The main reasons for citation exclusion during the full-text review were that the study design was not an RCT (39%) or there was no checklist intervention (34%
Study Characteristics
Characteristics of the included studies are summarized in Tables 1 and 2. Six of the studies were conducted in at least 1teaching hospital.30-35 The studies varied in target populations for both the checklist user and patients. The outcomes reported varied; 3 studies examined 30-day mortality,21,30,36 4 studies examined hospital length of stay,21,30,33,36 and 2 studies reported user compliance with the checklist.21,31 Five of the studies reported patient outcomes,21,30,33,35,36 and 5 studies reported provider-level outcomes related to patient safety (eg, compliance with checklist items such as communication of medications, isolation precautions, etc.).31-34,37
Description of Checklists
Supplementary File 4 (Table 3) provides a detailed breakdown of the checklists’ purpose and components. Six of the checklists were designed to directly reduce patient safety events,21,30,33,35-37 whereas 3 of the checklists were designed to indirectly reduce patient safety events by increasing compliance with processes of care.31,32,34 Six checklists were constructed and pilot tested by the research team conducting the RCT30-35 and the 3 remaining studies used modified versions of previously validated checklists.21,36,37 The number of items included in the checklist ranged from 2 to 54.
Impact of the Checklist
Table 4 summarizes the adverse events, medical errors, resource utilization and/or compliance reported for each checklist. Chaudhary et al. reported significant decreases in Grade III (requiring intervention)38 and IV (life-threatening)38 postoperative complications (23% v. 33%, P = 0.04) and 30-day mortality (5.7% vs 10.0%, P = 0.04) for patients assigned to the Modified WHO Surgical Safety Checklist compared to controls.21 Conversely, Haugen et al. reported a nonsignificant reduction in 30-day mortality between the WHO Surgical Safety Checklist group and controls (1.0% vs 1.6%, P = 0.151).36 Bassor et al. reported no significant difference in 30-day hospital readmission for decompensated heart failure for the heart failure discharge checklist group when compared to controls (6% vs. 4%, P = NS); however, an exploratory analysis that excluded patients who died during the follow-up period found a significant difference in 30-day readmission rates (2% vs. 20%, P = 0.02).30 Gentili et al. reported a higher proportion of patients with pain control in the checklist group compared to the controls (67.6% vs. 54.8%), as well as fewer incidents of analgesic therapy–related uncontrolled adverse events (25.9% vs. 49.9%); however, the statistical significance of these differences were not reported.35 The Writing Group for CHECKLIST-ICU reported no significant difference for in-hospital mortality between the checklist and control groups (adjusted odds ratio [AOR] 1.02, 95% CI, 0.82-1.26, P = 0.88), nor for the secondary clinical outcomes examined (Table 4).33 However, there was a significant difference between the checklist group and control group for 3 of the 7 outcomes related to processes of patient care, including a reduction in the use of both urinary catheters (adjusted rate ratio [ARR] 0.86, 95% CI, 0.80-0.93, P < 0.001) and central venous catheters (ARR 0.90, 95% CI 0.83-0.98, P = 0.02). Masson et al. reported that when using the FASTHUG-MAIDENS checklist, more drug-related problems were identified by pharmacy residents (in relation to the number identified by the ICU pharmacist) both per patient encounter (P = 0.008) and overall (P < 0.001).37 Ong et al. reported higher rates of compliance with isolation precautions for infectious diseases in the checklist group (71% vs. 38%, P < 0.01); however, compliance with the checklist was low (40%) and qualitative analyses found participants were dissatisfied with the checklist.31 Salzwedel et al. reported the number of items handed over by anesthesia residents postoperatively to be higher in the checklist group than the control group (48.7% vs. 32.4%, P < 0.001).32 In a more recent study, Salzwedel et al. reported that proportion of items deemed by the attending anesthesiologist as “must be handed over” were more often actually handed over by the anesthesia residents assigned to the checklist group when compared to controls (87.1% vs. 75.0%, P = 0.005).34
30-day Mortality
A random-effects model pooling data from the 3 studies that reported data for 30-day all cause mortality suggested a significant reduction with use of a checklist (OR 0.60, 95% CI, 0.41-0.89; P = 0.01, I2 = 0.0%, P = 0.573).
Study Quality
Supplementary File 5 (Table 5) summarizes the quality assessment of the 9 studies. The clarity of description for each intervention varied. All studies reported inclusion/exclusion criteria and randomization procedures. Three studies indicated that outcome assessors were blinded to intervention allocation;32,34,36 while this was unclear in 2 studies.21,30 Three studies reported baseline characteristics.21,30,36 Two studies reported power calculations;33,37 however, one study had a sample size that was less than that required to achieve the target power.37 The Jadad scores ranged from 1to 5.
DISCUSSION
This systematic review identified 9 RCTs that examined the impact of a checklist on patient safety outcomes in hospitalized patients. The studies employed checklists with different purposes and elements and measured different patient safety outcomes. The methodological quality of the included studies was moderate. In aggregate, the results suggest that checklists may be effective at improving patient safety outcomes, but the small number of moderate quality studies and the heterogeneity of interventions and outcome measures suggests that there is an urgent need for further evaluation.
The most important observation from our systematic review is the paucity of high quality evidence evaluating checklists’ impact on patient safety outcomes in acute inpatient care. The implementation of checklists is increasingly common as they are relatively low cost to develop and implement, and intuitively make sense. This is particularly true in an era of increasing efforts to standardize care as a means for improving quality and minimizing cost (ie, previous systematic reviews cite 38 unique studies).39 However, implementation of an inadequately tested checklist risks unintended consequences (eg, inefficient resource utilization).18 The small number of RCTs identified might be owing to quality improvement efforts traditionally focusing on ‘real life’ applicability over rigorous research methodology.40 The translation of evidence into clinical practice is known to be slow;41 however, these more rigorous methodologies reduce the risk of biases and generate high-quality evidence, which help to fulfill the necessity to identify best practices while avoiding these unintended consequences.
The studies varied both in the approaches used to develop checklists and in the number of items included (ranging from 2 to 54). What is the optimal method for developing a checklist and how does this impact their effectiveness?42 The answers to these questions are not known. However, this review highlights some important issues to consider when developing a checklist. As the number of items or complexity of a task increases, our ability to efficiently perform the task without aid decreases.43-45 As such, a well-designed checklist should detail explicit instructions on the what, where, when, and how of a given task in a fashion that ensures a consistent accuracy for completing the work.5 It is recommended that construction of a checklist follow the principles of human factors engineering: engage stakeholders and human factors experts in the design; are developed based on user needs and realities; list items in order of importance; are concise and subgroup sections of checklists by task or chronological order; ensure usability and evaluate potential negative consequences (eg time to complete); are pilot tested and validated before implementation; are updated as needed based the on generation of new findings or changes in operational procedures.46 These general principles of human factors engineering46 provide a practical approach for the development and evaluation of a checklist. In addition, standardization of operational definitions (ie, process, outcome, compliance) is important for study replication and robust meta-analyses.
Checklists used in aviation are perhaps best known12 and the evidence of their effectiveness is derived from the attribution of aviation errors to incomplete checklists.12 Although more recently implemented in medicine, checklists have the potential to guide the successful completion of complex tasks in healthcare.7 Systematic reviews of observational studies have been conducted for specific checklists (eg, WHO Surgical Safety Checklist) and for select patient populations (eg, surgical patients), and the number of included studies ranges from 7-27 (n = 38 unique studies).15,16,18,19 For example, Gillespie et al. in a systematic review and meta-analysis reported the implementation of Surgical Safety Checklists to be associated with a reduction in postoperative complications (relative risk [RR] 0.63, 95% CI, 0.58-0.72, P = < 0.001), but not mortality (RR 1.03, 95% CI, 0.73-1.4, P = 0.857).19 Similarly, Treadwell et al. reported in a systematic review of Surgical Safety Checklists that while data are promising, more evaluation of their impact on clinical outcomes is needed.18 These recommendations are nicely illustrated by Urbach et al.’s20 and O’Leary et al.’s47 evaluations of the mandatory adoption of Surgical Safety Checklists across all hospitals in Ontario, Canada, which respectively demonstrated no significant reductions in 30-day perioperatively conplications for both adult (OR 0.97, 95% CI, 0.90-1.03, P = 0.29) and pediatric (AOR 1.01, 95% CI, 0.90-1.14, P = 0.9) patients. These data not only highlight the need for further evaluation of checklists but are also a reminder that checklists and their associated implementation strategies are complex interventions for which there may be important differences between the efficacy reported in clinical trials and the effectiveness reported in implementation studies.48 This all suggests that if checklists are to be effective in improving patient safety, process evaluations of implementation49 and realist reviews of published studies50 may be important to determine optimal approaches for implementation. We believe that, based on the limited currently available evidence, there is urgency for further robust evaluations of checklists before their widespread implementation. If effective, they should be widely implemented. If ineffective, they should be abandoned to minimize unintended consequences and inefficient use of resources.
There are 4 primary limitations to this review that should be considered when interpreting the findings. First, the RCT design is not the study design employed by most quality improvement initiatives.40 While some quality improvement experts may argue that an RCT design is insufficiently flexible for applied settings, it does minimize the risk of biased assessments of intervention effectiveness. Second, our search strategy included an RCT filter. The filter helped restrict the number of citations to be reviewed (n = 11,225) but could have resulted in improperly indexed studies being excluded. To guard against this risk, we used the validated Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials,24 reviewed reference lists of citations included in the review, and solicited suggestions for missing studies from quality improvement experts. Third, our review was restricted to hospitalized patients. Although the studies evaluated commonly reported safety outcomes across patients with diverse clinical conditions, care settings, and providers that broadly reflect hospital-based care, evaluations of checklists in additional patient and provider groups are needed (eg, hospitalists). Furthermore, the effectiveness of checklists for improving patient safety outcomes in outpatients is important; however, the organizational and patient characteristics of these 2 settings (hospitalized vs outpatient) are sufficiently different to warrant separate systematic reviews. Finally, owing to the heterogeneity of the checklists used and outcomes measured, we were unable to perform a robust meta-analysis. Heterogeneity, combined with the small number of studies identified in our search, prevented us from applying statistical methods to assess for publication bias. This limitation of our systematic review highlights an important gap in the literature and emphasizes the importance of additional primary research to evaluate checklists.
In summary, we identified few RCTs that examined checklists designed to improve patient safety outcomes. The small number of existing studies suggests that checklists may improve patient safety outcomes; however, these observations were not reported for all outcomes examined and the studies were heterogeneous and of limited methodological quality. There is an urgent need for high-quality evaluations of the effectiveness of patient safety checklists in inpatient healthcare settings to substantiate their perceived benefits.
Acknowledgments
We would like to thank Diane Lorenzetti for her help with the development of the search strategy.
Disclosure: The authors have no known conflicts of interest to declare.
Jamie Boyd was supported by a W21C – Alberta Innovates-Health Solutions (AIHS) Collaborative Research and Innovation Opportunities (CRIO) Health Services Research graduate studentship. Guosong Wu was supported by a Western Regional Training Centre (WRTC) for Health Services Research graduate studentship. Dr. Stelfox was supported by a Population Health Investigator Award from Alberta Innovates Health Solutions.
Authors’ Contributions
HTS was responsible for the study’s conception. All 3 authors contributed to the study’s design and interpretation. JB and GW were responsible for searching the literature, reviewing abstracts, selecting full-text articles and critically appraising them. All 3 authors performed the analyses. JB drafted the manuscript and all 3 authors assisted in the successive revisions of the final manuscript. All authors have read and approved the final manuscript.
1. World Health Organization. Patient safety. Available at: http://www.who.int/patientsafety/about/en/. Accessed June 21, 2016.
2. Institute of Medicine. To err is human: Building a safer health system. In: Kohn L, Corrigan J, Donaldson M, eds. Institute of Medicine-Committee on Quality of Health Care in America. Washington DC: National Academy Press; 1999:86-101. PubMed
3. Institute of Medicine Committee on the Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington DC: National Academy Press; 2001. PubMed
4. Stelfox HT, Palmisani S, Scurlock C, Orav EJ, Bates DW. The “to err is human” report and the patient safety literature. Qual Saf Health Care. 2006; 15(3):174-178. PubMed
5. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost, PJ. Clinical review: Checklists - translating evidence into practice. Crit Care. 2009; 13(6):210. PubMed
6. Ely EW, Bennett PA, Bowton DL, Murphy SM, Florance AM, Haponik EF. Large scale implementation of a respiratory therapist-driven protocol for ventilator weaning. Am J Respir Crit Care Med. 1999; 159(2):439-446. PubMed
7. Gawande A. The checklist manifesto: How to get things right. Great Britain: Profile Books LTD; 2010.
8. Pronovost P, Vohr E. Safe patients, smart hospitals. New York, NY: Hudson Street Press; 2010.
9. Hughes RG. Advances in patient safety: Tools and strategies for quality improvement and patient safety. In: Hughes RG, ed. Patient safety and quality: An evidence-based handbook for nurses. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. PubMed
10. Henriksen K, Oppenheimer C, Leape LL, et al. Envisioning patient safety in the year 2025: Eight perspectives. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: New directions and alternative approaches. Rockville, MD: Agency for Healthcare Research and Quality; 2008. PubMed
11. Gaba DM, Howard SK. Patient safety: Fatigue among clinicians and the safety of patients. N Engl J Med. 2002; 347(16):1249-1255. PubMed
12. Degani A, Wiener EL. Cockpit checklists: Concepts, design, and use. Human Factors: The Journal of the Human Factors and Ergonomics Society 1993; 35(2):345-359.
13. Swain AD, Guttmann HE. Handbook of human reliability analysis with emphasis on nuclear power plant applications: Final report. Washington, DC: U.S. Nuclear Regulatroy Commission; 1983.
14. de Vries EN, Prins HA, Crolla RM, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010; 363(20):1928-1937. PubMed
15. Bergs J, Hellings J, Cleemput I, et al. Systematic review and meta-analysis of the effect of the world health organization surgical safety checklist on postoperative complications. Br J Surg. 2014; 101(3):150-158. PubMed
16. Pucher PH, Johnston MJ, Aggarwal R, Arora S, Darzi A. Effectiveness of interventions to improve patient handover in surgery: A systematic review. Surgery. 2015; 158(1):85-95. PubMed
17. Bergs J, Lambrechts F, Simons P, et al. Barriers and facilitators related to the implementation of surgical safety checklists: A systematic review of the qualitative evidence. BMJ Qual Saf. 2015; 23(12):776-786. PubMed
18. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: A systematic review of impacts and implementation. BMJ Qual Saf. 2014; 23(4):299-318. PubMed
19. Gillespie BM, Chaboyer W, Thalib L, John M, Fairweather N, Slater K. Effect of using a safety checklist on patient complications after surgery: A systematic review and meta-analysis. Anesthesiology. 2014; 120(6):1380-1389. PubMed
20. Reames BN, Krell RW, Campbell DA Jr, Dimick JB. A checklist-based intervention to improve surgical outcomes in michigan: Evaluation of the keystone surgery program. JAMA Surg. 2015; 150(3):208-215. PubMed
21. Chaudhary N, Varma V, Kapoor S, Mehta N, Kumaran V, Nundy S. Implementation of a surgical safety checklist and postoperative outcomes: A prospective randomized controlled study. J Gastrointest Surg. 2015; 19(5):935-942. PubMed
22. Reames BN, Krell RW, Campbell DA, Jr., Dimick JB. A checklist-based intervention to improve surgical outcomes in Michigan: Evaluation of the Keystone Surgery program. JAMA surgery. 2015; 150(3):208-215. PubMed
23. Liberati A, Altman DG, Tetzlaff J, et al. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med. 2009; 151(4):W65-94. PubMed
24. The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions, version 5.1.0. Oxford, UK: The Cochrane Collaboration, 2011.
25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-174. PubMed
26. Berlin JA. Does blinding of readers affect the results of meta-analyses? University of pennsylvania meta-analysis blinding study group. Lancet 1997;350(9072):185-186. PubMed
27. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996;17(1):1-12. PubMed
28. Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews: A product form the esrc methods programme. Available at: https://www.researchgate.net/profile/Mark_Rodgers4/publication/233866356_Guidance_on_the_conduct_of_narrative_synthesis_in_systematic_reviews_A_product_from_the_ESRC_Methods_Programme/links/02e7e5231e8f3a6183000000.pdf. Accessed June 17, 2016.
29. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177-188. PubMed
30. Basoor A, Doshi NC, Cotant JF, et al. Decreased readmissions and improved quality of care with the use of an inexpensive checklist in heart failure. Congest Heart Fail. 2013; 19(4):200-206. PubMed
31. Ong MS, Magrabi F, Post J, et al. Communication interventions to improve adherence to infection control precautions: A randomised crossover trial. BMC Infect Dis. 2013; 13:72. PubMed
32. Salzwedel C, Bartz HJ, Kuhnelt I, et al. The effect of a checklist on the quality of post-anaesthesia patient handover: A randomized controlled trial. Int J Qual Health Care. 2013; 25(2):176-181.
33. Implement Sci. 50. Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: Illustrating the method for implementation research. 2008; 337.BMJ. PubMed
49. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: The new medical research council guidance. 2014; 9(9):e108585.PloS one. PubMed
48. Gagliardi AR, Straus SE, Shojania KG, Urbach DR. Multiple interacting factors influence adherence, and outcomes associated with surgical safety checklists: A qualitative study.
2016; 188(9):E191-E198.CMAJ. PubMed
47. O’Leary JD, Wijeysundera DN, Crawford MW. Effect of surgical safety checklists on pediatric surgical complications in Ontario. Rockville, MD: Agency for Healthcare Research and Quality; 2013.Human factors and ergonomics. Making health care safer ii: An updated critical analysis of the evidence for patient safety practices. PubMed
46. Carayon P, Xie A, Kianfar SH. 2005; 16(1):70-76.Psychol Sci. 2004; 30(4):689-707.
45. Halford GS, Baker R, McCredden JE, Bain JD. How many variables can humans process? J Exp Psychol Hum Percept Perform. PubMed
44. Oberauer K, Kliegl R. Simultaneous cognitive operations in working memory after dual-task practice. 1956; 63(2):81-97.Psychol Rev. PubMed
43. Miller GA. The magical number seven plus or minus two: Some limits on our capacity for processing information. 2008; 20(1):22-30.Int J Qual Health Care. PubMed
42. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. 2011; 104(12):510-520.J R Soc Med. PubMed
41. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. 2015; 24(5):325-336.BMJ Qual Saf. PubMed
40. Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: A brief overview of possible study types. Washington (DC): National Academies Press (US); 2013.Best care at lower cost: The path to continuously learning health care in america. PubMed
39. Institute of Medicine. Committee on the learning health care system in America. In: Smith M, Saunders R, Stuckhardt L, et al., eds. 2004; 240(2):205-213.Ann Surg. PubMed
38. Dindo D, Demartines N, Clavien P-A. Classification of surgical complications: A new proposal with evaluation in a cohort of 6336 patients and results of a survey. 2013; 66(3):157-162.Can J Hosp Pharm. PubMed
37. Masson SC, Mabasa VH, Malyuk DL, Perrott JL. Validity evidence for fasthug-maidens, a mnemonic for identifying drug-related problems in the intensive care unit. 2015; 261(5):821-828.Ann Surg. PubMed
36. Haugen AS, Softeland E, Almeland SK, et al. Effect of the world health organization checklist on patient outcomes: A stepped wedge cluster randomized controlled trial.
2016; 12(2):199-205.Future Oncol. PubMed
35. Gentili M, Clerico M, Spizzichino M, Fanelli G. Use of a checklist to improve pain control in hospitalized cancer patients: the 38Checkpain project. 2016; 32:170-174.J Crit Care. PubMed
34. Salzwedel C, Mai V, Punke MA, Kluge S, Reuter DA. The effect of a checklist on the quality of patient handover from the operating room to the intensive care unit: A randomized controlled trial. 2016; 315(14):1480-1490.JAMA. PubMed
33. The Writing Group for CHECKLIST-ICU, Cavalcanti AB, Bozza FA, et al. Effect of a Quality Improvement Intervention With Daily Round Checklists, Goal Setting, and Clinician Prompting on Mortality of Critically Ill Patients: A Randomized Clinical Trial. PubMed
In response to widely publicized reports highlighting the challenges of suboptimal quality of healthcare, improving patient safety has been a leading healthcare initiative for more than 10 years.1-4 Numerous strategies to improve patient safety have been proposed,5-9 but improvements have been limited, which raises questions about whether the right approaches are being employed.10,11
Checklists have served as a foundation for the standardization and safety of aviation and nuclear power12,13 and are advocated as simple and effective instruments for ensuring safe care.7,14,15 Systematic reviews of observational studies suggest that checklists can reduce medical errors and adverse events,15-19 but these reviews are at risk of bias due to the limitations of observational methods. Furthermore, discordant results of recent high-profile evaluations of the World Health Organization (WHO) Surgical Safety Checklist highlight the need for checklist evaluations using rigorous study designs.20-22 Therefore, we sought to conduct a systematic review of RCTs (randomized controlled trials) to determine whether checklists, as a type of decision-support tool, are effective at improving patient safety outcomes in hospitalized patients.
METHODS
The study protocol was registered with the PROSPERO Register of Systematic Reviews (registration number: CRD42016037441) and developed according to the Preferred Reporting Items in Systematic Reviews and Meta-analyses (PRISMA) statement.23
Search Strategy
On December 8, 2016, we systematically searched Ovid MEDLINE, Ovid EMBASE, PubMed, and the Cochrane Central Register of Controlled Trials. The search was performed using no language or publication date restrictions and included 2 groups of terms (key words with similar characteristics): ‘checklists’ and ‘patient outcomes assessment’. We restricted our search to patient outcomes because these are more patient-oriented than the proximal processes of care that may not translate into outcomes. The search was restricted to RCTs using the Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials from the Cochrane Collaborative.24 The MEDLINE search strategy is depicted in Appendix I (Supplementary File 1). Reference lists of included articles were manually searched for additional publications. The search strategy was designed with the help of an information scientist (DL). EndNote X7 (Thomas Reuters, Philadelphia, PA, USA) was the reference software used for the management of citations.
Eligibility Criteria
We selected all studies reporting patient safety outcomes of a checklist intervention, using the following inclusion criteria: 1) acute care hospital inpatient population, 2) checklist intervention, 3) contain a control group (ie, no checklist), 4) report one or more patient safety outcome, as defined by the authors (eg, medical errors, adverse events, mortality), and 5) RCT design. We restricted our focus to inpatient populations given the heterogeneity of illness and patient care between acute and community settings. We defined a checklist as a tool that details the essential steps of a task, requiring the target provider to indicate whether an item was completed or not.1,7 Tools that included only 1 item (eg, electronic prompts) or did not require acknowledgement of the items (eg, guidelines) were excluded. We defined patient safety outcomes as the authors’ definition of patient safety (eg, medical error, adverse event, provider compliance with safety regulations).
Study Selection
Two reviewers (JMB, GW) independently, and in duplicate, reviewed the titles and abstracts of the retrieved citations against the eligibility criteria. The same 2 reviewers subsequently reviewed the full text of relevant articles for inclusion. Eligibility disagreements were resolved by consensus. A Kappa statistic was calculated for reviewer agreement of full-text screening.25 Reviewers were not blinded to author or journal names.26
Data Extraction
The structured data extraction form was calibrated using the first 2 articles. The 2 reviewers (JMB, GW) independently, and in duplicate, extracted data from included studies on the study characteristics, setting, study population, sample size, intervention used, outcomes examined, analytic method, and study quality. The data extraction form is depicted in Appendix II (Supplementary File 2). Coding discrepancies were resolved by consensus.
Quality Assessment
The 2 reviewers (JMB, GW) extracted data on study quality independently and in duplicate using 2 approaches. First, reviewers assessed study quality using a component method derived from the Cochrane Collaboration criteria.24 For each included study, the reviewers documented if the authors had adequately described inclusion/exclusion criteria, randomization, allocation concealment, blinding of participants/outcome assessors, attrition, cross over, baseline characteristics, and power calculation. Second, the reviewers calculated and reported the Jadad score for each included study, a validated assessment scale that assigns points (1 to 5) based on randomization, blinding, and attrition.27
Analysis
Owing to the heterogeneity of the data and the small number of studies that satisfied the inclusion criteria, the data were analyzed using guidelines for the narrative synthesis of a systematic review.28 Descriptive statistical findings from each included study were reported. The DerSimonian and Laird method for random-effects models was used to calculate a pooled estimate of 30-day all-cause mortality from the raw data available from a subset of studies (number of events, study population).29 Stata SE version 13.1 (Stata Corp, LP, College Station, TX) was used to perform the statistical analyses.
RESULTS
The literature search identified 11,225 unique citations from which 83 abstracts were eligible for full-text review. We identified 9 full-text articles for inclusion in the review (Figure 1 [Supplementary File 3]). The main reasons for citation exclusion during the full-text review were that the study design was not an RCT (39%) or there was no checklist intervention (34%
Study Characteristics
Characteristics of the included studies are summarized in Tables 1 and 2. Six of the studies were conducted in at least 1teaching hospital.30-35 The studies varied in target populations for both the checklist user and patients. The outcomes reported varied; 3 studies examined 30-day mortality,21,30,36 4 studies examined hospital length of stay,21,30,33,36 and 2 studies reported user compliance with the checklist.21,31 Five of the studies reported patient outcomes,21,30,33,35,36 and 5 studies reported provider-level outcomes related to patient safety (eg, compliance with checklist items such as communication of medications, isolation precautions, etc.).31-34,37
Description of Checklists
Supplementary File 4 (Table 3) provides a detailed breakdown of the checklists’ purpose and components. Six of the checklists were designed to directly reduce patient safety events,21,30,33,35-37 whereas 3 of the checklists were designed to indirectly reduce patient safety events by increasing compliance with processes of care.31,32,34 Six checklists were constructed and pilot tested by the research team conducting the RCT30-35 and the 3 remaining studies used modified versions of previously validated checklists.21,36,37 The number of items included in the checklist ranged from 2 to 54.
Impact of the Checklist
Table 4 summarizes the adverse events, medical errors, resource utilization and/or compliance reported for each checklist. Chaudhary et al. reported significant decreases in Grade III (requiring intervention)38 and IV (life-threatening)38 postoperative complications (23% v. 33%, P = 0.04) and 30-day mortality (5.7% vs 10.0%, P = 0.04) for patients assigned to the Modified WHO Surgical Safety Checklist compared to controls.21 Conversely, Haugen et al. reported a nonsignificant reduction in 30-day mortality between the WHO Surgical Safety Checklist group and controls (1.0% vs 1.6%, P = 0.151).36 Bassor et al. reported no significant difference in 30-day hospital readmission for decompensated heart failure for the heart failure discharge checklist group when compared to controls (6% vs. 4%, P = NS); however, an exploratory analysis that excluded patients who died during the follow-up period found a significant difference in 30-day readmission rates (2% vs. 20%, P = 0.02).30 Gentili et al. reported a higher proportion of patients with pain control in the checklist group compared to the controls (67.6% vs. 54.8%), as well as fewer incidents of analgesic therapy–related uncontrolled adverse events (25.9% vs. 49.9%); however, the statistical significance of these differences were not reported.35 The Writing Group for CHECKLIST-ICU reported no significant difference for in-hospital mortality between the checklist and control groups (adjusted odds ratio [AOR] 1.02, 95% CI, 0.82-1.26, P = 0.88), nor for the secondary clinical outcomes examined (Table 4).33 However, there was a significant difference between the checklist group and control group for 3 of the 7 outcomes related to processes of patient care, including a reduction in the use of both urinary catheters (adjusted rate ratio [ARR] 0.86, 95% CI, 0.80-0.93, P < 0.001) and central venous catheters (ARR 0.90, 95% CI 0.83-0.98, P = 0.02). Masson et al. reported that when using the FASTHUG-MAIDENS checklist, more drug-related problems were identified by pharmacy residents (in relation to the number identified by the ICU pharmacist) both per patient encounter (P = 0.008) and overall (P < 0.001).37 Ong et al. reported higher rates of compliance with isolation precautions for infectious diseases in the checklist group (71% vs. 38%, P < 0.01); however, compliance with the checklist was low (40%) and qualitative analyses found participants were dissatisfied with the checklist.31 Salzwedel et al. reported the number of items handed over by anesthesia residents postoperatively to be higher in the checklist group than the control group (48.7% vs. 32.4%, P < 0.001).32 In a more recent study, Salzwedel et al. reported that proportion of items deemed by the attending anesthesiologist as “must be handed over” were more often actually handed over by the anesthesia residents assigned to the checklist group when compared to controls (87.1% vs. 75.0%, P = 0.005).34
30-day Mortality
A random-effects model pooling data from the 3 studies that reported data for 30-day all cause mortality suggested a significant reduction with use of a checklist (OR 0.60, 95% CI, 0.41-0.89; P = 0.01, I2 = 0.0%, P = 0.573).
Study Quality
Supplementary File 5 (Table 5) summarizes the quality assessment of the 9 studies. The clarity of description for each intervention varied. All studies reported inclusion/exclusion criteria and randomization procedures. Three studies indicated that outcome assessors were blinded to intervention allocation;32,34,36 while this was unclear in 2 studies.21,30 Three studies reported baseline characteristics.21,30,36 Two studies reported power calculations;33,37 however, one study had a sample size that was less than that required to achieve the target power.37 The Jadad scores ranged from 1to 5.
DISCUSSION
This systematic review identified 9 RCTs that examined the impact of a checklist on patient safety outcomes in hospitalized patients. The studies employed checklists with different purposes and elements and measured different patient safety outcomes. The methodological quality of the included studies was moderate. In aggregate, the results suggest that checklists may be effective at improving patient safety outcomes, but the small number of moderate quality studies and the heterogeneity of interventions and outcome measures suggests that there is an urgent need for further evaluation.
The most important observation from our systematic review is the paucity of high quality evidence evaluating checklists’ impact on patient safety outcomes in acute inpatient care. The implementation of checklists is increasingly common as they are relatively low cost to develop and implement, and intuitively make sense. This is particularly true in an era of increasing efforts to standardize care as a means for improving quality and minimizing cost (ie, previous systematic reviews cite 38 unique studies).39 However, implementation of an inadequately tested checklist risks unintended consequences (eg, inefficient resource utilization).18 The small number of RCTs identified might be owing to quality improvement efforts traditionally focusing on ‘real life’ applicability over rigorous research methodology.40 The translation of evidence into clinical practice is known to be slow;41 however, these more rigorous methodologies reduce the risk of biases and generate high-quality evidence, which help to fulfill the necessity to identify best practices while avoiding these unintended consequences.
The studies varied both in the approaches used to develop checklists and in the number of items included (ranging from 2 to 54). What is the optimal method for developing a checklist and how does this impact their effectiveness?42 The answers to these questions are not known. However, this review highlights some important issues to consider when developing a checklist. As the number of items or complexity of a task increases, our ability to efficiently perform the task without aid decreases.43-45 As such, a well-designed checklist should detail explicit instructions on the what, where, when, and how of a given task in a fashion that ensures a consistent accuracy for completing the work.5 It is recommended that construction of a checklist follow the principles of human factors engineering: engage stakeholders and human factors experts in the design; are developed based on user needs and realities; list items in order of importance; are concise and subgroup sections of checklists by task or chronological order; ensure usability and evaluate potential negative consequences (eg time to complete); are pilot tested and validated before implementation; are updated as needed based the on generation of new findings or changes in operational procedures.46 These general principles of human factors engineering46 provide a practical approach for the development and evaluation of a checklist. In addition, standardization of operational definitions (ie, process, outcome, compliance) is important for study replication and robust meta-analyses.
Checklists used in aviation are perhaps best known12 and the evidence of their effectiveness is derived from the attribution of aviation errors to incomplete checklists.12 Although more recently implemented in medicine, checklists have the potential to guide the successful completion of complex tasks in healthcare.7 Systematic reviews of observational studies have been conducted for specific checklists (eg, WHO Surgical Safety Checklist) and for select patient populations (eg, surgical patients), and the number of included studies ranges from 7-27 (n = 38 unique studies).15,16,18,19 For example, Gillespie et al. in a systematic review and meta-analysis reported the implementation of Surgical Safety Checklists to be associated with a reduction in postoperative complications (relative risk [RR] 0.63, 95% CI, 0.58-0.72, P = < 0.001), but not mortality (RR 1.03, 95% CI, 0.73-1.4, P = 0.857).19 Similarly, Treadwell et al. reported in a systematic review of Surgical Safety Checklists that while data are promising, more evaluation of their impact on clinical outcomes is needed.18 These recommendations are nicely illustrated by Urbach et al.’s20 and O’Leary et al.’s47 evaluations of the mandatory adoption of Surgical Safety Checklists across all hospitals in Ontario, Canada, which respectively demonstrated no significant reductions in 30-day perioperatively conplications for both adult (OR 0.97, 95% CI, 0.90-1.03, P = 0.29) and pediatric (AOR 1.01, 95% CI, 0.90-1.14, P = 0.9) patients. These data not only highlight the need for further evaluation of checklists but are also a reminder that checklists and their associated implementation strategies are complex interventions for which there may be important differences between the efficacy reported in clinical trials and the effectiveness reported in implementation studies.48 This all suggests that if checklists are to be effective in improving patient safety, process evaluations of implementation49 and realist reviews of published studies50 may be important to determine optimal approaches for implementation. We believe that, based on the limited currently available evidence, there is urgency for further robust evaluations of checklists before their widespread implementation. If effective, they should be widely implemented. If ineffective, they should be abandoned to minimize unintended consequences and inefficient use of resources.
There are 4 primary limitations to this review that should be considered when interpreting the findings. First, the RCT design is not the study design employed by most quality improvement initiatives.40 While some quality improvement experts may argue that an RCT design is insufficiently flexible for applied settings, it does minimize the risk of biased assessments of intervention effectiveness. Second, our search strategy included an RCT filter. The filter helped restrict the number of citations to be reviewed (n = 11,225) but could have resulted in improperly indexed studies being excluded. To guard against this risk, we used the validated Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials,24 reviewed reference lists of citations included in the review, and solicited suggestions for missing studies from quality improvement experts. Third, our review was restricted to hospitalized patients. Although the studies evaluated commonly reported safety outcomes across patients with diverse clinical conditions, care settings, and providers that broadly reflect hospital-based care, evaluations of checklists in additional patient and provider groups are needed (eg, hospitalists). Furthermore, the effectiveness of checklists for improving patient safety outcomes in outpatients is important; however, the organizational and patient characteristics of these 2 settings (hospitalized vs outpatient) are sufficiently different to warrant separate systematic reviews. Finally, owing to the heterogeneity of the checklists used and outcomes measured, we were unable to perform a robust meta-analysis. Heterogeneity, combined with the small number of studies identified in our search, prevented us from applying statistical methods to assess for publication bias. This limitation of our systematic review highlights an important gap in the literature and emphasizes the importance of additional primary research to evaluate checklists.
In summary, we identified few RCTs that examined checklists designed to improve patient safety outcomes. The small number of existing studies suggests that checklists may improve patient safety outcomes; however, these observations were not reported for all outcomes examined and the studies were heterogeneous and of limited methodological quality. There is an urgent need for high-quality evaluations of the effectiveness of patient safety checklists in inpatient healthcare settings to substantiate their perceived benefits.
Acknowledgments
We would like to thank Diane Lorenzetti for her help with the development of the search strategy.
Disclosure: The authors have no known conflicts of interest to declare.
Jamie Boyd was supported by a W21C – Alberta Innovates-Health Solutions (AIHS) Collaborative Research and Innovation Opportunities (CRIO) Health Services Research graduate studentship. Guosong Wu was supported by a Western Regional Training Centre (WRTC) for Health Services Research graduate studentship. Dr. Stelfox was supported by a Population Health Investigator Award from Alberta Innovates Health Solutions.
Authors’ Contributions
HTS was responsible for the study’s conception. All 3 authors contributed to the study’s design and interpretation. JB and GW were responsible for searching the literature, reviewing abstracts, selecting full-text articles and critically appraising them. All 3 authors performed the analyses. JB drafted the manuscript and all 3 authors assisted in the successive revisions of the final manuscript. All authors have read and approved the final manuscript.
In response to widely publicized reports highlighting the challenges of suboptimal quality of healthcare, improving patient safety has been a leading healthcare initiative for more than 10 years.1-4 Numerous strategies to improve patient safety have been proposed,5-9 but improvements have been limited, which raises questions about whether the right approaches are being employed.10,11
Checklists have served as a foundation for the standardization and safety of aviation and nuclear power12,13 and are advocated as simple and effective instruments for ensuring safe care.7,14,15 Systematic reviews of observational studies suggest that checklists can reduce medical errors and adverse events,15-19 but these reviews are at risk of bias due to the limitations of observational methods. Furthermore, discordant results of recent high-profile evaluations of the World Health Organization (WHO) Surgical Safety Checklist highlight the need for checklist evaluations using rigorous study designs.20-22 Therefore, we sought to conduct a systematic review of RCTs (randomized controlled trials) to determine whether checklists, as a type of decision-support tool, are effective at improving patient safety outcomes in hospitalized patients.
METHODS
The study protocol was registered with the PROSPERO Register of Systematic Reviews (registration number: CRD42016037441) and developed according to the Preferred Reporting Items in Systematic Reviews and Meta-analyses (PRISMA) statement.23
Search Strategy
On December 8, 2016, we systematically searched Ovid MEDLINE, Ovid EMBASE, PubMed, and the Cochrane Central Register of Controlled Trials. The search was performed using no language or publication date restrictions and included 2 groups of terms (key words with similar characteristics): ‘checklists’ and ‘patient outcomes assessment’. We restricted our search to patient outcomes because these are more patient-oriented than the proximal processes of care that may not translate into outcomes. The search was restricted to RCTs using the Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials from the Cochrane Collaborative.24 The MEDLINE search strategy is depicted in Appendix I (Supplementary File 1). Reference lists of included articles were manually searched for additional publications. The search strategy was designed with the help of an information scientist (DL). EndNote X7 (Thomas Reuters, Philadelphia, PA, USA) was the reference software used for the management of citations.
Eligibility Criteria
We selected all studies reporting patient safety outcomes of a checklist intervention, using the following inclusion criteria: 1) acute care hospital inpatient population, 2) checklist intervention, 3) contain a control group (ie, no checklist), 4) report one or more patient safety outcome, as defined by the authors (eg, medical errors, adverse events, mortality), and 5) RCT design. We restricted our focus to inpatient populations given the heterogeneity of illness and patient care between acute and community settings. We defined a checklist as a tool that details the essential steps of a task, requiring the target provider to indicate whether an item was completed or not.1,7 Tools that included only 1 item (eg, electronic prompts) or did not require acknowledgement of the items (eg, guidelines) were excluded. We defined patient safety outcomes as the authors’ definition of patient safety (eg, medical error, adverse event, provider compliance with safety regulations).
Study Selection
Two reviewers (JMB, GW) independently, and in duplicate, reviewed the titles and abstracts of the retrieved citations against the eligibility criteria. The same 2 reviewers subsequently reviewed the full text of relevant articles for inclusion. Eligibility disagreements were resolved by consensus. A Kappa statistic was calculated for reviewer agreement of full-text screening.25 Reviewers were not blinded to author or journal names.26
Data Extraction
The structured data extraction form was calibrated using the first 2 articles. The 2 reviewers (JMB, GW) independently, and in duplicate, extracted data from included studies on the study characteristics, setting, study population, sample size, intervention used, outcomes examined, analytic method, and study quality. The data extraction form is depicted in Appendix II (Supplementary File 2). Coding discrepancies were resolved by consensus.
Quality Assessment
The 2 reviewers (JMB, GW) extracted data on study quality independently and in duplicate using 2 approaches. First, reviewers assessed study quality using a component method derived from the Cochrane Collaboration criteria.24 For each included study, the reviewers documented if the authors had adequately described inclusion/exclusion criteria, randomization, allocation concealment, blinding of participants/outcome assessors, attrition, cross over, baseline characteristics, and power calculation. Second, the reviewers calculated and reported the Jadad score for each included study, a validated assessment scale that assigns points (1 to 5) based on randomization, blinding, and attrition.27
Analysis
Owing to the heterogeneity of the data and the small number of studies that satisfied the inclusion criteria, the data were analyzed using guidelines for the narrative synthesis of a systematic review.28 Descriptive statistical findings from each included study were reported. The DerSimonian and Laird method for random-effects models was used to calculate a pooled estimate of 30-day all-cause mortality from the raw data available from a subset of studies (number of events, study population).29 Stata SE version 13.1 (Stata Corp, LP, College Station, TX) was used to perform the statistical analyses.
RESULTS
The literature search identified 11,225 unique citations from which 83 abstracts were eligible for full-text review. We identified 9 full-text articles for inclusion in the review (Figure 1 [Supplementary File 3]). The main reasons for citation exclusion during the full-text review were that the study design was not an RCT (39%) or there was no checklist intervention (34%
Study Characteristics
Characteristics of the included studies are summarized in Tables 1 and 2. Six of the studies were conducted in at least 1teaching hospital.30-35 The studies varied in target populations for both the checklist user and patients. The outcomes reported varied; 3 studies examined 30-day mortality,21,30,36 4 studies examined hospital length of stay,21,30,33,36 and 2 studies reported user compliance with the checklist.21,31 Five of the studies reported patient outcomes,21,30,33,35,36 and 5 studies reported provider-level outcomes related to patient safety (eg, compliance with checklist items such as communication of medications, isolation precautions, etc.).31-34,37
Description of Checklists
Supplementary File 4 (Table 3) provides a detailed breakdown of the checklists’ purpose and components. Six of the checklists were designed to directly reduce patient safety events,21,30,33,35-37 whereas 3 of the checklists were designed to indirectly reduce patient safety events by increasing compliance with processes of care.31,32,34 Six checklists were constructed and pilot tested by the research team conducting the RCT30-35 and the 3 remaining studies used modified versions of previously validated checklists.21,36,37 The number of items included in the checklist ranged from 2 to 54.
Impact of the Checklist
Table 4 summarizes the adverse events, medical errors, resource utilization and/or compliance reported for each checklist. Chaudhary et al. reported significant decreases in Grade III (requiring intervention)38 and IV (life-threatening)38 postoperative complications (23% v. 33%, P = 0.04) and 30-day mortality (5.7% vs 10.0%, P = 0.04) for patients assigned to the Modified WHO Surgical Safety Checklist compared to controls.21 Conversely, Haugen et al. reported a nonsignificant reduction in 30-day mortality between the WHO Surgical Safety Checklist group and controls (1.0% vs 1.6%, P = 0.151).36 Bassor et al. reported no significant difference in 30-day hospital readmission for decompensated heart failure for the heart failure discharge checklist group when compared to controls (6% vs. 4%, P = NS); however, an exploratory analysis that excluded patients who died during the follow-up period found a significant difference in 30-day readmission rates (2% vs. 20%, P = 0.02).30 Gentili et al. reported a higher proportion of patients with pain control in the checklist group compared to the controls (67.6% vs. 54.8%), as well as fewer incidents of analgesic therapy–related uncontrolled adverse events (25.9% vs. 49.9%); however, the statistical significance of these differences were not reported.35 The Writing Group for CHECKLIST-ICU reported no significant difference for in-hospital mortality between the checklist and control groups (adjusted odds ratio [AOR] 1.02, 95% CI, 0.82-1.26, P = 0.88), nor for the secondary clinical outcomes examined (Table 4).33 However, there was a significant difference between the checklist group and control group for 3 of the 7 outcomes related to processes of patient care, including a reduction in the use of both urinary catheters (adjusted rate ratio [ARR] 0.86, 95% CI, 0.80-0.93, P < 0.001) and central venous catheters (ARR 0.90, 95% CI 0.83-0.98, P = 0.02). Masson et al. reported that when using the FASTHUG-MAIDENS checklist, more drug-related problems were identified by pharmacy residents (in relation to the number identified by the ICU pharmacist) both per patient encounter (P = 0.008) and overall (P < 0.001).37 Ong et al. reported higher rates of compliance with isolation precautions for infectious diseases in the checklist group (71% vs. 38%, P < 0.01); however, compliance with the checklist was low (40%) and qualitative analyses found participants were dissatisfied with the checklist.31 Salzwedel et al. reported the number of items handed over by anesthesia residents postoperatively to be higher in the checklist group than the control group (48.7% vs. 32.4%, P < 0.001).32 In a more recent study, Salzwedel et al. reported that proportion of items deemed by the attending anesthesiologist as “must be handed over” were more often actually handed over by the anesthesia residents assigned to the checklist group when compared to controls (87.1% vs. 75.0%, P = 0.005).34
30-day Mortality
A random-effects model pooling data from the 3 studies that reported data for 30-day all cause mortality suggested a significant reduction with use of a checklist (OR 0.60, 95% CI, 0.41-0.89; P = 0.01, I2 = 0.0%, P = 0.573).
Study Quality
Supplementary File 5 (Table 5) summarizes the quality assessment of the 9 studies. The clarity of description for each intervention varied. All studies reported inclusion/exclusion criteria and randomization procedures. Three studies indicated that outcome assessors were blinded to intervention allocation;32,34,36 while this was unclear in 2 studies.21,30 Three studies reported baseline characteristics.21,30,36 Two studies reported power calculations;33,37 however, one study had a sample size that was less than that required to achieve the target power.37 The Jadad scores ranged from 1to 5.
DISCUSSION
This systematic review identified 9 RCTs that examined the impact of a checklist on patient safety outcomes in hospitalized patients. The studies employed checklists with different purposes and elements and measured different patient safety outcomes. The methodological quality of the included studies was moderate. In aggregate, the results suggest that checklists may be effective at improving patient safety outcomes, but the small number of moderate quality studies and the heterogeneity of interventions and outcome measures suggests that there is an urgent need for further evaluation.
The most important observation from our systematic review is the paucity of high quality evidence evaluating checklists’ impact on patient safety outcomes in acute inpatient care. The implementation of checklists is increasingly common as they are relatively low cost to develop and implement, and intuitively make sense. This is particularly true in an era of increasing efforts to standardize care as a means for improving quality and minimizing cost (ie, previous systematic reviews cite 38 unique studies).39 However, implementation of an inadequately tested checklist risks unintended consequences (eg, inefficient resource utilization).18 The small number of RCTs identified might be owing to quality improvement efforts traditionally focusing on ‘real life’ applicability over rigorous research methodology.40 The translation of evidence into clinical practice is known to be slow;41 however, these more rigorous methodologies reduce the risk of biases and generate high-quality evidence, which help to fulfill the necessity to identify best practices while avoiding these unintended consequences.
The studies varied both in the approaches used to develop checklists and in the number of items included (ranging from 2 to 54). What is the optimal method for developing a checklist and how does this impact their effectiveness?42 The answers to these questions are not known. However, this review highlights some important issues to consider when developing a checklist. As the number of items or complexity of a task increases, our ability to efficiently perform the task without aid decreases.43-45 As such, a well-designed checklist should detail explicit instructions on the what, where, when, and how of a given task in a fashion that ensures a consistent accuracy for completing the work.5 It is recommended that construction of a checklist follow the principles of human factors engineering: engage stakeholders and human factors experts in the design; are developed based on user needs and realities; list items in order of importance; are concise and subgroup sections of checklists by task or chronological order; ensure usability and evaluate potential negative consequences (eg time to complete); are pilot tested and validated before implementation; are updated as needed based the on generation of new findings or changes in operational procedures.46 These general principles of human factors engineering46 provide a practical approach for the development and evaluation of a checklist. In addition, standardization of operational definitions (ie, process, outcome, compliance) is important for study replication and robust meta-analyses.
Checklists used in aviation are perhaps best known12 and the evidence of their effectiveness is derived from the attribution of aviation errors to incomplete checklists.12 Although more recently implemented in medicine, checklists have the potential to guide the successful completion of complex tasks in healthcare.7 Systematic reviews of observational studies have been conducted for specific checklists (eg, WHO Surgical Safety Checklist) and for select patient populations (eg, surgical patients), and the number of included studies ranges from 7-27 (n = 38 unique studies).15,16,18,19 For example, Gillespie et al. in a systematic review and meta-analysis reported the implementation of Surgical Safety Checklists to be associated with a reduction in postoperative complications (relative risk [RR] 0.63, 95% CI, 0.58-0.72, P = < 0.001), but not mortality (RR 1.03, 95% CI, 0.73-1.4, P = 0.857).19 Similarly, Treadwell et al. reported in a systematic review of Surgical Safety Checklists that while data are promising, more evaluation of their impact on clinical outcomes is needed.18 These recommendations are nicely illustrated by Urbach et al.’s20 and O’Leary et al.’s47 evaluations of the mandatory adoption of Surgical Safety Checklists across all hospitals in Ontario, Canada, which respectively demonstrated no significant reductions in 30-day perioperatively conplications for both adult (OR 0.97, 95% CI, 0.90-1.03, P = 0.29) and pediatric (AOR 1.01, 95% CI, 0.90-1.14, P = 0.9) patients. These data not only highlight the need for further evaluation of checklists but are also a reminder that checklists and their associated implementation strategies are complex interventions for which there may be important differences between the efficacy reported in clinical trials and the effectiveness reported in implementation studies.48 This all suggests that if checklists are to be effective in improving patient safety, process evaluations of implementation49 and realist reviews of published studies50 may be important to determine optimal approaches for implementation. We believe that, based on the limited currently available evidence, there is urgency for further robust evaluations of checklists before their widespread implementation. If effective, they should be widely implemented. If ineffective, they should be abandoned to minimize unintended consequences and inefficient use of resources.
There are 4 primary limitations to this review that should be considered when interpreting the findings. First, the RCT design is not the study design employed by most quality improvement initiatives.40 While some quality improvement experts may argue that an RCT design is insufficiently flexible for applied settings, it does minimize the risk of biased assessments of intervention effectiveness. Second, our search strategy included an RCT filter. The filter helped restrict the number of citations to be reviewed (n = 11,225) but could have resulted in improperly indexed studies being excluded. To guard against this risk, we used the validated Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials,24 reviewed reference lists of citations included in the review, and solicited suggestions for missing studies from quality improvement experts. Third, our review was restricted to hospitalized patients. Although the studies evaluated commonly reported safety outcomes across patients with diverse clinical conditions, care settings, and providers that broadly reflect hospital-based care, evaluations of checklists in additional patient and provider groups are needed (eg, hospitalists). Furthermore, the effectiveness of checklists for improving patient safety outcomes in outpatients is important; however, the organizational and patient characteristics of these 2 settings (hospitalized vs outpatient) are sufficiently different to warrant separate systematic reviews. Finally, owing to the heterogeneity of the checklists used and outcomes measured, we were unable to perform a robust meta-analysis. Heterogeneity, combined with the small number of studies identified in our search, prevented us from applying statistical methods to assess for publication bias. This limitation of our systematic review highlights an important gap in the literature and emphasizes the importance of additional primary research to evaluate checklists.
In summary, we identified few RCTs that examined checklists designed to improve patient safety outcomes. The small number of existing studies suggests that checklists may improve patient safety outcomes; however, these observations were not reported for all outcomes examined and the studies were heterogeneous and of limited methodological quality. There is an urgent need for high-quality evaluations of the effectiveness of patient safety checklists in inpatient healthcare settings to substantiate their perceived benefits.
Acknowledgments
We would like to thank Diane Lorenzetti for her help with the development of the search strategy.
Disclosure: The authors have no known conflicts of interest to declare.
Jamie Boyd was supported by a W21C – Alberta Innovates-Health Solutions (AIHS) Collaborative Research and Innovation Opportunities (CRIO) Health Services Research graduate studentship. Guosong Wu was supported by a Western Regional Training Centre (WRTC) for Health Services Research graduate studentship. Dr. Stelfox was supported by a Population Health Investigator Award from Alberta Innovates Health Solutions.
Authors’ Contributions
HTS was responsible for the study’s conception. All 3 authors contributed to the study’s design and interpretation. JB and GW were responsible for searching the literature, reviewing abstracts, selecting full-text articles and critically appraising them. All 3 authors performed the analyses. JB drafted the manuscript and all 3 authors assisted in the successive revisions of the final manuscript. All authors have read and approved the final manuscript.
1. World Health Organization. Patient safety. Available at: http://www.who.int/patientsafety/about/en/. Accessed June 21, 2016.
2. Institute of Medicine. To err is human: Building a safer health system. In: Kohn L, Corrigan J, Donaldson M, eds. Institute of Medicine-Committee on Quality of Health Care in America. Washington DC: National Academy Press; 1999:86-101. PubMed
3. Institute of Medicine Committee on the Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington DC: National Academy Press; 2001. PubMed
4. Stelfox HT, Palmisani S, Scurlock C, Orav EJ, Bates DW. The “to err is human” report and the patient safety literature. Qual Saf Health Care. 2006; 15(3):174-178. PubMed
5. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost, PJ. Clinical review: Checklists - translating evidence into practice. Crit Care. 2009; 13(6):210. PubMed
6. Ely EW, Bennett PA, Bowton DL, Murphy SM, Florance AM, Haponik EF. Large scale implementation of a respiratory therapist-driven protocol for ventilator weaning. Am J Respir Crit Care Med. 1999; 159(2):439-446. PubMed
7. Gawande A. The checklist manifesto: How to get things right. Great Britain: Profile Books LTD; 2010.
8. Pronovost P, Vohr E. Safe patients, smart hospitals. New York, NY: Hudson Street Press; 2010.
9. Hughes RG. Advances in patient safety: Tools and strategies for quality improvement and patient safety. In: Hughes RG, ed. Patient safety and quality: An evidence-based handbook for nurses. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. PubMed
10. Henriksen K, Oppenheimer C, Leape LL, et al. Envisioning patient safety in the year 2025: Eight perspectives. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: New directions and alternative approaches. Rockville, MD: Agency for Healthcare Research and Quality; 2008. PubMed
11. Gaba DM, Howard SK. Patient safety: Fatigue among clinicians and the safety of patients. N Engl J Med. 2002; 347(16):1249-1255. PubMed
12. Degani A, Wiener EL. Cockpit checklists: Concepts, design, and use. Human Factors: The Journal of the Human Factors and Ergonomics Society 1993; 35(2):345-359.
13. Swain AD, Guttmann HE. Handbook of human reliability analysis with emphasis on nuclear power plant applications: Final report. Washington, DC: U.S. Nuclear Regulatroy Commission; 1983.
14. de Vries EN, Prins HA, Crolla RM, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010; 363(20):1928-1937. PubMed
15. Bergs J, Hellings J, Cleemput I, et al. Systematic review and meta-analysis of the effect of the world health organization surgical safety checklist on postoperative complications. Br J Surg. 2014; 101(3):150-158. PubMed
16. Pucher PH, Johnston MJ, Aggarwal R, Arora S, Darzi A. Effectiveness of interventions to improve patient handover in surgery: A systematic review. Surgery. 2015; 158(1):85-95. PubMed
17. Bergs J, Lambrechts F, Simons P, et al. Barriers and facilitators related to the implementation of surgical safety checklists: A systematic review of the qualitative evidence. BMJ Qual Saf. 2015; 23(12):776-786. PubMed
18. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: A systematic review of impacts and implementation. BMJ Qual Saf. 2014; 23(4):299-318. PubMed
19. Gillespie BM, Chaboyer W, Thalib L, John M, Fairweather N, Slater K. Effect of using a safety checklist on patient complications after surgery: A systematic review and meta-analysis. Anesthesiology. 2014; 120(6):1380-1389. PubMed
20. Reames BN, Krell RW, Campbell DA Jr, Dimick JB. A checklist-based intervention to improve surgical outcomes in michigan: Evaluation of the keystone surgery program. JAMA Surg. 2015; 150(3):208-215. PubMed
21. Chaudhary N, Varma V, Kapoor S, Mehta N, Kumaran V, Nundy S. Implementation of a surgical safety checklist and postoperative outcomes: A prospective randomized controlled study. J Gastrointest Surg. 2015; 19(5):935-942. PubMed
22. Reames BN, Krell RW, Campbell DA, Jr., Dimick JB. A checklist-based intervention to improve surgical outcomes in Michigan: Evaluation of the Keystone Surgery program. JAMA surgery. 2015; 150(3):208-215. PubMed
23. Liberati A, Altman DG, Tetzlaff J, et al. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med. 2009; 151(4):W65-94. PubMed
24. The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions, version 5.1.0. Oxford, UK: The Cochrane Collaboration, 2011.
25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-174. PubMed
26. Berlin JA. Does blinding of readers affect the results of meta-analyses? University of pennsylvania meta-analysis blinding study group. Lancet 1997;350(9072):185-186. PubMed
27. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996;17(1):1-12. PubMed
28. Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews: A product form the esrc methods programme. Available at: https://www.researchgate.net/profile/Mark_Rodgers4/publication/233866356_Guidance_on_the_conduct_of_narrative_synthesis_in_systematic_reviews_A_product_from_the_ESRC_Methods_Programme/links/02e7e5231e8f3a6183000000.pdf. Accessed June 17, 2016.
29. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177-188. PubMed
30. Basoor A, Doshi NC, Cotant JF, et al. Decreased readmissions and improved quality of care with the use of an inexpensive checklist in heart failure. Congest Heart Fail. 2013; 19(4):200-206. PubMed
31. Ong MS, Magrabi F, Post J, et al. Communication interventions to improve adherence to infection control precautions: A randomised crossover trial. BMC Infect Dis. 2013; 13:72. PubMed
32. Salzwedel C, Bartz HJ, Kuhnelt I, et al. The effect of a checklist on the quality of post-anaesthesia patient handover: A randomized controlled trial. Int J Qual Health Care. 2013; 25(2):176-181.
33. Implement Sci. 50. Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: Illustrating the method for implementation research. 2008; 337.BMJ. PubMed
49. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: The new medical research council guidance. 2014; 9(9):e108585.PloS one. PubMed
48. Gagliardi AR, Straus SE, Shojania KG, Urbach DR. Multiple interacting factors influence adherence, and outcomes associated with surgical safety checklists: A qualitative study.
2016; 188(9):E191-E198.CMAJ. PubMed
47. O’Leary JD, Wijeysundera DN, Crawford MW. Effect of surgical safety checklists on pediatric surgical complications in Ontario. Rockville, MD: Agency for Healthcare Research and Quality; 2013.Human factors and ergonomics. Making health care safer ii: An updated critical analysis of the evidence for patient safety practices. PubMed
46. Carayon P, Xie A, Kianfar SH. 2005; 16(1):70-76.Psychol Sci. 2004; 30(4):689-707.
45. Halford GS, Baker R, McCredden JE, Bain JD. How many variables can humans process? J Exp Psychol Hum Percept Perform. PubMed
44. Oberauer K, Kliegl R. Simultaneous cognitive operations in working memory after dual-task practice. 1956; 63(2):81-97.Psychol Rev. PubMed
43. Miller GA. The magical number seven plus or minus two: Some limits on our capacity for processing information. 2008; 20(1):22-30.Int J Qual Health Care. PubMed
42. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. 2011; 104(12):510-520.J R Soc Med. PubMed
41. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. 2015; 24(5):325-336.BMJ Qual Saf. PubMed
40. Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: A brief overview of possible study types. Washington (DC): National Academies Press (US); 2013.Best care at lower cost: The path to continuously learning health care in america. PubMed
39. Institute of Medicine. Committee on the learning health care system in America. In: Smith M, Saunders R, Stuckhardt L, et al., eds. 2004; 240(2):205-213.Ann Surg. PubMed
38. Dindo D, Demartines N, Clavien P-A. Classification of surgical complications: A new proposal with evaluation in a cohort of 6336 patients and results of a survey. 2013; 66(3):157-162.Can J Hosp Pharm. PubMed
37. Masson SC, Mabasa VH, Malyuk DL, Perrott JL. Validity evidence for fasthug-maidens, a mnemonic for identifying drug-related problems in the intensive care unit. 2015; 261(5):821-828.Ann Surg. PubMed
36. Haugen AS, Softeland E, Almeland SK, et al. Effect of the world health organization checklist on patient outcomes: A stepped wedge cluster randomized controlled trial.
2016; 12(2):199-205.Future Oncol. PubMed
35. Gentili M, Clerico M, Spizzichino M, Fanelli G. Use of a checklist to improve pain control in hospitalized cancer patients: the 38Checkpain project. 2016; 32:170-174.J Crit Care. PubMed
34. Salzwedel C, Mai V, Punke MA, Kluge S, Reuter DA. The effect of a checklist on the quality of patient handover from the operating room to the intensive care unit: A randomized controlled trial. 2016; 315(14):1480-1490.JAMA. PubMed
33. The Writing Group for CHECKLIST-ICU, Cavalcanti AB, Bozza FA, et al. Effect of a Quality Improvement Intervention With Daily Round Checklists, Goal Setting, and Clinician Prompting on Mortality of Critically Ill Patients: A Randomized Clinical Trial. PubMed
1. World Health Organization. Patient safety. Available at: http://www.who.int/patientsafety/about/en/. Accessed June 21, 2016.
2. Institute of Medicine. To err is human: Building a safer health system. In: Kohn L, Corrigan J, Donaldson M, eds. Institute of Medicine-Committee on Quality of Health Care in America. Washington DC: National Academy Press; 1999:86-101. PubMed
3. Institute of Medicine Committee on the Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington DC: National Academy Press; 2001. PubMed
4. Stelfox HT, Palmisani S, Scurlock C, Orav EJ, Bates DW. The “to err is human” report and the patient safety literature. Qual Saf Health Care. 2006; 15(3):174-178. PubMed
5. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost, PJ. Clinical review: Checklists - translating evidence into practice. Crit Care. 2009; 13(6):210. PubMed
6. Ely EW, Bennett PA, Bowton DL, Murphy SM, Florance AM, Haponik EF. Large scale implementation of a respiratory therapist-driven protocol for ventilator weaning. Am J Respir Crit Care Med. 1999; 159(2):439-446. PubMed
7. Gawande A. The checklist manifesto: How to get things right. Great Britain: Profile Books LTD; 2010.
8. Pronovost P, Vohr E. Safe patients, smart hospitals. New York, NY: Hudson Street Press; 2010.
9. Hughes RG. Advances in patient safety: Tools and strategies for quality improvement and patient safety. In: Hughes RG, ed. Patient safety and quality: An evidence-based handbook for nurses. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. PubMed
10. Henriksen K, Oppenheimer C, Leape LL, et al. Envisioning patient safety in the year 2025: Eight perspectives. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: New directions and alternative approaches. Rockville, MD: Agency for Healthcare Research and Quality; 2008. PubMed
11. Gaba DM, Howard SK. Patient safety: Fatigue among clinicians and the safety of patients. N Engl J Med. 2002; 347(16):1249-1255. PubMed
12. Degani A, Wiener EL. Cockpit checklists: Concepts, design, and use. Human Factors: The Journal of the Human Factors and Ergonomics Society 1993; 35(2):345-359.
13. Swain AD, Guttmann HE. Handbook of human reliability analysis with emphasis on nuclear power plant applications: Final report. Washington, DC: U.S. Nuclear Regulatroy Commission; 1983.
14. de Vries EN, Prins HA, Crolla RM, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010; 363(20):1928-1937. PubMed
15. Bergs J, Hellings J, Cleemput I, et al. Systematic review and meta-analysis of the effect of the world health organization surgical safety checklist on postoperative complications. Br J Surg. 2014; 101(3):150-158. PubMed
16. Pucher PH, Johnston MJ, Aggarwal R, Arora S, Darzi A. Effectiveness of interventions to improve patient handover in surgery: A systematic review. Surgery. 2015; 158(1):85-95. PubMed
17. Bergs J, Lambrechts F, Simons P, et al. Barriers and facilitators related to the implementation of surgical safety checklists: A systematic review of the qualitative evidence. BMJ Qual Saf. 2015; 23(12):776-786. PubMed
18. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: A systematic review of impacts and implementation. BMJ Qual Saf. 2014; 23(4):299-318. PubMed
19. Gillespie BM, Chaboyer W, Thalib L, John M, Fairweather N, Slater K. Effect of using a safety checklist on patient complications after surgery: A systematic review and meta-analysis. Anesthesiology. 2014; 120(6):1380-1389. PubMed
20. Reames BN, Krell RW, Campbell DA Jr, Dimick JB. A checklist-based intervention to improve surgical outcomes in michigan: Evaluation of the keystone surgery program. JAMA Surg. 2015; 150(3):208-215. PubMed
21. Chaudhary N, Varma V, Kapoor S, Mehta N, Kumaran V, Nundy S. Implementation of a surgical safety checklist and postoperative outcomes: A prospective randomized controlled study. J Gastrointest Surg. 2015; 19(5):935-942. PubMed
22. Reames BN, Krell RW, Campbell DA, Jr., Dimick JB. A checklist-based intervention to improve surgical outcomes in Michigan: Evaluation of the Keystone Surgery program. JAMA surgery. 2015; 150(3):208-215. PubMed
23. Liberati A, Altman DG, Tetzlaff J, et al. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med. 2009; 151(4):W65-94. PubMed
24. The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions, version 5.1.0. Oxford, UK: The Cochrane Collaboration, 2011.
25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-174. PubMed
26. Berlin JA. Does blinding of readers affect the results of meta-analyses? University of pennsylvania meta-analysis blinding study group. Lancet 1997;350(9072):185-186. PubMed
27. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996;17(1):1-12. PubMed
28. Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews: A product form the esrc methods programme. Available at: https://www.researchgate.net/profile/Mark_Rodgers4/publication/233866356_Guidance_on_the_conduct_of_narrative_synthesis_in_systematic_reviews_A_product_from_the_ESRC_Methods_Programme/links/02e7e5231e8f3a6183000000.pdf. Accessed June 17, 2016.
29. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7(3):177-188. PubMed
30. Basoor A, Doshi NC, Cotant JF, et al. Decreased readmissions and improved quality of care with the use of an inexpensive checklist in heart failure. Congest Heart Fail. 2013; 19(4):200-206. PubMed
31. Ong MS, Magrabi F, Post J, et al. Communication interventions to improve adherence to infection control precautions: A randomised crossover trial. BMC Infect Dis. 2013; 13:72. PubMed
32. Salzwedel C, Bartz HJ, Kuhnelt I, et al. The effect of a checklist on the quality of post-anaesthesia patient handover: A randomized controlled trial. Int J Qual Health Care. 2013; 25(2):176-181.
33. Implement Sci. 50. Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: Illustrating the method for implementation research. 2008; 337.BMJ. PubMed
49. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: The new medical research council guidance. 2014; 9(9):e108585.PloS one. PubMed
48. Gagliardi AR, Straus SE, Shojania KG, Urbach DR. Multiple interacting factors influence adherence, and outcomes associated with surgical safety checklists: A qualitative study.
2016; 188(9):E191-E198.CMAJ. PubMed
47. O’Leary JD, Wijeysundera DN, Crawford MW. Effect of surgical safety checklists on pediatric surgical complications in Ontario. Rockville, MD: Agency for Healthcare Research and Quality; 2013.Human factors and ergonomics. Making health care safer ii: An updated critical analysis of the evidence for patient safety practices. PubMed
46. Carayon P, Xie A, Kianfar SH. 2005; 16(1):70-76.Psychol Sci. 2004; 30(4):689-707.
45. Halford GS, Baker R, McCredden JE, Bain JD. How many variables can humans process? J Exp Psychol Hum Percept Perform. PubMed
44. Oberauer K, Kliegl R. Simultaneous cognitive operations in working memory after dual-task practice. 1956; 63(2):81-97.Psychol Rev. PubMed
43. Miller GA. The magical number seven plus or minus two: Some limits on our capacity for processing information. 2008; 20(1):22-30.Int J Qual Health Care. PubMed
42. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. 2011; 104(12):510-520.J R Soc Med. PubMed
41. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: Understanding time lags in translational research. 2015; 24(5):325-336.BMJ Qual Saf. PubMed
40. Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: A brief overview of possible study types. Washington (DC): National Academies Press (US); 2013.Best care at lower cost: The path to continuously learning health care in america. PubMed
39. Institute of Medicine. Committee on the learning health care system in America. In: Smith M, Saunders R, Stuckhardt L, et al., eds. 2004; 240(2):205-213.Ann Surg. PubMed
38. Dindo D, Demartines N, Clavien P-A. Classification of surgical complications: A new proposal with evaluation in a cohort of 6336 patients and results of a survey. 2013; 66(3):157-162.Can J Hosp Pharm. PubMed
37. Masson SC, Mabasa VH, Malyuk DL, Perrott JL. Validity evidence for fasthug-maidens, a mnemonic for identifying drug-related problems in the intensive care unit. 2015; 261(5):821-828.Ann Surg. PubMed
36. Haugen AS, Softeland E, Almeland SK, et al. Effect of the world health organization checklist on patient outcomes: A stepped wedge cluster randomized controlled trial.
2016; 12(2):199-205.Future Oncol. PubMed
35. Gentili M, Clerico M, Spizzichino M, Fanelli G. Use of a checklist to improve pain control in hospitalized cancer patients: the 38Checkpain project. 2016; 32:170-174.J Crit Care. PubMed
34. Salzwedel C, Mai V, Punke MA, Kluge S, Reuter DA. The effect of a checklist on the quality of patient handover from the operating room to the intensive care unit: A randomized controlled trial. 2016; 315(14):1480-1490.JAMA. PubMed
33. The Writing Group for CHECKLIST-ICU, Cavalcanti AB, Bozza FA, et al. Effect of a Quality Improvement Intervention With Daily Round Checklists, Goal Setting, and Clinician Prompting on Mortality of Critically Ill Patients: A Randomized Clinical Trial. PubMed
© 2017 Society of Hospital Medicine