Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.