Few consumers would choose to dine at a restaurant if they knew the kitchen was infested with cockroaches. Few patients would choose to undergo a liver transplant in a hospital that was performing the procedure for the first time. In most sectors, consumers gather information about quality (and price) from the marketplace, where economic theory predicts that rational behavior and competition will lead to continuous improvement over time. However, for some goods and services, information is sparse and asymmetric between consumers and suppliers. In sectors where consumer health is at risk, society has often intervened to assure minimum standards. Yet sometimes these efforts have fallen short. In healthcare, physician licensure and hospital accreditation (eg, through the Joint Commission), although providing an important foundation to assure safety, have not come close to solving the widespread quality problems.[1] Basic regulatory requirements for restaurants have also proven inadequate to prevent food‐borne illness. Consumer trust, without information, can be a recipe (or prescription) for trouble.

In response, high‐profile efforts have been introduced to publicize the quality and safety of service providers. One example is Hospital Compare, Medicare's national quality reporting program for US hospitals.[2] The New York City sanitary grade inspection program is a parallel effort for restaurants. Although customers can judge how much they like the food from a restaurantor look up reviews at Yelp.comthey face greater difficulty identifying whether a restaurant was responsible for making them sick. By publicizing restaurants' sanitation conditions, the New York City inspection program seeks to use market forces to decrease food‐borne illness by deterring consumers from eating at restaurants with poor sanitation grades.

The aims of Hospital Compare and the New York City sanitary inspection program are fundamentally similar. Both initiatives seek to address a common market failure resulting in the consumer's lack of information on quality and safety. By infusing the market with information, these programs enable consumers to make better choices and encourage service providers to improve quality and safety.[3] Despite the promise of these programs, a copious literature about the effects of public quality reporting in healthcare has found mixed results.[4, 5] Although the performance measures in any public reporting program must be valid and reliable, good measures are not sufficient to achieve the goals of public reporting. To engage patients, reported results must also be accessible, understandable, and meaningful. Both patients' lack of knowledge about the reports[6] and patients' inability to effectively use these data to make better decisions[7] are some reasons why public quality reporting has fallen short of its expectations. This article argues that the New York City program is much better structured to positively affect patient choice, and holds important lessons for public quality reporting in US hospitals.

CONTRASTS BETWEEN HOSPITAL COMPARE AND THE NEW YORK CITY RESTAURANT SANITARY INSPECTION PROGRAM

Hospital Compare reports performance for 108 separate quality indicators related to quality and patient safety for US hospitals (Table 1). These are a combination of structure measures (eg, hospital participation in a systematic database for cardiac surgery), process of care measures (eg, acute myocardial infarction patients receiving fibrinolytic therapy within 30 minutes of hospital arrival), outcomes (eg, 30‐day mortality and readmission), and patient experience measures (eg, how you would rate your communication with your physician). Hospital Compare data, frequently based on hospital quality performance 1 to 3 years prior to publication, are displayed on a website. Hospitals do not receive a summary measure of quality or safety.[8] Hospitals face financial incentives that are tied to measure reporting[9] and performance for some of the measures on Hospital Compare.[10, 11] Hospital accreditation is only loosely related to performance on these measures.

The New York City sanitation program regularly inspects restaurants and scores them on a standard set of indicators that correspond to critical violations (eg, food is contaminated by mouse droppings) or general violations (eg, garbage is not adequately covered).[12] Points are assigned to each type and severity of violation, and the sum of the points are converted into a summary grade of A, B, or C. Restaurants can dispute the grades, receiving a grade pending designation until the dispute is adjudicated. After inspection, sanitation grades are immediately posted on restaurants' front door or window, providing current information that is clearly visible to consumers before entering. More detailed information on sanitation violations is also available on a website. If restaurants receive an A grade, they face no additional inspections for 1 year, but poorly graded restaurants may receive monthly inspections. Restaurants face fines from violations and are subject to closure from severe violations. Recently proposed changes would decrease fines and give restaurants greater opportunities to appeal grades, but leave the program otherwise intact.[13]

IMPLICATIONS FOR PUBLIC QUALITY REPORTING IN HOSPITALS

Along with value‐based payment reforms, public quality reporting is one of the few major system‐level approaches that is being implemented in the US to improve quality and safety in healthcare. However, without a simple and understandable display of information that is available when a patient needs it, quality and safety information will likely go unused.[14] Hospital Compare leaves it up the patient to find the quality and safety information and does little to help patients understand and use the information effectively. Hospital Compare asks patients to do far more work, which is perhaps why it has been largely ignored by patients.[2, 15] The New York City sanitation inspection program evaluates restaurants, prominently displays an understandable summary result, and puts the scoring details in the background. Although peer‐reviewed evaluations of the New York City sanitation inspection program have not yet been published, internal data show that the program has decreased customer concern about getting sick, improved sanitary practices, and decreased salmonella.[16] Evidence from a similar program in Los Angeles County found that hygiene grades steered consumers toward restaurants with better sanitary conditions and decreased food‐borne illness.[17]

The nature of choice in healthcare, particularly the choice of hospital, is much different than it is for restaurants. In some areas, a single hospital may serve a large geographical area, severely limiting choice. Even when patients have the ability to receive care at different hospitals, choice may be limited because patients are referred to a specific hospital by their outpatient physician or are brought to a hospital during an emergency.[18] In these cases, quality grades on the front doors of hospitals would not affect patient decisions, at least for that admission. Nonetheless, if quality grades were posted on the front doors of hospitals, patients receiving both inpatient and outpatient care would see the grades, and could use the information to make future decisions. Posted grades may also lead patients to review more in‐depth quality information related to their condition on the Hospital Compare website. Posted quality grades would also increase the visibility of the grades for other stakeholdersincluding the media and boards of directorsmagnifying their salience and impact.

How quality information is displayed and summarized can make or break public reporting programs. The New York City sanitation inspection program displays summarized, composite measures in the form of widely understood letter grades. Hospital Compare, however, displays myriad, unrelated performance measures that are not summarized into a global quality or safety measure. This information display is at odds with best practice. Patients find it difficult to synthesize data from multiple performance indicators to determine the relative quality of healthcare providers or insurance plans.⁷ In many cases, more information can lead to worse decision making.[19] Patients' difficulty making optimal choices has been noted in numerous healthcare settings, including purchasing Medicare Part D plans[20] and choosing health plans.[21] Recent evidence suggests that Nursing Home Compare's shift from an unsummarized collection of disparate performance measures to a 5‐star rating system has led patients to choose higher‐ranked facilities.[22] The fact that commercial providers of product quality information, such as Consumer Reports[23] and US News and World Report,[24] publish global summary scores, in addition to component scores, is a hint that this style of reporting is more appealing to consumers. Reports suggest that Medicare is moving toward a 5‐star quality rating system for hospitals,[8] which is a welcome development.

Different types of patients may demand different types of quality information, and a single summary measure for Hospital Compare may not meet the needs of a diverse set of patients. Nonetheless, the benefits from an actionable, understandable, comprehensive, and appropriate summary measure likely outweigh the costs of a potential mismatch for certain types of patients. Many of the performance measures on Hospital Compare already apply broadly to diverse sets of patients (eg, the structure measures, patient experience, and surgical safety) and are not specific to certain disease areas. Global summary measures could be complemented by separate component scores (eg, by disease area or domain of quality) for patients who wanted information on different aspects of care.

The inspection regime that underlies the New York City sanitary inspection program has parallels in healthcare that could be extended to Hospital Compare. For instance, the Joint Commission performs surprise inspections of hospitals as part of its accreditation process. The publicly reported 5‐star ratings for nursing homes are also based, in part, on inspection results.[25] Results from these types of inspections can capture up‐to‐date information on important dimensions of quality and safety that are not available in standard administrative data sources. Incorporating inspection results into Hospital Compare could increase both the timeliness and validity of the reporting.

The New York City sanitation inspection program is not a panacea: the indicators may not capture all relevant aspects of restaurant sanitation, some research suggests that past sanitary grades do not predict future grades,[26] and sanitary grade inflation over time has the potential to mask meaningful differences in sanitary conditions that are related to food‐borne illness.[16, 26] However, by providing understandable and meaningful reports at the point of service, the New York City program is well designed to encourage sanitation improvement through both consumer and supplier behavior.

Where the New York City sanitation inspection program succeeds, Hospital Compare fails. Hospital Compare is not patient centered, and it is not working for patients. Medicare can learn from the New York City restaurant sanitation inspection program to enhance the effects of public reporting by presenting information to consumers that is relevant, easy to access and interpret, and up to date. The greater complexity of hospital product lines should not deter these efforts. Patients' lives, not just the health of their gastrointestinal tracts, are at stake.

Few consumers would choose to dine at a restaurant if they knew the kitchen was infested with cockroaches. Few patients would choose to undergo a liver transplant in a hospital that was performing the procedure for the first time. In most sectors, consumers gather information about quality (and price) from the marketplace, where economic theory predicts that rational behavior and competition will lead to continuous improvement over time. However, for some goods and services, information is sparse and asymmetric between consumers and suppliers. In sectors where consumer health is at risk, society has often intervened to assure minimum standards. Yet sometimes these efforts have fallen short. In healthcare, physician licensure and hospital accreditation (eg, through the Joint Commission), although providing an important foundation to assure safety, have not come close to solving the widespread quality problems.[1] Basic regulatory requirements for restaurants have also proven inadequate to prevent food‐borne illness. Consumer trust, without information, can be a recipe (or prescription) for trouble.

In response, high‐profile efforts have been introduced to publicize the quality and safety of service providers. One example is Hospital Compare, Medicare's national quality reporting program for US hospitals.[2] The New York City sanitary grade inspection program is a parallel effort for restaurants. Although customers can judge how much they like the food from a restaurantor look up reviews at Yelp.comthey face greater difficulty identifying whether a restaurant was responsible for making them sick. By publicizing restaurants' sanitation conditions, the New York City inspection program seeks to use market forces to decrease food‐borne illness by deterring consumers from eating at restaurants with poor sanitation grades.

The aims of Hospital Compare and the New York City sanitary inspection program are fundamentally similar. Both initiatives seek to address a common market failure resulting in the consumer's lack of information on quality and safety. By infusing the market with information, these programs enable consumers to make better choices and encourage service providers to improve quality and safety.[3] Despite the promise of these programs, a copious literature about the effects of public quality reporting in healthcare has found mixed results.[4, 5] Although the performance measures in any public reporting program must be valid and reliable, good measures are not sufficient to achieve the goals of public reporting. To engage patients, reported results must also be accessible, understandable, and meaningful. Both patients' lack of knowledge about the reports[6] and patients' inability to effectively use these data to make better decisions[7] are some reasons why public quality reporting has fallen short of its expectations. This article argues that the New York City program is much better structured to positively affect patient choice, and holds important lessons for public quality reporting in US hospitals.

CONTRASTS BETWEEN HOSPITAL COMPARE AND THE NEW YORK CITY RESTAURANT SANITARY INSPECTION PROGRAM

Hospital Compare reports performance for 108 separate quality indicators related to quality and patient safety for US hospitals (Table 1). These are a combination of structure measures (eg, hospital participation in a systematic database for cardiac surgery), process of care measures (eg, acute myocardial infarction patients receiving fibrinolytic therapy within 30 minutes of hospital arrival), outcomes (eg, 30‐day mortality and readmission), and patient experience measures (eg, how you would rate your communication with your physician). Hospital Compare data, frequently based on hospital quality performance 1 to 3 years prior to publication, are displayed on a website. Hospitals do not receive a summary measure of quality or safety.[8] Hospitals face financial incentives that are tied to measure reporting[9] and performance for some of the measures on Hospital Compare.[10, 11] Hospital accreditation is only loosely related to performance on these measures.

The New York City sanitation program regularly inspects restaurants and scores them on a standard set of indicators that correspond to critical violations (eg, food is contaminated by mouse droppings) or general violations (eg, garbage is not adequately covered).[12] Points are assigned to each type and severity of violation, and the sum of the points are converted into a summary grade of A, B, or C. Restaurants can dispute the grades, receiving a grade pending designation until the dispute is adjudicated. After inspection, sanitation grades are immediately posted on restaurants' front door or window, providing current information that is clearly visible to consumers before entering. More detailed information on sanitation violations is also available on a website. If restaurants receive an A grade, they face no additional inspections for 1 year, but poorly graded restaurants may receive monthly inspections. Restaurants face fines from violations and are subject to closure from severe violations. Recently proposed changes would decrease fines and give restaurants greater opportunities to appeal grades, but leave the program otherwise intact.[13]

IMPLICATIONS FOR PUBLIC QUALITY REPORTING IN HOSPITALS

Along with value‐based payment reforms, public quality reporting is one of the few major system‐level approaches that is being implemented in the US to improve quality and safety in healthcare. However, without a simple and understandable display of information that is available when a patient needs it, quality and safety information will likely go unused.[14] Hospital Compare leaves it up the patient to find the quality and safety information and does little to help patients understand and use the information effectively. Hospital Compare asks patients to do far more work, which is perhaps why it has been largely ignored by patients.[2, 15] The New York City sanitation inspection program evaluates restaurants, prominently displays an understandable summary result, and puts the scoring details in the background. Although peer‐reviewed evaluations of the New York City sanitation inspection program have not yet been published, internal data show that the program has decreased customer concern about getting sick, improved sanitary practices, and decreased salmonella.[16] Evidence from a similar program in Los Angeles County found that hygiene grades steered consumers toward restaurants with better sanitary conditions and decreased food‐borne illness.[17]

The nature of choice in healthcare, particularly the choice of hospital, is much different than it is for restaurants. In some areas, a single hospital may serve a large geographical area, severely limiting choice. Even when patients have the ability to receive care at different hospitals, choice may be limited because patients are referred to a specific hospital by their outpatient physician or are brought to a hospital during an emergency.[18] In these cases, quality grades on the front doors of hospitals would not affect patient decisions, at least for that admission. Nonetheless, if quality grades were posted on the front doors of hospitals, patients receiving both inpatient and outpatient care would see the grades, and could use the information to make future decisions. Posted grades may also lead patients to review more in‐depth quality information related to their condition on the Hospital Compare website. Posted quality grades would also increase the visibility of the grades for other stakeholdersincluding the media and boards of directorsmagnifying their salience and impact.

How quality information is displayed and summarized can make or break public reporting programs. The New York City sanitation inspection program displays summarized, composite measures in the form of widely understood letter grades. Hospital Compare, however, displays myriad, unrelated performance measures that are not summarized into a global quality or safety measure. This information display is at odds with best practice. Patients find it difficult to synthesize data from multiple performance indicators to determine the relative quality of healthcare providers or insurance plans.⁷ In many cases, more information can lead to worse decision making.[19] Patients' difficulty making optimal choices has been noted in numerous healthcare settings, including purchasing Medicare Part D plans[20] and choosing health plans.[21] Recent evidence suggests that Nursing Home Compare's shift from an unsummarized collection of disparate performance measures to a 5‐star rating system has led patients to choose higher‐ranked facilities.[22] The fact that commercial providers of product quality information, such as Consumer Reports[23] and US News and World Report,[24] publish global summary scores, in addition to component scores, is a hint that this style of reporting is more appealing to consumers. Reports suggest that Medicare is moving toward a 5‐star quality rating system for hospitals,[8] which is a welcome development.

Different types of patients may demand different types of quality information, and a single summary measure for Hospital Compare may not meet the needs of a diverse set of patients. Nonetheless, the benefits from an actionable, understandable, comprehensive, and appropriate summary measure likely outweigh the costs of a potential mismatch for certain types of patients. Many of the performance measures on Hospital Compare already apply broadly to diverse sets of patients (eg, the structure measures, patient experience, and surgical safety) and are not specific to certain disease areas. Global summary measures could be complemented by separate component scores (eg, by disease area or domain of quality) for patients who wanted information on different aspects of care.

The inspection regime that underlies the New York City sanitary inspection program has parallels in healthcare that could be extended to Hospital Compare. For instance, the Joint Commission performs surprise inspections of hospitals as part of its accreditation process. The publicly reported 5‐star ratings for nursing homes are also based, in part, on inspection results.[25] Results from these types of inspections can capture up‐to‐date information on important dimensions of quality and safety that are not available in standard administrative data sources. Incorporating inspection results into Hospital Compare could increase both the timeliness and validity of the reporting.

The New York City sanitation inspection program is not a panacea: the indicators may not capture all relevant aspects of restaurant sanitation, some research suggests that past sanitary grades do not predict future grades,[26] and sanitary grade inflation over time has the potential to mask meaningful differences in sanitary conditions that are related to food‐borne illness.[16, 26] However, by providing understandable and meaningful reports at the point of service, the New York City program is well designed to encourage sanitation improvement through both consumer and supplier behavior.

Where the New York City sanitation inspection program succeeds, Hospital Compare fails. Hospital Compare is not patient centered, and it is not working for patients. Medicare can learn from the New York City restaurant sanitation inspection program to enhance the effects of public reporting by presenting information to consumers that is relevant, easy to access and interpret, and up to date. The greater complexity of hospital product lines should not deter these efforts. Patients' lives, not just the health of their gastrointestinal tracts, are at stake.

Few consumers would choose to dine at a restaurant if they knew the kitchen was infested with cockroaches. Few patients would choose to undergo a liver transplant in a hospital that was performing the procedure for the first time. In most sectors, consumers gather information about quality (and price) from the marketplace, where economic theory predicts that rational behavior and competition will lead to continuous improvement over time. However, for some goods and services, information is sparse and asymmetric between consumers and suppliers. In sectors where consumer health is at risk, society has often intervened to assure minimum standards. Yet sometimes these efforts have fallen short. In healthcare, physician licensure and hospital accreditation (eg, through the Joint Commission), although providing an important foundation to assure safety, have not come close to solving the widespread quality problems.[1] Basic regulatory requirements for restaurants have also proven inadequate to prevent food‐borne illness. Consumer trust, without information, can be a recipe (or prescription) for trouble.

In response, high‐profile efforts have been introduced to publicize the quality and safety of service providers. One example is Hospital Compare, Medicare's national quality reporting program for US hospitals.[2] The New York City sanitary grade inspection program is a parallel effort for restaurants. Although customers can judge how much they like the food from a restaurantor look up reviews at Yelp.comthey face greater difficulty identifying whether a restaurant was responsible for making them sick. By publicizing restaurants' sanitation conditions, the New York City inspection program seeks to use market forces to decrease food‐borne illness by deterring consumers from eating at restaurants with poor sanitation grades.

The aims of Hospital Compare and the New York City sanitary inspection program are fundamentally similar. Both initiatives seek to address a common market failure resulting in the consumer's lack of information on quality and safety. By infusing the market with information, these programs enable consumers to make better choices and encourage service providers to improve quality and safety.[3] Despite the promise of these programs, a copious literature about the effects of public quality reporting in healthcare has found mixed results.[4, 5] Although the performance measures in any public reporting program must be valid and reliable, good measures are not sufficient to achieve the goals of public reporting. To engage patients, reported results must also be accessible, understandable, and meaningful. Both patients' lack of knowledge about the reports[6] and patients' inability to effectively use these data to make better decisions[7] are some reasons why public quality reporting has fallen short of its expectations. This article argues that the New York City program is much better structured to positively affect patient choice, and holds important lessons for public quality reporting in US hospitals.

CONTRASTS BETWEEN HOSPITAL COMPARE AND THE NEW YORK CITY RESTAURANT SANITARY INSPECTION PROGRAM

Hospital Compare reports performance for 108 separate quality indicators related to quality and patient safety for US hospitals (Table 1). These are a combination of structure measures (eg, hospital participation in a systematic database for cardiac surgery), process of care measures (eg, acute myocardial infarction patients receiving fibrinolytic therapy within 30 minutes of hospital arrival), outcomes (eg, 30‐day mortality and readmission), and patient experience measures (eg, how you would rate your communication with your physician). Hospital Compare data, frequently based on hospital quality performance 1 to 3 years prior to publication, are displayed on a website. Hospitals do not receive a summary measure of quality or safety.[8] Hospitals face financial incentives that are tied to measure reporting[9] and performance for some of the measures on Hospital Compare.[10, 11] Hospital accreditation is only loosely related to performance on these measures.

The New York City sanitation program regularly inspects restaurants and scores them on a standard set of indicators that correspond to critical violations (eg, food is contaminated by mouse droppings) or general violations (eg, garbage is not adequately covered).[12] Points are assigned to each type and severity of violation, and the sum of the points are converted into a summary grade of A, B, or C. Restaurants can dispute the grades, receiving a grade pending designation until the dispute is adjudicated. After inspection, sanitation grades are immediately posted on restaurants' front door or window, providing current information that is clearly visible to consumers before entering. More detailed information on sanitation violations is also available on a website. If restaurants receive an A grade, they face no additional inspections for 1 year, but poorly graded restaurants may receive monthly inspections. Restaurants face fines from violations and are subject to closure from severe violations. Recently proposed changes would decrease fines and give restaurants greater opportunities to appeal grades, but leave the program otherwise intact.[13]

IMPLICATIONS FOR PUBLIC QUALITY REPORTING IN HOSPITALS

Along with value‐based payment reforms, public quality reporting is one of the few major system‐level approaches that is being implemented in the US to improve quality and safety in healthcare. However, without a simple and understandable display of information that is available when a patient needs it, quality and safety information will likely go unused.[14] Hospital Compare leaves it up the patient to find the quality and safety information and does little to help patients understand and use the information effectively. Hospital Compare asks patients to do far more work, which is perhaps why it has been largely ignored by patients.[2, 15] The New York City sanitation inspection program evaluates restaurants, prominently displays an understandable summary result, and puts the scoring details in the background. Although peer‐reviewed evaluations of the New York City sanitation inspection program have not yet been published, internal data show that the program has decreased customer concern about getting sick, improved sanitary practices, and decreased salmonella.[16] Evidence from a similar program in Los Angeles County found that hygiene grades steered consumers toward restaurants with better sanitary conditions and decreased food‐borne illness.[17]

The nature of choice in healthcare, particularly the choice of hospital, is much different than it is for restaurants. In some areas, a single hospital may serve a large geographical area, severely limiting choice. Even when patients have the ability to receive care at different hospitals, choice may be limited because patients are referred to a specific hospital by their outpatient physician or are brought to a hospital during an emergency.[18] In these cases, quality grades on the front doors of hospitals would not affect patient decisions, at least for that admission. Nonetheless, if quality grades were posted on the front doors of hospitals, patients receiving both inpatient and outpatient care would see the grades, and could use the information to make future decisions. Posted grades may also lead patients to review more in‐depth quality information related to their condition on the Hospital Compare website. Posted quality grades would also increase the visibility of the grades for other stakeholdersincluding the media and boards of directorsmagnifying their salience and impact.

How quality information is displayed and summarized can make or break public reporting programs. The New York City sanitation inspection program displays summarized, composite measures in the form of widely understood letter grades. Hospital Compare, however, displays myriad, unrelated performance measures that are not summarized into a global quality or safety measure. This information display is at odds with best practice. Patients find it difficult to synthesize data from multiple performance indicators to determine the relative quality of healthcare providers or insurance plans.⁷ In many cases, more information can lead to worse decision making.[19] Patients' difficulty making optimal choices has been noted in numerous healthcare settings, including purchasing Medicare Part D plans[20] and choosing health plans.[21] Recent evidence suggests that Nursing Home Compare's shift from an unsummarized collection of disparate performance measures to a 5‐star rating system has led patients to choose higher‐ranked facilities.[22] The fact that commercial providers of product quality information, such as Consumer Reports[23] and US News and World Report,[24] publish global summary scores, in addition to component scores, is a hint that this style of reporting is more appealing to consumers. Reports suggest that Medicare is moving toward a 5‐star quality rating system for hospitals,[8] which is a welcome development.

Different types of patients may demand different types of quality information, and a single summary measure for Hospital Compare may not meet the needs of a diverse set of patients. Nonetheless, the benefits from an actionable, understandable, comprehensive, and appropriate summary measure likely outweigh the costs of a potential mismatch for certain types of patients. Many of the performance measures on Hospital Compare already apply broadly to diverse sets of patients (eg, the structure measures, patient experience, and surgical safety) and are not specific to certain disease areas. Global summary measures could be complemented by separate component scores (eg, by disease area or domain of quality) for patients who wanted information on different aspects of care.

The inspection regime that underlies the New York City sanitary inspection program has parallels in healthcare that could be extended to Hospital Compare. For instance, the Joint Commission performs surprise inspections of hospitals as part of its accreditation process. The publicly reported 5‐star ratings for nursing homes are also based, in part, on inspection results.[25] Results from these types of inspections can capture up‐to‐date information on important dimensions of quality and safety that are not available in standard administrative data sources. Incorporating inspection results into Hospital Compare could increase both the timeliness and validity of the reporting.

The New York City sanitation inspection program is not a panacea: the indicators may not capture all relevant aspects of restaurant sanitation, some research suggests that past sanitary grades do not predict future grades,[26] and sanitary grade inflation over time has the potential to mask meaningful differences in sanitary conditions that are related to food‐borne illness.[16, 26] However, by providing understandable and meaningful reports at the point of service, the New York City program is well designed to encourage sanitation improvement through both consumer and supplier behavior.

Where the New York City sanitation inspection program succeeds, Hospital Compare fails. Hospital Compare is not patient centered, and it is not working for patients. Medicare can learn from the New York City restaurant sanitation inspection program to enhance the effects of public reporting by presenting information to consumers that is relevant, easy to access and interpret, and up to date. The greater complexity of hospital product lines should not deter these efforts. Patients' lives, not just the health of their gastrointestinal tracts, are at stake.

There are described advantages to documenting in an electronic health record (EHR).[1, 2, 3, 4, 5] There has been, however, an unanticipated decline in certain aspects of documentation quality after implementing EHRs,[6, 7, 8] for example, the overinclusion of data (note clutter) and inappropriate use of copy‐paste.[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]

The objectives of this pilot study were to examine the effectiveness of an intervention bundle designed to improve resident progress notes written in an EHR (Epic Systems Corp., Verona, WI) and to establish the reliability of an audit tool used to assess the notes. Prior to this intervention, we provided no formal education for our residents about documentation in the EHR and had no policy governing format or content. The institutional review board at the University of Wisconsin approved this study.

METHODS

RESULTS

The ICC was 0.96 (95% confidence interval: 0.91‐0.98), indicating an excellent level of inter‐rater reliability. There was a significant improvement in the total score (see Supporting Information, Appendix 5, in the online version of this article) between the preintervention (mean 9.72, standard deviation [SD] 1.52) and postintervention (mean 11.72, SD 1.62) periods (P<0.0001).

Table 1 shows the percentage of yes responses to each individual item in the pre‐ and postintervention periods. Our intervention had a significant impact on reducing vital sign clutter (4% preintervention, 84% postintervention, P<0.0001) and other visual clutter within the note (0% preintervention, 28% postintervention, P=0.0035). We did not observe a significant impact on the reduction of input/output or lab clutter. There was no significant difference observed in the inclusion of the medication list. No significant improvements were seen in questions related to copy‐paste. The intervention had no significant impact on areas with an already high baseline performance: newly written interval histories, newly written physical exams, newly written plans, and the inclusion of discrete diagnostic language for abnormal labs.

DISCUSSION

There are described advantages to documenting in an electronic health record (EHR).[1, 2, 3, 4, 5] There has been, however, an unanticipated decline in certain aspects of documentation quality after implementing EHRs,[6, 7, 8] for example, the overinclusion of data (note clutter) and inappropriate use of copy‐paste.[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]

The objectives of this pilot study were to examine the effectiveness of an intervention bundle designed to improve resident progress notes written in an EHR (Epic Systems Corp., Verona, WI) and to establish the reliability of an audit tool used to assess the notes. Prior to this intervention, we provided no formal education for our residents about documentation in the EHR and had no policy governing format or content. The institutional review board at the University of Wisconsin approved this study.

METHODS

RESULTS

The ICC was 0.96 (95% confidence interval: 0.91‐0.98), indicating an excellent level of inter‐rater reliability. There was a significant improvement in the total score (see Supporting Information, Appendix 5, in the online version of this article) between the preintervention (mean 9.72, standard deviation [SD] 1.52) and postintervention (mean 11.72, SD 1.62) periods (P<0.0001).

Table 1 shows the percentage of yes responses to each individual item in the pre‐ and postintervention periods. Our intervention had a significant impact on reducing vital sign clutter (4% preintervention, 84% postintervention, P<0.0001) and other visual clutter within the note (0% preintervention, 28% postintervention, P=0.0035). We did not observe a significant impact on the reduction of input/output or lab clutter. There was no significant difference observed in the inclusion of the medication list. No significant improvements were seen in questions related to copy‐paste. The intervention had no significant impact on areas with an already high baseline performance: newly written interval histories, newly written physical exams, newly written plans, and the inclusion of discrete diagnostic language for abnormal labs.

DISCUSSION

There are described advantages to documenting in an electronic health record (EHR).[1, 2, 3, 4, 5] There has been, however, an unanticipated decline in certain aspects of documentation quality after implementing EHRs,[6, 7, 8] for example, the overinclusion of data (note clutter) and inappropriate use of copy‐paste.[6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]

The objectives of this pilot study were to examine the effectiveness of an intervention bundle designed to improve resident progress notes written in an EHR (Epic Systems Corp., Verona, WI) and to establish the reliability of an audit tool used to assess the notes. Prior to this intervention, we provided no formal education for our residents about documentation in the EHR and had no policy governing format or content. The institutional review board at the University of Wisconsin approved this study.

METHODS

RESULTS

The ICC was 0.96 (95% confidence interval: 0.91‐0.98), indicating an excellent level of inter‐rater reliability. There was a significant improvement in the total score (see Supporting Information, Appendix 5, in the online version of this article) between the preintervention (mean 9.72, standard deviation [SD] 1.52) and postintervention (mean 11.72, SD 1.62) periods (P<0.0001).

Table 1 shows the percentage of yes responses to each individual item in the pre‐ and postintervention periods. Our intervention had a significant impact on reducing vital sign clutter (4% preintervention, 84% postintervention, P<0.0001) and other visual clutter within the note (0% preintervention, 28% postintervention, P=0.0035). We did not observe a significant impact on the reduction of input/output or lab clutter. There was no significant difference observed in the inclusion of the medication list. No significant improvements were seen in questions related to copy‐paste. The intervention had no significant impact on areas with an already high baseline performance: newly written interval histories, newly written physical exams, newly written plans, and the inclusion of discrete diagnostic language for abnormal labs.

DISCUSSION

Sepsis remains a significant healthcare burden and is the sixth most common reason for hospitalization in the United States. For patients presenting with severe sepsis, mortality rates are approximately 30%,[1, 2] and sepsis remains the most expensive reason for hospitalization. In 2009, septicemia accounted for nearly $15.4 billion in aggregate hospital costs.[2]

Early identification of sepsis and the timely implementation of goal‐directed therapy significantly decrease sepsis‐related mortality and are cost‐effective,[3, 4, 5] highlighting the need for new clinical strategies to aid in early diagnosis. To date, most studies have focused on the screening and management of sepsis in the emergency department and intensive care unit (ICU),[6, 7] and less is known about the benefits of screening in non‐ICU settings. In the non‐ICU setting, conditions may go unrecognized and treatments delayed. Evidence suggests that patients diagnosed with severe sepsis in the non‐ICU setting are almost twice as likely to die as those diagnosed in an emergency department.[8, 9]

Application of a sepsis screening tool to both medical and surgical patients poses an additional challenge that may impact the screen's performance. The specificity may be compromised by noninfectious causes of systemic inflammatory response syndrome (SIRS) commonly seen in the postsurgical patient. For example, the tachycardia and fever often seen in the postoperative patient are sufficient to qualify for SIRS, making the diagnosis of sepsis more challenging. The purpose of this study was to examine the performance of a nurse‐driven, simple sepsis screening tool in a mixed medical and surgical non‐ICU setting.

METHODS

Screening Tool

A sepsis screening tool was developed as part of a broader initiative to improve sepsis‐related morbidity and mortality at our hospital. The screening tool was adapted from the severe sepsis screening tool created by the Surviving Sepsis Campaign and Institute for Healthcare Improvement,[10] and consisted of a simple 3‐tiered paper‐based screening assessment that was to be completed by the bedside RN (Figure 1). RNs on the pilot medical/surgical intermediate care unit performed the screening assessment with their regular patient assessment at the beginning of each shift.

Figure 1

The first tier of the tool screened for the presence of SIRS. Positive parameters included heart rate >90, temperature >38C or <36C, white blood cell count >12,000 or<4000 or >10% bands, and/or respiratory rate >20 or partial pressure of carbon dioxide (PaCO₂) <32 mm Hg. To decrease the number of false‐positive screens in patients whose abnormal vitals could already be attributed to a condition other than sepsis, these symptoms were only scored if they had emerged within the previous 8 hours.

If patients met 2 SIRS criteria, the nurse would move to the second tier of the tool, which involved consideration of possible infection as a contributor to a patient's clinical condition as well as a source of infection. If infection was not suspected, further screening was terminated. If infection was suspected, the patient then met criteria for a positive sepsis screen, and a third tier of screening involving assessment of organ dysfunction was initiated.

If the patient screened positive for sepsis (2 SIRS and suspicion for new infection) or severe sepsis (sepsis with end‐organ dysfunction), nurses were instructed to document this in the patient's electronic medical record (EMR) and call the primary team to initiate actions following the hospital‐wide sepsis guidelines. Any subsequent actions were recorded in the patient's EMR.

RESULTS

Over a 1‐month time period, 2143 screens were completed on 245 patients (169 surgical, 76 medical). The overall incidence of sepsis on the treatment unit during this time period was 9%. Surgical patients had an 8.9% incidence of sepsis, and medical patients had an incidence of 9.2%.

Screening tool performance is presented in Table 1. The screening tool had 95.5% sensitivity and 91.9% specificity, with no significant differences in performance between surgical and medical patients. The overall negative predictive value was 99.5%, also with comparable performance in both surgical and medical patients (P = 0.89). The overall positive predictive value (PPV) was 70% in medical patients and 48% in surgical patients (P = 0.12). Screening tool accuracy for medical and surgical patients was 92%.

Clinical Activities

Of the 39 patients who screened positive for sepsis, nurses classified 20 with sepsis and 19 with severe sepsis. Of these 39 patients, 33 were included in our descriptive analysis of the effect of positive screening results on clinical activity (3 were excluded for admission for sepsis and 3 for missing data). As a comparison, we randomly selected 30 patients of the 206 patients who screened negative for sepsis to evaluate clinical activity before and after a negative screen.

Characteristics of patients screening positive and negative for sepsis are reported in Table 2. We found no statistically significant differences in age, sex, length of hospital stay, or mortality amongst all groups.

Table 2.Patient Characteristics of 33 Patients With a Positive Sepsis Screen and 30 Randomly Selected Patients With Negative Sepsis Screens

Patient Characteristics	Surgery (Positive)	Medicine (Positive)	Surgery (Negative)	Medicine (Negative)	P Value
NOTE: Abbreviations: IQR, interquartile range; N/A, not applicable; PODs, postoperative days.
No.	26	7	20	10
Age, y, mean	57.8 ( 16.5)	72.4 ( 16.8)	64.6 ( 19.4)	63.6 ( 16.8)	0.25
% Male (no.)	50% (13)	57% (4)	60% (12)	60% (6)	0.27
Length of stay, d, median (IQR)	9 (716.7)	7 (5.511.5)	11 (7.722)	8 (421)	0.38
No. of PODs until first positive screen, d, median (IQR)	2 (13)	N/A	N/A	N/A
% Mortality (no.)	0%	14% (1)	5% (1)	10% (1)	0.19

Figure 2 illustrates differences in the proportion of patients receiving a clinical action before and after a negative or positive screening test result. In the cohort of 33 patients screening positive for sepsis, clinical action after a positive screen was taken in 4 of the 7 (50%) medical patients and 11 of 26 (42%) surgical patients. In patients screening negative for sepsis we found only 1 incident in which a sepsis‐related action was taken after a negative screen. In this case the patient was admitted to the ICU within 8 hours of a negative screen, though there was no explicit documentation that sepsis was the reason for this admission.

Figure 2

We compared the proportion of patients receiving sepsis‐related treatment before either a negative or positive screen and found no significant difference (Table 3). We then compared the proportion of patients receiving sepsis‐related actions after a positive or negative screening test result and found that the proportion of patients receiving antibiotics, blood cultures, and lactate measurement was significantly higher for patients with a positive sepsis screening result compared to those with a negative screening result (Table 3).

Table 3.Comparison of the Proportion of Patients Receiving Sepsis‐Related Clinical Actions Before and After a Positive or Negative Screen

Intervention and Group	Proportion	P Value
NOTE: Abbreviations: ICU, intensive care unit.
Before screening test
Antibiotics		0.066
Positive screen	45%
Negative screen	23%
Lactate		0.837
Positive screen	15%
Negative screen	13%
Blood culture		0.181
Positive screen	18%
Negative screen	17%
Fluid administration		0.564
Positive screen	6%
Negative screen	10%
ICU transfer/consult		0.337
Positive screen	3%
Negative screen	0%
After screening test
Antibiotics		0.006
Positive screen	58%
Negative screen	23%
Lactate		0.018
Positive screen	36%
Negative screen	13%
Blood Culture		0.002
Positive screen	24%
Negative screen	17%
Fluid administration		0.112
Positive screen	24%
Negative screen	10%
ICU transfer/consult		0.175
Positive screen	9%
Negative screen	3%

DISCUSSION

Improving recognition and time to treatment of sepsis in a non‐ICU setting is an important step toward decreasing sepsis‐related mortality. Lundberg and colleagues found that mortality rates for patients diagnosed with septic shock on a general ward were higher than for patients diagnosed in the ICU, even though ward patients were younger and healthier at baseline.[8] For ward patients, treatment delays were most profound in initiating vasoactive therapies, and minor delays were encountered in initiating fluid resuscitation. In their international study on the impact of early goal‐directed therapy guidelines, Levy and colleagues found that patients diagnosed with severe sepsis on the wards were almost twice as likely to die as patients diagnosed with sepsis in the emergency department.[9]

We are the first to report about an accurate nurse‐driven SIRS‐based sepsis screening protocol that is effective in the early identification of sepsis in both medical and surgical patients in an intermediate care setting. We found no significant difference in the screening tool performance between the medical and surgical cohorts. This is an important comparison given that SIRS criteria alone can be nonspecific in the postoperative population, where it is common to have hemodynamic changes, elevation of inflammatory markers, and fevers from noninfectious sources.

Our sepsis screening tool was designed in 3 tiers to improve its specificity. The first tier was based strictly on SIRS criteria (eg, tachycardia or fever), whereas the second and third tiers served to increase the specificity of the screening tool by instructing the evaluator to assess possible sources of infection and assess for objective signs of organ dysfunction. We relied heavily on the nursing staff to assess for the presence or absence of infection and believe that the educational component prior to initiating the screening protocol was vital.

EMR‐based screening tools that rely purely on physiologic data have been considered for the early detection and management of sepsis, although they lack the specificity gained through the incorporation of clinical judgment.[13] Sawyer and colleagues report using a real‐time EMR‐based method for early sepsis detection in non‐ICU patients that is based solely on objective measures; however, their PPV was only 19.5%. The model we describe in this study is one that incorporates real‐time physiologic data available from an EMR coupled with the clinical judgment of a bedside registered nurse. As our data suggest, this provides a screen that is both sensitive and specific.

It is interesting to note that in our assessment of clinical action taken 8 hours after a positive screening test (the interval after which a new screening test was performed), the rate of diagnostic workup and/or treatment for sepsis was relatively low. One reason for this could have been that the treating team had suspicion for sepsis prior to a positive screen and had already initiated clinical action. Of the 51 recorded clinical actions taken around the time of a positive screen, the majority (67%) occurred before the screening result. It is also possible that clinical action was not pursued because the treatment team disagreed with a diagnosis of sepsis. Of all the false positive screening cases, manual chart review confirmed that these patients did not have sepsis, nor did they develop sepsis during their index hospital stay. Last, we only recorded clinical actions taken within 8 hours of the first positive screen for sepsis and measured 5 very specific actions. Thus, our analysis may have missed actions taken after 8 hours or actions that differed from the 5 we chose to assess.

Even with the apparently low levels of new clinical activity after a positive screen, when compared to patients who screened negative for sepsis, a significantly higher number of patients who had a positive screen received antibiotics, had lactate measured, and had blood cultures drawn. We did not find a significant difference in the proportion of patients receiving a sepsis‐related clinical action before a screening result (positive or negative), which suggests that a positive screening test may have led to increased clinical action.

A limitation of our study is its small size and that it was conducted in 1 pilot unit. Additionally, our retrospective analysis of clinical care inhibited our ability to fully understand a patient's clinical course or retrieve missing data points. A related limitation is that we could not ascertain how often the screening tool did not identify a case of sepsis before it was clinically diagnosed. Assessing the temporal performance of our screening tool is of great interest and may be more easily performed using an electronic version of the screening tool, which is currently in development.

Using ICD‐9 codes to determine the true‐negative cohort is another limitation of our study. It is well documented that use of administrative data can lead to inaccurate classification of patients.[14] To address this, we performed random audits of 30 test‐negative patients. In doing so we did not find any errors in classification.

Although we did not find a significant difference in screening tool performance between surgical and medical patients, the PPV of the tool was lower in the surgical population (48%) compared to the medical population (70%). The lower PPV observed in surgical patients could be attributable to an overall lower incidence of sepsis in this cohort as well as possible errors in initial assessment of infection, which can be difficult in postsurgical patients. Our retrospective analysis included data from the early months of the screening protocol, a time in which nursing staff was still developing clinical acumen in identifying sepsis. However, this could have led nurses to either overestimate or underestimate the presence of infection in either patient group.

Suspicion for infection is the cornerstone definition of sepsis, and in our screening protocol nurses were charged with making this decision based on their knowledge of the patient's clinical course and current status. Issues concerning nurses' recognition of infection symptoms are an area of opportunity for further research and education and could aid in improving PPV. Clinical judgment could be further bolstered by adding promising laboratory tests such as C‐reactive protein or procalcitonin as objective adjuncts to an initial assessment for sepsis,[15] which could potentially increase screening test PPV.

CONCLUSIONS

A simple screening tool for sepsis performed by the bedside nurse can provide a means to successfully identify sepsis early and lead to more timely diagnostics and treatment in both medical and surgical patients in an intermediate care setting.

Sepsis remains a significant healthcare burden and is the sixth most common reason for hospitalization in the United States. For patients presenting with severe sepsis, mortality rates are approximately 30%,[1, 2] and sepsis remains the most expensive reason for hospitalization. In 2009, septicemia accounted for nearly $15.4 billion in aggregate hospital costs.[2]

Early identification of sepsis and the timely implementation of goal‐directed therapy significantly decrease sepsis‐related mortality and are cost‐effective,[3, 4, 5] highlighting the need for new clinical strategies to aid in early diagnosis. To date, most studies have focused on the screening and management of sepsis in the emergency department and intensive care unit (ICU),[6, 7] and less is known about the benefits of screening in non‐ICU settings. In the non‐ICU setting, conditions may go unrecognized and treatments delayed. Evidence suggests that patients diagnosed with severe sepsis in the non‐ICU setting are almost twice as likely to die as those diagnosed in an emergency department.[8, 9]

Application of a sepsis screening tool to both medical and surgical patients poses an additional challenge that may impact the screen's performance. The specificity may be compromised by noninfectious causes of systemic inflammatory response syndrome (SIRS) commonly seen in the postsurgical patient. For example, the tachycardia and fever often seen in the postoperative patient are sufficient to qualify for SIRS, making the diagnosis of sepsis more challenging. The purpose of this study was to examine the performance of a nurse‐driven, simple sepsis screening tool in a mixed medical and surgical non‐ICU setting.

METHODS

Screening Tool

A sepsis screening tool was developed as part of a broader initiative to improve sepsis‐related morbidity and mortality at our hospital. The screening tool was adapted from the severe sepsis screening tool created by the Surviving Sepsis Campaign and Institute for Healthcare Improvement,[10] and consisted of a simple 3‐tiered paper‐based screening assessment that was to be completed by the bedside RN (Figure 1). RNs on the pilot medical/surgical intermediate care unit performed the screening assessment with their regular patient assessment at the beginning of each shift.

Figure 1

The first tier of the tool screened for the presence of SIRS. Positive parameters included heart rate >90, temperature >38C or <36C, white blood cell count >12,000 or<4000 or >10% bands, and/or respiratory rate >20 or partial pressure of carbon dioxide (PaCO₂) <32 mm Hg. To decrease the number of false‐positive screens in patients whose abnormal vitals could already be attributed to a condition other than sepsis, these symptoms were only scored if they had emerged within the previous 8 hours.

If patients met 2 SIRS criteria, the nurse would move to the second tier of the tool, which involved consideration of possible infection as a contributor to a patient's clinical condition as well as a source of infection. If infection was not suspected, further screening was terminated. If infection was suspected, the patient then met criteria for a positive sepsis screen, and a third tier of screening involving assessment of organ dysfunction was initiated.

If the patient screened positive for sepsis (2 SIRS and suspicion for new infection) or severe sepsis (sepsis with end‐organ dysfunction), nurses were instructed to document this in the patient's electronic medical record (EMR) and call the primary team to initiate actions following the hospital‐wide sepsis guidelines. Any subsequent actions were recorded in the patient's EMR.

RESULTS

Over a 1‐month time period, 2143 screens were completed on 245 patients (169 surgical, 76 medical). The overall incidence of sepsis on the treatment unit during this time period was 9%. Surgical patients had an 8.9% incidence of sepsis, and medical patients had an incidence of 9.2%.

Screening tool performance is presented in Table 1. The screening tool had 95.5% sensitivity and 91.9% specificity, with no significant differences in performance between surgical and medical patients. The overall negative predictive value was 99.5%, also with comparable performance in both surgical and medical patients (P = 0.89). The overall positive predictive value (PPV) was 70% in medical patients and 48% in surgical patients (P = 0.12). Screening tool accuracy for medical and surgical patients was 92%.

Clinical Activities

Of the 39 patients who screened positive for sepsis, nurses classified 20 with sepsis and 19 with severe sepsis. Of these 39 patients, 33 were included in our descriptive analysis of the effect of positive screening results on clinical activity (3 were excluded for admission for sepsis and 3 for missing data). As a comparison, we randomly selected 30 patients of the 206 patients who screened negative for sepsis to evaluate clinical activity before and after a negative screen.

Characteristics of patients screening positive and negative for sepsis are reported in Table 2. We found no statistically significant differences in age, sex, length of hospital stay, or mortality amongst all groups.

Table 2.Patient Characteristics of 33 Patients With a Positive Sepsis Screen and 30 Randomly Selected Patients With Negative Sepsis Screens

Patient Characteristics	Surgery (Positive)	Medicine (Positive)	Surgery (Negative)	Medicine (Negative)	P Value
NOTE: Abbreviations: IQR, interquartile range; N/A, not applicable; PODs, postoperative days.
No.	26	7	20	10
Age, y, mean	57.8 ( 16.5)	72.4 ( 16.8)	64.6 ( 19.4)	63.6 ( 16.8)	0.25
% Male (no.)	50% (13)	57% (4)	60% (12)	60% (6)	0.27
Length of stay, d, median (IQR)	9 (716.7)	7 (5.511.5)	11 (7.722)	8 (421)	0.38
No. of PODs until first positive screen, d, median (IQR)	2 (13)	N/A	N/A	N/A
% Mortality (no.)	0%	14% (1)	5% (1)	10% (1)	0.19

Figure 2 illustrates differences in the proportion of patients receiving a clinical action before and after a negative or positive screening test result. In the cohort of 33 patients screening positive for sepsis, clinical action after a positive screen was taken in 4 of the 7 (50%) medical patients and 11 of 26 (42%) surgical patients. In patients screening negative for sepsis we found only 1 incident in which a sepsis‐related action was taken after a negative screen. In this case the patient was admitted to the ICU within 8 hours of a negative screen, though there was no explicit documentation that sepsis was the reason for this admission.

Figure 2

We compared the proportion of patients receiving sepsis‐related treatment before either a negative or positive screen and found no significant difference (Table 3). We then compared the proportion of patients receiving sepsis‐related actions after a positive or negative screening test result and found that the proportion of patients receiving antibiotics, blood cultures, and lactate measurement was significantly higher for patients with a positive sepsis screening result compared to those with a negative screening result (Table 3).

Table 3.Comparison of the Proportion of Patients Receiving Sepsis‐Related Clinical Actions Before and After a Positive or Negative Screen

Intervention and Group	Proportion	P Value
NOTE: Abbreviations: ICU, intensive care unit.
Before screening test
Antibiotics		0.066
Positive screen	45%
Negative screen	23%
Lactate		0.837
Positive screen	15%
Negative screen	13%
Blood culture		0.181
Positive screen	18%
Negative screen	17%
Fluid administration		0.564
Positive screen	6%
Negative screen	10%
ICU transfer/consult		0.337
Positive screen	3%
Negative screen	0%
After screening test
Antibiotics		0.006
Positive screen	58%
Negative screen	23%
Lactate		0.018
Positive screen	36%
Negative screen	13%
Blood Culture		0.002
Positive screen	24%
Negative screen	17%
Fluid administration		0.112
Positive screen	24%
Negative screen	10%
ICU transfer/consult		0.175
Positive screen	9%
Negative screen	3%

DISCUSSION

Improving recognition and time to treatment of sepsis in a non‐ICU setting is an important step toward decreasing sepsis‐related mortality. Lundberg and colleagues found that mortality rates for patients diagnosed with septic shock on a general ward were higher than for patients diagnosed in the ICU, even though ward patients were younger and healthier at baseline.[8] For ward patients, treatment delays were most profound in initiating vasoactive therapies, and minor delays were encountered in initiating fluid resuscitation. In their international study on the impact of early goal‐directed therapy guidelines, Levy and colleagues found that patients diagnosed with severe sepsis on the wards were almost twice as likely to die as patients diagnosed with sepsis in the emergency department.[9]

We are the first to report about an accurate nurse‐driven SIRS‐based sepsis screening protocol that is effective in the early identification of sepsis in both medical and surgical patients in an intermediate care setting. We found no significant difference in the screening tool performance between the medical and surgical cohorts. This is an important comparison given that SIRS criteria alone can be nonspecific in the postoperative population, where it is common to have hemodynamic changes, elevation of inflammatory markers, and fevers from noninfectious sources.

Our sepsis screening tool was designed in 3 tiers to improve its specificity. The first tier was based strictly on SIRS criteria (eg, tachycardia or fever), whereas the second and third tiers served to increase the specificity of the screening tool by instructing the evaluator to assess possible sources of infection and assess for objective signs of organ dysfunction. We relied heavily on the nursing staff to assess for the presence or absence of infection and believe that the educational component prior to initiating the screening protocol was vital.

EMR‐based screening tools that rely purely on physiologic data have been considered for the early detection and management of sepsis, although they lack the specificity gained through the incorporation of clinical judgment.[13] Sawyer and colleagues report using a real‐time EMR‐based method for early sepsis detection in non‐ICU patients that is based solely on objective measures; however, their PPV was only 19.5%. The model we describe in this study is one that incorporates real‐time physiologic data available from an EMR coupled with the clinical judgment of a bedside registered nurse. As our data suggest, this provides a screen that is both sensitive and specific.

It is interesting to note that in our assessment of clinical action taken 8 hours after a positive screening test (the interval after which a new screening test was performed), the rate of diagnostic workup and/or treatment for sepsis was relatively low. One reason for this could have been that the treating team had suspicion for sepsis prior to a positive screen and had already initiated clinical action. Of the 51 recorded clinical actions taken around the time of a positive screen, the majority (67%) occurred before the screening result. It is also possible that clinical action was not pursued because the treatment team disagreed with a diagnosis of sepsis. Of all the false positive screening cases, manual chart review confirmed that these patients did not have sepsis, nor did they develop sepsis during their index hospital stay. Last, we only recorded clinical actions taken within 8 hours of the first positive screen for sepsis and measured 5 very specific actions. Thus, our analysis may have missed actions taken after 8 hours or actions that differed from the 5 we chose to assess.

Even with the apparently low levels of new clinical activity after a positive screen, when compared to patients who screened negative for sepsis, a significantly higher number of patients who had a positive screen received antibiotics, had lactate measured, and had blood cultures drawn. We did not find a significant difference in the proportion of patients receiving a sepsis‐related clinical action before a screening result (positive or negative), which suggests that a positive screening test may have led to increased clinical action.

A limitation of our study is its small size and that it was conducted in 1 pilot unit. Additionally, our retrospective analysis of clinical care inhibited our ability to fully understand a patient's clinical course or retrieve missing data points. A related limitation is that we could not ascertain how often the screening tool did not identify a case of sepsis before it was clinically diagnosed. Assessing the temporal performance of our screening tool is of great interest and may be more easily performed using an electronic version of the screening tool, which is currently in development.

Using ICD‐9 codes to determine the true‐negative cohort is another limitation of our study. It is well documented that use of administrative data can lead to inaccurate classification of patients.[14] To address this, we performed random audits of 30 test‐negative patients. In doing so we did not find any errors in classification.

Although we did not find a significant difference in screening tool performance between surgical and medical patients, the PPV of the tool was lower in the surgical population (48%) compared to the medical population (70%). The lower PPV observed in surgical patients could be attributable to an overall lower incidence of sepsis in this cohort as well as possible errors in initial assessment of infection, which can be difficult in postsurgical patients. Our retrospective analysis included data from the early months of the screening protocol, a time in which nursing staff was still developing clinical acumen in identifying sepsis. However, this could have led nurses to either overestimate or underestimate the presence of infection in either patient group.

Suspicion for infection is the cornerstone definition of sepsis, and in our screening protocol nurses were charged with making this decision based on their knowledge of the patient's clinical course and current status. Issues concerning nurses' recognition of infection symptoms are an area of opportunity for further research and education and could aid in improving PPV. Clinical judgment could be further bolstered by adding promising laboratory tests such as C‐reactive protein or procalcitonin as objective adjuncts to an initial assessment for sepsis,[15] which could potentially increase screening test PPV.

CONCLUSIONS

A simple screening tool for sepsis performed by the bedside nurse can provide a means to successfully identify sepsis early and lead to more timely diagnostics and treatment in both medical and surgical patients in an intermediate care setting.

Sepsis remains a significant healthcare burden and is the sixth most common reason for hospitalization in the United States. For patients presenting with severe sepsis, mortality rates are approximately 30%,[1, 2] and sepsis remains the most expensive reason for hospitalization. In 2009, septicemia accounted for nearly $15.4 billion in aggregate hospital costs.[2]

Early identification of sepsis and the timely implementation of goal‐directed therapy significantly decrease sepsis‐related mortality and are cost‐effective,[3, 4, 5] highlighting the need for new clinical strategies to aid in early diagnosis. To date, most studies have focused on the screening and management of sepsis in the emergency department and intensive care unit (ICU),[6, 7] and less is known about the benefits of screening in non‐ICU settings. In the non‐ICU setting, conditions may go unrecognized and treatments delayed. Evidence suggests that patients diagnosed with severe sepsis in the non‐ICU setting are almost twice as likely to die as those diagnosed in an emergency department.[8, 9]

Application of a sepsis screening tool to both medical and surgical patients poses an additional challenge that may impact the screen's performance. The specificity may be compromised by noninfectious causes of systemic inflammatory response syndrome (SIRS) commonly seen in the postsurgical patient. For example, the tachycardia and fever often seen in the postoperative patient are sufficient to qualify for SIRS, making the diagnosis of sepsis more challenging. The purpose of this study was to examine the performance of a nurse‐driven, simple sepsis screening tool in a mixed medical and surgical non‐ICU setting.

METHODS

Screening Tool

A sepsis screening tool was developed as part of a broader initiative to improve sepsis‐related morbidity and mortality at our hospital. The screening tool was adapted from the severe sepsis screening tool created by the Surviving Sepsis Campaign and Institute for Healthcare Improvement,[10] and consisted of a simple 3‐tiered paper‐based screening assessment that was to be completed by the bedside RN (Figure 1). RNs on the pilot medical/surgical intermediate care unit performed the screening assessment with their regular patient assessment at the beginning of each shift.

Figure 1

The first tier of the tool screened for the presence of SIRS. Positive parameters included heart rate >90, temperature >38C or <36C, white blood cell count >12,000 or<4000 or >10% bands, and/or respiratory rate >20 or partial pressure of carbon dioxide (PaCO₂) <32 mm Hg. To decrease the number of false‐positive screens in patients whose abnormal vitals could already be attributed to a condition other than sepsis, these symptoms were only scored if they had emerged within the previous 8 hours.

If patients met 2 SIRS criteria, the nurse would move to the second tier of the tool, which involved consideration of possible infection as a contributor to a patient's clinical condition as well as a source of infection. If infection was not suspected, further screening was terminated. If infection was suspected, the patient then met criteria for a positive sepsis screen, and a third tier of screening involving assessment of organ dysfunction was initiated.

If the patient screened positive for sepsis (2 SIRS and suspicion for new infection) or severe sepsis (sepsis with end‐organ dysfunction), nurses were instructed to document this in the patient's electronic medical record (EMR) and call the primary team to initiate actions following the hospital‐wide sepsis guidelines. Any subsequent actions were recorded in the patient's EMR.

RESULTS

Over a 1‐month time period, 2143 screens were completed on 245 patients (169 surgical, 76 medical). The overall incidence of sepsis on the treatment unit during this time period was 9%. Surgical patients had an 8.9% incidence of sepsis, and medical patients had an incidence of 9.2%.

Screening tool performance is presented in Table 1. The screening tool had 95.5% sensitivity and 91.9% specificity, with no significant differences in performance between surgical and medical patients. The overall negative predictive value was 99.5%, also with comparable performance in both surgical and medical patients (P = 0.89). The overall positive predictive value (PPV) was 70% in medical patients and 48% in surgical patients (P = 0.12). Screening tool accuracy for medical and surgical patients was 92%.

Clinical Activities

Of the 39 patients who screened positive for sepsis, nurses classified 20 with sepsis and 19 with severe sepsis. Of these 39 patients, 33 were included in our descriptive analysis of the effect of positive screening results on clinical activity (3 were excluded for admission for sepsis and 3 for missing data). As a comparison, we randomly selected 30 patients of the 206 patients who screened negative for sepsis to evaluate clinical activity before and after a negative screen.

Characteristics of patients screening positive and negative for sepsis are reported in Table 2. We found no statistically significant differences in age, sex, length of hospital stay, or mortality amongst all groups.

Table 2.Patient Characteristics of 33 Patients With a Positive Sepsis Screen and 30 Randomly Selected Patients With Negative Sepsis Screens

Patient Characteristics	Surgery (Positive)	Medicine (Positive)	Surgery (Negative)	Medicine (Negative)	P Value
NOTE: Abbreviations: IQR, interquartile range; N/A, not applicable; PODs, postoperative days.
No.	26	7	20	10
Age, y, mean	57.8 ( 16.5)	72.4 ( 16.8)	64.6 ( 19.4)	63.6 ( 16.8)	0.25
% Male (no.)	50% (13)	57% (4)	60% (12)	60% (6)	0.27
Length of stay, d, median (IQR)	9 (716.7)	7 (5.511.5)	11 (7.722)	8 (421)	0.38
No. of PODs until first positive screen, d, median (IQR)	2 (13)	N/A	N/A	N/A
% Mortality (no.)	0%	14% (1)	5% (1)	10% (1)	0.19

Figure 2 illustrates differences in the proportion of patients receiving a clinical action before and after a negative or positive screening test result. In the cohort of 33 patients screening positive for sepsis, clinical action after a positive screen was taken in 4 of the 7 (50%) medical patients and 11 of 26 (42%) surgical patients. In patients screening negative for sepsis we found only 1 incident in which a sepsis‐related action was taken after a negative screen. In this case the patient was admitted to the ICU within 8 hours of a negative screen, though there was no explicit documentation that sepsis was the reason for this admission.

Figure 2

We compared the proportion of patients receiving sepsis‐related treatment before either a negative or positive screen and found no significant difference (Table 3). We then compared the proportion of patients receiving sepsis‐related actions after a positive or negative screening test result and found that the proportion of patients receiving antibiotics, blood cultures, and lactate measurement was significantly higher for patients with a positive sepsis screening result compared to those with a negative screening result (Table 3).

Table 3.Comparison of the Proportion of Patients Receiving Sepsis‐Related Clinical Actions Before and After a Positive or Negative Screen

Intervention and Group	Proportion	P Value
NOTE: Abbreviations: ICU, intensive care unit.
Before screening test
Antibiotics		0.066
Positive screen	45%
Negative screen	23%
Lactate		0.837
Positive screen	15%
Negative screen	13%
Blood culture		0.181
Positive screen	18%
Negative screen	17%
Fluid administration		0.564
Positive screen	6%
Negative screen	10%
ICU transfer/consult		0.337
Positive screen	3%
Negative screen	0%
After screening test
Antibiotics		0.006
Positive screen	58%
Negative screen	23%
Lactate		0.018
Positive screen	36%
Negative screen	13%
Blood Culture		0.002
Positive screen	24%
Negative screen	17%
Fluid administration		0.112
Positive screen	24%
Negative screen	10%
ICU transfer/consult		0.175
Positive screen	9%
Negative screen	3%

DISCUSSION

Improving recognition and time to treatment of sepsis in a non‐ICU setting is an important step toward decreasing sepsis‐related mortality. Lundberg and colleagues found that mortality rates for patients diagnosed with septic shock on a general ward were higher than for patients diagnosed in the ICU, even though ward patients were younger and healthier at baseline.[8] For ward patients, treatment delays were most profound in initiating vasoactive therapies, and minor delays were encountered in initiating fluid resuscitation. In their international study on the impact of early goal‐directed therapy guidelines, Levy and colleagues found that patients diagnosed with severe sepsis on the wards were almost twice as likely to die as patients diagnosed with sepsis in the emergency department.[9]

We are the first to report about an accurate nurse‐driven SIRS‐based sepsis screening protocol that is effective in the early identification of sepsis in both medical and surgical patients in an intermediate care setting. We found no significant difference in the screening tool performance between the medical and surgical cohorts. This is an important comparison given that SIRS criteria alone can be nonspecific in the postoperative population, where it is common to have hemodynamic changes, elevation of inflammatory markers, and fevers from noninfectious sources.

Our sepsis screening tool was designed in 3 tiers to improve its specificity. The first tier was based strictly on SIRS criteria (eg, tachycardia or fever), whereas the second and third tiers served to increase the specificity of the screening tool by instructing the evaluator to assess possible sources of infection and assess for objective signs of organ dysfunction. We relied heavily on the nursing staff to assess for the presence or absence of infection and believe that the educational component prior to initiating the screening protocol was vital.

EMR‐based screening tools that rely purely on physiologic data have been considered for the early detection and management of sepsis, although they lack the specificity gained through the incorporation of clinical judgment.[13] Sawyer and colleagues report using a real‐time EMR‐based method for early sepsis detection in non‐ICU patients that is based solely on objective measures; however, their PPV was only 19.5%. The model we describe in this study is one that incorporates real‐time physiologic data available from an EMR coupled with the clinical judgment of a bedside registered nurse. As our data suggest, this provides a screen that is both sensitive and specific.

It is interesting to note that in our assessment of clinical action taken 8 hours after a positive screening test (the interval after which a new screening test was performed), the rate of diagnostic workup and/or treatment for sepsis was relatively low. One reason for this could have been that the treating team had suspicion for sepsis prior to a positive screen and had already initiated clinical action. Of the 51 recorded clinical actions taken around the time of a positive screen, the majority (67%) occurred before the screening result. It is also possible that clinical action was not pursued because the treatment team disagreed with a diagnosis of sepsis. Of all the false positive screening cases, manual chart review confirmed that these patients did not have sepsis, nor did they develop sepsis during their index hospital stay. Last, we only recorded clinical actions taken within 8 hours of the first positive screen for sepsis and measured 5 very specific actions. Thus, our analysis may have missed actions taken after 8 hours or actions that differed from the 5 we chose to assess.

Even with the apparently low levels of new clinical activity after a positive screen, when compared to patients who screened negative for sepsis, a significantly higher number of patients who had a positive screen received antibiotics, had lactate measured, and had blood cultures drawn. We did not find a significant difference in the proportion of patients receiving a sepsis‐related clinical action before a screening result (positive or negative), which suggests that a positive screening test may have led to increased clinical action.

A limitation of our study is its small size and that it was conducted in 1 pilot unit. Additionally, our retrospective analysis of clinical care inhibited our ability to fully understand a patient's clinical course or retrieve missing data points. A related limitation is that we could not ascertain how often the screening tool did not identify a case of sepsis before it was clinically diagnosed. Assessing the temporal performance of our screening tool is of great interest and may be more easily performed using an electronic version of the screening tool, which is currently in development.

Using ICD‐9 codes to determine the true‐negative cohort is another limitation of our study. It is well documented that use of administrative data can lead to inaccurate classification of patients.[14] To address this, we performed random audits of 30 test‐negative patients. In doing so we did not find any errors in classification.

Although we did not find a significant difference in screening tool performance between surgical and medical patients, the PPV of the tool was lower in the surgical population (48%) compared to the medical population (70%). The lower PPV observed in surgical patients could be attributable to an overall lower incidence of sepsis in this cohort as well as possible errors in initial assessment of infection, which can be difficult in postsurgical patients. Our retrospective analysis included data from the early months of the screening protocol, a time in which nursing staff was still developing clinical acumen in identifying sepsis. However, this could have led nurses to either overestimate or underestimate the presence of infection in either patient group.

Suspicion for infection is the cornerstone definition of sepsis, and in our screening protocol nurses were charged with making this decision based on their knowledge of the patient's clinical course and current status. Issues concerning nurses' recognition of infection symptoms are an area of opportunity for further research and education and could aid in improving PPV. Clinical judgment could be further bolstered by adding promising laboratory tests such as C‐reactive protein or procalcitonin as objective adjuncts to an initial assessment for sepsis,[15] which could potentially increase screening test PPV.

CONCLUSIONS

A simple screening tool for sepsis performed by the bedside nurse can provide a means to successfully identify sepsis early and lead to more timely diagnostics and treatment in both medical and surgical patients in an intermediate care setting.