User login
Derivation of a Clinical Model to Predict Unchanged Inpatient Echocardiograms
Transthoracic echocardiography (TTE) is one of the most commonly ordered diagnostic tests in healthcare. Studies of Medicare beneficiaries, for example, have shown that each year, approximately 20% undergo at least 1 TTE, including 4% who have 2 or more.1 TTE utilization rates increased dramatically in the 1990s and early 2000s. Between 1999 and 2008, for example, the rate of use of TTE per Medicare beneficiary nearly doubled.2 In 2014, echocardiography accounted for 10% of all Medicare spending for imaging services, or approximately $930 million.3 In response to concerns about the possible unnecessary use of TTE, the American Heart Association and American Society of Echocardiography developed Appropriate Use Criteria (AUC) in 2007 and 2011, which describe appropriate versus inappropriate indications for TTE.4 Subsequent studies have shown that rather than rooting out inappropriate studies, the vast majority of ordered studies appear to be appropriate according to the AUC criteria.5 The AUC criteria have also been criticized for being based on expert opinion rather than clinical evidence.6 Repeat TTE, defined as TTE done within 1 year of a prior TTE, represents 24% to 42% of all studies,7-9 and 31% of all Medicare beneficiaries who have a TTE get a repeat TTE within 1 year.10 In the present study, we reviewed all inpatient TTE performed over 1 year and described the group that have had a prior TTE within the past year (“repeat TTE”). We then derived a clinical prediction model to predict unchanged repeat TTE, with the goal of defining a subset of studies that are potentially unnecessary.
METHODS
The West Haven Connecticut Veteran’s Administration Hospital (WHVA), located outside New Haven, Connecticut, is a 228-bed tertiary care center affiliated with Yale University School of Medicine. Potential subjects were identified from review of the electronic medical records of all inpatients who had an inpatient echocardiogram between October 1, 2013, and September 30, 2014. Patient’s records were reviewed by using a standardized data extraction form for demographics, comorbidity, cardiovascular risk factors, service ordering the TTE, intensive care unit (ICU) location, prior TTE abnormalities, TTE indication, AUC category, time between TTEs, technical quality of TTE, electrocardiogram (ECG) abnormalities, history of intervening acute coronary syndrome, cardiac surgery, and revascularization. Candidate predictors included any variables suspected by the authors as being potentially associated with the primary outcome of changed repeat TTE. All patients who had an inpatient TTE and a prior TTE within the Veterans Affairs (VA) system within the past year were included in the study. Repeat studies from the same admission were only counted as 1 TTE and patients had to have had a prior TTE from a different admission or a prior outpatient TTE to be included. Patients who did not have a prior TTE within the past year or who had only a transesophageal echocardiogram or stress echocardiography were excluded. Suboptimal studies were included but noted as limited quality. The study was approved by the WHVA Institutional Review Board. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement was used in planning and reporting this study.11
TTEs were classified as normal, mildly abnormal, or with a major abnormality based on previously published definitions.12-14 Any abnormality was defined as any left ventricle (LV) dysfunction (left ventricular ejection fraction [LVEF] <55%), any aortic or mitral valve stenosis, any regional wall motion abnormality, any right ventricular dysfunction, any pulmonary hypertension, mild or greater valvular regurgitation, any diastolic dysfunction, moderate or greater pericardial effusion, any ventricular hypertrophy, or any other significant abnormality including thrombus, vegetation, or tamponade. Major abnormality was defined as moderate or greater LV dysfunction (LVEF <45%), moderate or greater valvular regurgitation, moderate or greater valvular stenosis (aortic or mitral valve area <1.5 cm²), any regional wall motion abnormality, right ventricular dysfunction, moderate or greater pulmonary hypertension, moderate or greater diastolic dysfunction, moderate or greater pericardial effusion, or any other major abnormality including thrombus, vegetation, tumor, or tamponade. Repeat TTEs were classified as changed or unchanged. Changed TTEs were divided into any new abnormality or improvement or a new major abnormality or improvement. Any new abnormality or improvement was defined as any new TTE abnormality that had not previously been described or in which there was a change of at least 1 severity grade from a prior TTE, including improvement by 1 grade. A new major TTE abnormality or improvement was defined as any new major TTE abnormality that had previously been normal, or if there had been a prior abnormality, a change in at least 1 severity grade for LVEF or 2 severity grades for abnormal valvular, pericardial, or prior pulmonary hypertension, including improvement by 2 severity grades. A change from mild to moderate mitral regurgitation therefore was classified as a nonmajor change, whereas a change from mild to severe was classified as major. All TTE classifications were based on the electronic TTE reports and were reviewed by 2 independent investigators (CG and JC) blinded to the patients’ other clinical characteristics. For TTE studies in which the investigators agreed, that determination was the final classification. Disagreements were reviewed and the final classification was determined by consensus.
In an analogous manner, ECGs were classified as normal, mildly abnormal, or with a major abnormality based on previous definitions in the literature.15 Major abnormality was defined as atrial fibrillation or flutter, high-degree atrioventricular blocks, left bundle-branch block, right bundle-branch block, indeterminate conduction delay, q-wave myocardial infarction, isolated ischemic abnormalities, left ventricular hypertrophy with ST-T abnormalities, other arrhythmias including supraventricular tachycardia (SVT) or ventricular tachycardia (VT), low voltage (peak-to-peak QRS amplitude of <5 mm in the limb leads and/or <10 mm in the precordial leads), paced rhythm, sinus tachycardia (heart rate [HR] >100) or bradycardia (HR <50). Mild ECG abnormality was defined as low-grade atrioventricular blocks, borderline prolonged ventricular excitation, prolonged ventricular repolarization, isolated minor Q and ST-T abnormalities, left ventricular hypertrophy without ST-T abnormalities, left atrial enlargement, atrial or ventricular premature beats, or fascicular blocks. New major ECG abnormalities were any of the listed major ECG abnormalities that were not present on ECGs prior to the admission during which the repeat TTE was performed.
Other study definitions included intervening acute myocardial infarction (AMI), which was defined by any intervening history of elevated troponins, regardless of symptoms or ECG changes and including demand ischemia. Chronic kidney disease (CKD) was defined as an abnormal serum creatinine on 2 or more occasions 3 months apart. Active cancer was defined as receiving chemotherapy or palliative care for advanced cancer. Valvular heart disease was defined as prior moderate or severe valvular stenosis or regurgitation.
For analysis, we first compared patients with repeat TTE with major changes with those without major changes. For comparison of dichotomous variables, χ2 or Fisher exact tests were used. For continuous variables, Student t test or the Mann-Whitney U test were performed. Because many of the frequencies of individual AUC criteria were small, related AUC criteria were grouped for analysis as grouped by the tables of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance (ACCF/ASE/AHA) Guideline.4 Criteria groupings that were significantly less likely to have major TTE changes on analysis were classified as low risk and criteria that were significantly more likely were classified as high risk. Criteria groupings that were not significantly associated with TTE change were classified as average risk. All variables with P values less than 0.05 on bivariate analysis were then entered into a multivariate logistic regression analysis with major TTE change as the dependent variable, using backward stepwise variable selection with entry and exit criteria of P < 0.05 and P > 0.10, respectively. Scores were derived by converting the regression coefficients of independently predictive variables in the logistic regression model into corresponding integers. A total score was calculated for each patient by summing up the points for each independently significant variable. Model performance was described by calculating a C statistic by creation of a receiver operating characteristic curve to assess discrimination, and by performing the Hosmer and Lemeshow test to assess calibration. Internal validation was assessed by calculating the C statistic using the statistical method of bootstrapping in which the data were resampled multiple times (n = 200) and the average resultant C statistic reported. The bootstrap analysis was performed using R version 3.1 (R Foundation for Statistical Computing, Vienna, Austria). All other analyses were performed using SPSS version 21.0 (IBM, Armonk, New York). P values <0.05 were considered significant.
RESULTS
During the 1-year study period, there were 3944 medical/surgical admissions for 3266 patients and 845 inpatient TTEs obtained on 601 patients. Of all patients who were admitted, 601/3266 (18.4%) had at least 1 inpatient TTE. Of these 601 TTEs, 211 (35%) had a TTE within the VA system during the prior year. Of the 211 repeat TTEs, 67 (32%) were unchanged, 66 (31%) had minor changes, and 78 (37%) had major changes. The kappa statistic for agreement between extractors for “major TTE change” was 0.91, P < 0.001. The 10 most common AUC indications for TTE, which accounted for 72% of all studies, are listed in Table 1. The initial AUCs assigned by reviewers were the same in 187 of 211 TTEs (kappa 0.86, P < 0.001). Most indications were not associated with TTE outcome, although studies ordered for AUC indications 1 and 2 were less likely be associated with major changes and AUC indications 22 and 47 were more likely to be associated with major changes. Table 2 shows the comparison of the 78 patients that had repeat TTE with major changes compared with the 133 patients that did not. Nine variables were significantly different between the 2 groups; repeat TTEs with major changes were more likely to have dementia, be ordered by the surgery service, be located in an ICU, have major new ECG changes, have had prior valvular heart disease, have had an intervening AMI or cardiac surgery, or be in a high-risk AUC category. Patients with CKD were less likely to have major changes. Table 3 shows the results of the multivariate analysis; CKD, intervening AMI, prior valvular heart disease, major new ECG changes, and intervening cardiac surgery all independently predicted major changes on repeat TTE. Based on the β-coefficient for each variable, a point system was assigned to each variable and a total score calculated for each patient. Most variables had β-coefficients close to 1 and were therefore assigned a score of 1. CKD was associated with a lower risk of major TTE abnormality and was assigned a negative score. Intervening AMI was associated with a β-coefficient of 2.2 and was assigned a score of 2. Based on the points assigned to each variable and its presence or absence for each patient, a total score, which we named the CAVES score, was calculated. The acronym CAVES stands for CKD, AMI, valvular disease, ECG changes, and surgery (cardiac). Table 4 shows the frequencies of each score for each patient, ranging from patients with CKD and no other risk factors who scored −1 to patients without CKD who had all 4 of the other variables who scored 5. The prevalence of major TTE change for the full cohort was 37%. For the group with a CAVES score of −1, the probability was only 5.6%; for the group with a score of 0, the probability was 17.7%; and for the group with a score ≥1, the probability was 55.3%.
The bootstrap corrected C statistic for the model was 0.78 (95% confidence interval, 0.72-0.85), indicating good discrimination. The Hosmer and Lemeshow test showed nonsignificance, indicating good calibration (χ2 = 5.20, df = 6, P = 0.52).
DISCUSSION
In this retrospective study, we found that approximately 18% of all patients admitted to the hospital had an inpatient TTE performed, and that approximately 35% of this group had a prior TTE within the past year. Of the group with prior TTEs within the past year, 37% had a major new change and 63% had either minor or no changes. Prior studies have reported similar high rates of repeat TTE7-9 and of major changes on repeat TTE.8,14,16 On multivariate analysis, we found that 5 variables were independent predictors of new changes on TTE—absence of CKD, intervening AMI, intervening cardiac surgery, history of valvular heart disease, and major new ECG changes. We developed and internally validated a risk score based on these 5 variables, which was found to have good overall accuracy as measured by the bootstrap corrected C statistic. The simplified version of the score divides patients into low, intermediate, and high risk for major changes on TTE. The low-risk group, defined as the group with no risk factors, had an approximately 6% risk of a major TTE change; the intermediate risk group, defined as a score of 0, had an 18% risk of major TTE change; and the high-risk group, defined as a score of 1 or greater, had a 55% chance of major TTE change. We believe that this risk score, if further validated, will potentially allow hospital-based clinicians to estimate the chance of a major change on TTE prior to ordering the study. For the low-risk group, this may indicate that the study is unnecessary. Conversely, for patients at high risk, this may offer further evidence that it will be useful to obtain a repeat TTE.
In summary, we have developed a simple score to predict the likelihood of major changes on repeat TTEs for hospitalized patients. The CAVES score identified 8.5% of patients as being low risk for changed repeat TTE, 37% at intermediate risk, and 54% at high risk for major changes. We believe that the CAVES score, if further validated, may be used to risk stratify patients for ordering TTE and to potentially avoid unnecessary repeat studies.
Disclosure
The authors indicated no conflicts of interest.
1. Virnig BA, Shippee SN, O’Donnell B, Zeglin J, Parashuram S. Data point 20: echocardiography trends. In Trends in the Use of Echocardiography, 2007 to 2011. Rockville, MD: Agency for Healthcare Research and Quality; 2014. p 1-21. PubMed
2. Andrus BW, Welch HG. Medicare services provided by cardiologists in the United States: 1999-2008. Circ Cardiovasc Qual Outcomes. 2012;5(1):31-36. PubMed
3. Report to the Congress: Medicare Payment Policy. 2016; 105. http://www.medpac.gov/docs/default-source/data-book/june-2016-data-book-section-7-ambulatory-care.pdf?sfvrsn=0. Accessed on August 14, 2017.
4. American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 Appropriate use criteria for echocardiography. A report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance Endorsed by the American College of Chest Physicians. J Am Coll Cardiol. 2011;57(9):1126-1166. PubMed
5. Matulevicius SA, Rohatgi A, Das SR, Price AL, DeLuna A, Reimold SC. Appropriate use and clinical impact of transthoracic echocardiography. JAMA Intern Med. 2013;173(17):1600-1607. PubMed
6. Ioannidis JP. Appropriate vs clinically useful diagnostic tests. JAMA Intern Med. 2013;173(17):1607-1609. PubMed
7. Ghatak A, Pullatt R, Vyse S, Silverman DI. Appropriateness criteria are an imprecise measure for repeat echocardiograms. Echocardiography. 2011;28(2):131-135. PubMed
8. Koshy TP, Rohatgi A, Das SR, et al. The association of abnormal findings on transthoracic echocardiography with 2011 Appropriate Use Criteria and clinical impact. Int J Cardiovasc Imaging. 2015;31(3):521-528. PubMed
9. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transesophageal echocardiography. J Am Soc Echocardiogr. 2012;25(11):1170-1175. PubMed
10. Welch HG, Hayes KJ, Frost C. Repeat testing among Medicare beneficiaries. Arch Intern Med. 2012;172(22):1745-1751. PubMed
11. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735-736. PubMed
12. Ward RP, Mansour IN, Lemieux N, Gera N, Mehta R, Lang RM. Prospective evaluation of the clinical application of the American College of Cardiology Foundation/American Society of Echocardiography Appropriateness Criteria for transthoracic echocardiography. JACC Cardiovasc Imaging. 2008;1(5):663-671. PubMed
13. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transthoracic echocardiography in various clinical settings. J Am Soc Echocardiogr. 2012;25(11):1162-1169. PubMed
14. Mansour IN, Razi RR, Bhave NM, Ward RP. Comparison of the updated 2011 appropriate use criteria for echocardiography to the original criteria for transthoracic, transesophageal, and stress echocardiography. J Am Soc Echocardiogr. 2012;25(11):1153-1161. PubMed
15. Denes P, Larson JC, Lloyd-Jones DM, Prineas RJ, Greenland P. Major and minor ECG abnormalities in asymptomatic women and risk of cardiovascular events and mortality. JAMA. 2007;297(9):978-985. PubMed
16. Kirkpatrick JN, Ky B, Rahmouni HW, et al. Application of appropriateness criteria in outpatient transthoracic echocardiography. J Am Soc Echocardiogr. 2009;22(1):53-59. PubMed
Transthoracic echocardiography (TTE) is one of the most commonly ordered diagnostic tests in healthcare. Studies of Medicare beneficiaries, for example, have shown that each year, approximately 20% undergo at least 1 TTE, including 4% who have 2 or more.1 TTE utilization rates increased dramatically in the 1990s and early 2000s. Between 1999 and 2008, for example, the rate of use of TTE per Medicare beneficiary nearly doubled.2 In 2014, echocardiography accounted for 10% of all Medicare spending for imaging services, or approximately $930 million.3 In response to concerns about the possible unnecessary use of TTE, the American Heart Association and American Society of Echocardiography developed Appropriate Use Criteria (AUC) in 2007 and 2011, which describe appropriate versus inappropriate indications for TTE.4 Subsequent studies have shown that rather than rooting out inappropriate studies, the vast majority of ordered studies appear to be appropriate according to the AUC criteria.5 The AUC criteria have also been criticized for being based on expert opinion rather than clinical evidence.6 Repeat TTE, defined as TTE done within 1 year of a prior TTE, represents 24% to 42% of all studies,7-9 and 31% of all Medicare beneficiaries who have a TTE get a repeat TTE within 1 year.10 In the present study, we reviewed all inpatient TTE performed over 1 year and described the group that have had a prior TTE within the past year (“repeat TTE”). We then derived a clinical prediction model to predict unchanged repeat TTE, with the goal of defining a subset of studies that are potentially unnecessary.
METHODS
The West Haven Connecticut Veteran’s Administration Hospital (WHVA), located outside New Haven, Connecticut, is a 228-bed tertiary care center affiliated with Yale University School of Medicine. Potential subjects were identified from review of the electronic medical records of all inpatients who had an inpatient echocardiogram between October 1, 2013, and September 30, 2014. Patient’s records were reviewed by using a standardized data extraction form for demographics, comorbidity, cardiovascular risk factors, service ordering the TTE, intensive care unit (ICU) location, prior TTE abnormalities, TTE indication, AUC category, time between TTEs, technical quality of TTE, electrocardiogram (ECG) abnormalities, history of intervening acute coronary syndrome, cardiac surgery, and revascularization. Candidate predictors included any variables suspected by the authors as being potentially associated with the primary outcome of changed repeat TTE. All patients who had an inpatient TTE and a prior TTE within the Veterans Affairs (VA) system within the past year were included in the study. Repeat studies from the same admission were only counted as 1 TTE and patients had to have had a prior TTE from a different admission or a prior outpatient TTE to be included. Patients who did not have a prior TTE within the past year or who had only a transesophageal echocardiogram or stress echocardiography were excluded. Suboptimal studies were included but noted as limited quality. The study was approved by the WHVA Institutional Review Board. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement was used in planning and reporting this study.11
TTEs were classified as normal, mildly abnormal, or with a major abnormality based on previously published definitions.12-14 Any abnormality was defined as any left ventricle (LV) dysfunction (left ventricular ejection fraction [LVEF] <55%), any aortic or mitral valve stenosis, any regional wall motion abnormality, any right ventricular dysfunction, any pulmonary hypertension, mild or greater valvular regurgitation, any diastolic dysfunction, moderate or greater pericardial effusion, any ventricular hypertrophy, or any other significant abnormality including thrombus, vegetation, or tamponade. Major abnormality was defined as moderate or greater LV dysfunction (LVEF <45%), moderate or greater valvular regurgitation, moderate or greater valvular stenosis (aortic or mitral valve area <1.5 cm²), any regional wall motion abnormality, right ventricular dysfunction, moderate or greater pulmonary hypertension, moderate or greater diastolic dysfunction, moderate or greater pericardial effusion, or any other major abnormality including thrombus, vegetation, tumor, or tamponade. Repeat TTEs were classified as changed or unchanged. Changed TTEs were divided into any new abnormality or improvement or a new major abnormality or improvement. Any new abnormality or improvement was defined as any new TTE abnormality that had not previously been described or in which there was a change of at least 1 severity grade from a prior TTE, including improvement by 1 grade. A new major TTE abnormality or improvement was defined as any new major TTE abnormality that had previously been normal, or if there had been a prior abnormality, a change in at least 1 severity grade for LVEF or 2 severity grades for abnormal valvular, pericardial, or prior pulmonary hypertension, including improvement by 2 severity grades. A change from mild to moderate mitral regurgitation therefore was classified as a nonmajor change, whereas a change from mild to severe was classified as major. All TTE classifications were based on the electronic TTE reports and were reviewed by 2 independent investigators (CG and JC) blinded to the patients’ other clinical characteristics. For TTE studies in which the investigators agreed, that determination was the final classification. Disagreements were reviewed and the final classification was determined by consensus.
In an analogous manner, ECGs were classified as normal, mildly abnormal, or with a major abnormality based on previous definitions in the literature.15 Major abnormality was defined as atrial fibrillation or flutter, high-degree atrioventricular blocks, left bundle-branch block, right bundle-branch block, indeterminate conduction delay, q-wave myocardial infarction, isolated ischemic abnormalities, left ventricular hypertrophy with ST-T abnormalities, other arrhythmias including supraventricular tachycardia (SVT) or ventricular tachycardia (VT), low voltage (peak-to-peak QRS amplitude of <5 mm in the limb leads and/or <10 mm in the precordial leads), paced rhythm, sinus tachycardia (heart rate [HR] >100) or bradycardia (HR <50). Mild ECG abnormality was defined as low-grade atrioventricular blocks, borderline prolonged ventricular excitation, prolonged ventricular repolarization, isolated minor Q and ST-T abnormalities, left ventricular hypertrophy without ST-T abnormalities, left atrial enlargement, atrial or ventricular premature beats, or fascicular blocks. New major ECG abnormalities were any of the listed major ECG abnormalities that were not present on ECGs prior to the admission during which the repeat TTE was performed.
Other study definitions included intervening acute myocardial infarction (AMI), which was defined by any intervening history of elevated troponins, regardless of symptoms or ECG changes and including demand ischemia. Chronic kidney disease (CKD) was defined as an abnormal serum creatinine on 2 or more occasions 3 months apart. Active cancer was defined as receiving chemotherapy or palliative care for advanced cancer. Valvular heart disease was defined as prior moderate or severe valvular stenosis or regurgitation.
For analysis, we first compared patients with repeat TTE with major changes with those without major changes. For comparison of dichotomous variables, χ2 or Fisher exact tests were used. For continuous variables, Student t test or the Mann-Whitney U test were performed. Because many of the frequencies of individual AUC criteria were small, related AUC criteria were grouped for analysis as grouped by the tables of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance (ACCF/ASE/AHA) Guideline.4 Criteria groupings that were significantly less likely to have major TTE changes on analysis were classified as low risk and criteria that were significantly more likely were classified as high risk. Criteria groupings that were not significantly associated with TTE change were classified as average risk. All variables with P values less than 0.05 on bivariate analysis were then entered into a multivariate logistic regression analysis with major TTE change as the dependent variable, using backward stepwise variable selection with entry and exit criteria of P < 0.05 and P > 0.10, respectively. Scores were derived by converting the regression coefficients of independently predictive variables in the logistic regression model into corresponding integers. A total score was calculated for each patient by summing up the points for each independently significant variable. Model performance was described by calculating a C statistic by creation of a receiver operating characteristic curve to assess discrimination, and by performing the Hosmer and Lemeshow test to assess calibration. Internal validation was assessed by calculating the C statistic using the statistical method of bootstrapping in which the data were resampled multiple times (n = 200) and the average resultant C statistic reported. The bootstrap analysis was performed using R version 3.1 (R Foundation for Statistical Computing, Vienna, Austria). All other analyses were performed using SPSS version 21.0 (IBM, Armonk, New York). P values <0.05 were considered significant.
RESULTS
During the 1-year study period, there were 3944 medical/surgical admissions for 3266 patients and 845 inpatient TTEs obtained on 601 patients. Of all patients who were admitted, 601/3266 (18.4%) had at least 1 inpatient TTE. Of these 601 TTEs, 211 (35%) had a TTE within the VA system during the prior year. Of the 211 repeat TTEs, 67 (32%) were unchanged, 66 (31%) had minor changes, and 78 (37%) had major changes. The kappa statistic for agreement between extractors for “major TTE change” was 0.91, P < 0.001. The 10 most common AUC indications for TTE, which accounted for 72% of all studies, are listed in Table 1. The initial AUCs assigned by reviewers were the same in 187 of 211 TTEs (kappa 0.86, P < 0.001). Most indications were not associated with TTE outcome, although studies ordered for AUC indications 1 and 2 were less likely be associated with major changes and AUC indications 22 and 47 were more likely to be associated with major changes. Table 2 shows the comparison of the 78 patients that had repeat TTE with major changes compared with the 133 patients that did not. Nine variables were significantly different between the 2 groups; repeat TTEs with major changes were more likely to have dementia, be ordered by the surgery service, be located in an ICU, have major new ECG changes, have had prior valvular heart disease, have had an intervening AMI or cardiac surgery, or be in a high-risk AUC category. Patients with CKD were less likely to have major changes. Table 3 shows the results of the multivariate analysis; CKD, intervening AMI, prior valvular heart disease, major new ECG changes, and intervening cardiac surgery all independently predicted major changes on repeat TTE. Based on the β-coefficient for each variable, a point system was assigned to each variable and a total score calculated for each patient. Most variables had β-coefficients close to 1 and were therefore assigned a score of 1. CKD was associated with a lower risk of major TTE abnormality and was assigned a negative score. Intervening AMI was associated with a β-coefficient of 2.2 and was assigned a score of 2. Based on the points assigned to each variable and its presence or absence for each patient, a total score, which we named the CAVES score, was calculated. The acronym CAVES stands for CKD, AMI, valvular disease, ECG changes, and surgery (cardiac). Table 4 shows the frequencies of each score for each patient, ranging from patients with CKD and no other risk factors who scored −1 to patients without CKD who had all 4 of the other variables who scored 5. The prevalence of major TTE change for the full cohort was 37%. For the group with a CAVES score of −1, the probability was only 5.6%; for the group with a score of 0, the probability was 17.7%; and for the group with a score ≥1, the probability was 55.3%.
The bootstrap corrected C statistic for the model was 0.78 (95% confidence interval, 0.72-0.85), indicating good discrimination. The Hosmer and Lemeshow test showed nonsignificance, indicating good calibration (χ2 = 5.20, df = 6, P = 0.52).
DISCUSSION
In this retrospective study, we found that approximately 18% of all patients admitted to the hospital had an inpatient TTE performed, and that approximately 35% of this group had a prior TTE within the past year. Of the group with prior TTEs within the past year, 37% had a major new change and 63% had either minor or no changes. Prior studies have reported similar high rates of repeat TTE7-9 and of major changes on repeat TTE.8,14,16 On multivariate analysis, we found that 5 variables were independent predictors of new changes on TTE—absence of CKD, intervening AMI, intervening cardiac surgery, history of valvular heart disease, and major new ECG changes. We developed and internally validated a risk score based on these 5 variables, which was found to have good overall accuracy as measured by the bootstrap corrected C statistic. The simplified version of the score divides patients into low, intermediate, and high risk for major changes on TTE. The low-risk group, defined as the group with no risk factors, had an approximately 6% risk of a major TTE change; the intermediate risk group, defined as a score of 0, had an 18% risk of major TTE change; and the high-risk group, defined as a score of 1 or greater, had a 55% chance of major TTE change. We believe that this risk score, if further validated, will potentially allow hospital-based clinicians to estimate the chance of a major change on TTE prior to ordering the study. For the low-risk group, this may indicate that the study is unnecessary. Conversely, for patients at high risk, this may offer further evidence that it will be useful to obtain a repeat TTE.
In summary, we have developed a simple score to predict the likelihood of major changes on repeat TTEs for hospitalized patients. The CAVES score identified 8.5% of patients as being low risk for changed repeat TTE, 37% at intermediate risk, and 54% at high risk for major changes. We believe that the CAVES score, if further validated, may be used to risk stratify patients for ordering TTE and to potentially avoid unnecessary repeat studies.
Disclosure
The authors indicated no conflicts of interest.
Transthoracic echocardiography (TTE) is one of the most commonly ordered diagnostic tests in healthcare. Studies of Medicare beneficiaries, for example, have shown that each year, approximately 20% undergo at least 1 TTE, including 4% who have 2 or more.1 TTE utilization rates increased dramatically in the 1990s and early 2000s. Between 1999 and 2008, for example, the rate of use of TTE per Medicare beneficiary nearly doubled.2 In 2014, echocardiography accounted for 10% of all Medicare spending for imaging services, or approximately $930 million.3 In response to concerns about the possible unnecessary use of TTE, the American Heart Association and American Society of Echocardiography developed Appropriate Use Criteria (AUC) in 2007 and 2011, which describe appropriate versus inappropriate indications for TTE.4 Subsequent studies have shown that rather than rooting out inappropriate studies, the vast majority of ordered studies appear to be appropriate according to the AUC criteria.5 The AUC criteria have also been criticized for being based on expert opinion rather than clinical evidence.6 Repeat TTE, defined as TTE done within 1 year of a prior TTE, represents 24% to 42% of all studies,7-9 and 31% of all Medicare beneficiaries who have a TTE get a repeat TTE within 1 year.10 In the present study, we reviewed all inpatient TTE performed over 1 year and described the group that have had a prior TTE within the past year (“repeat TTE”). We then derived a clinical prediction model to predict unchanged repeat TTE, with the goal of defining a subset of studies that are potentially unnecessary.
METHODS
The West Haven Connecticut Veteran’s Administration Hospital (WHVA), located outside New Haven, Connecticut, is a 228-bed tertiary care center affiliated with Yale University School of Medicine. Potential subjects were identified from review of the electronic medical records of all inpatients who had an inpatient echocardiogram between October 1, 2013, and September 30, 2014. Patient’s records were reviewed by using a standardized data extraction form for demographics, comorbidity, cardiovascular risk factors, service ordering the TTE, intensive care unit (ICU) location, prior TTE abnormalities, TTE indication, AUC category, time between TTEs, technical quality of TTE, electrocardiogram (ECG) abnormalities, history of intervening acute coronary syndrome, cardiac surgery, and revascularization. Candidate predictors included any variables suspected by the authors as being potentially associated with the primary outcome of changed repeat TTE. All patients who had an inpatient TTE and a prior TTE within the Veterans Affairs (VA) system within the past year were included in the study. Repeat studies from the same admission were only counted as 1 TTE and patients had to have had a prior TTE from a different admission or a prior outpatient TTE to be included. Patients who did not have a prior TTE within the past year or who had only a transesophageal echocardiogram or stress echocardiography were excluded. Suboptimal studies were included but noted as limited quality. The study was approved by the WHVA Institutional Review Board. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis statement was used in planning and reporting this study.11
TTEs were classified as normal, mildly abnormal, or with a major abnormality based on previously published definitions.12-14 Any abnormality was defined as any left ventricle (LV) dysfunction (left ventricular ejection fraction [LVEF] <55%), any aortic or mitral valve stenosis, any regional wall motion abnormality, any right ventricular dysfunction, any pulmonary hypertension, mild or greater valvular regurgitation, any diastolic dysfunction, moderate or greater pericardial effusion, any ventricular hypertrophy, or any other significant abnormality including thrombus, vegetation, or tamponade. Major abnormality was defined as moderate or greater LV dysfunction (LVEF <45%), moderate or greater valvular regurgitation, moderate or greater valvular stenosis (aortic or mitral valve area <1.5 cm²), any regional wall motion abnormality, right ventricular dysfunction, moderate or greater pulmonary hypertension, moderate or greater diastolic dysfunction, moderate or greater pericardial effusion, or any other major abnormality including thrombus, vegetation, tumor, or tamponade. Repeat TTEs were classified as changed or unchanged. Changed TTEs were divided into any new abnormality or improvement or a new major abnormality or improvement. Any new abnormality or improvement was defined as any new TTE abnormality that had not previously been described or in which there was a change of at least 1 severity grade from a prior TTE, including improvement by 1 grade. A new major TTE abnormality or improvement was defined as any new major TTE abnormality that had previously been normal, or if there had been a prior abnormality, a change in at least 1 severity grade for LVEF or 2 severity grades for abnormal valvular, pericardial, or prior pulmonary hypertension, including improvement by 2 severity grades. A change from mild to moderate mitral regurgitation therefore was classified as a nonmajor change, whereas a change from mild to severe was classified as major. All TTE classifications were based on the electronic TTE reports and were reviewed by 2 independent investigators (CG and JC) blinded to the patients’ other clinical characteristics. For TTE studies in which the investigators agreed, that determination was the final classification. Disagreements were reviewed and the final classification was determined by consensus.
In an analogous manner, ECGs were classified as normal, mildly abnormal, or with a major abnormality based on previous definitions in the literature.15 Major abnormality was defined as atrial fibrillation or flutter, high-degree atrioventricular blocks, left bundle-branch block, right bundle-branch block, indeterminate conduction delay, q-wave myocardial infarction, isolated ischemic abnormalities, left ventricular hypertrophy with ST-T abnormalities, other arrhythmias including supraventricular tachycardia (SVT) or ventricular tachycardia (VT), low voltage (peak-to-peak QRS amplitude of <5 mm in the limb leads and/or <10 mm in the precordial leads), paced rhythm, sinus tachycardia (heart rate [HR] >100) or bradycardia (HR <50). Mild ECG abnormality was defined as low-grade atrioventricular blocks, borderline prolonged ventricular excitation, prolonged ventricular repolarization, isolated minor Q and ST-T abnormalities, left ventricular hypertrophy without ST-T abnormalities, left atrial enlargement, atrial or ventricular premature beats, or fascicular blocks. New major ECG abnormalities were any of the listed major ECG abnormalities that were not present on ECGs prior to the admission during which the repeat TTE was performed.
Other study definitions included intervening acute myocardial infarction (AMI), which was defined by any intervening history of elevated troponins, regardless of symptoms or ECG changes and including demand ischemia. Chronic kidney disease (CKD) was defined as an abnormal serum creatinine on 2 or more occasions 3 months apart. Active cancer was defined as receiving chemotherapy or palliative care for advanced cancer. Valvular heart disease was defined as prior moderate or severe valvular stenosis or regurgitation.
For analysis, we first compared patients with repeat TTE with major changes with those without major changes. For comparison of dichotomous variables, χ2 or Fisher exact tests were used. For continuous variables, Student t test or the Mann-Whitney U test were performed. Because many of the frequencies of individual AUC criteria were small, related AUC criteria were grouped for analysis as grouped by the tables of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance (ACCF/ASE/AHA) Guideline.4 Criteria groupings that were significantly less likely to have major TTE changes on analysis were classified as low risk and criteria that were significantly more likely were classified as high risk. Criteria groupings that were not significantly associated with TTE change were classified as average risk. All variables with P values less than 0.05 on bivariate analysis were then entered into a multivariate logistic regression analysis with major TTE change as the dependent variable, using backward stepwise variable selection with entry and exit criteria of P < 0.05 and P > 0.10, respectively. Scores were derived by converting the regression coefficients of independently predictive variables in the logistic regression model into corresponding integers. A total score was calculated for each patient by summing up the points for each independently significant variable. Model performance was described by calculating a C statistic by creation of a receiver operating characteristic curve to assess discrimination, and by performing the Hosmer and Lemeshow test to assess calibration. Internal validation was assessed by calculating the C statistic using the statistical method of bootstrapping in which the data were resampled multiple times (n = 200) and the average resultant C statistic reported. The bootstrap analysis was performed using R version 3.1 (R Foundation for Statistical Computing, Vienna, Austria). All other analyses were performed using SPSS version 21.0 (IBM, Armonk, New York). P values <0.05 were considered significant.
RESULTS
During the 1-year study period, there were 3944 medical/surgical admissions for 3266 patients and 845 inpatient TTEs obtained on 601 patients. Of all patients who were admitted, 601/3266 (18.4%) had at least 1 inpatient TTE. Of these 601 TTEs, 211 (35%) had a TTE within the VA system during the prior year. Of the 211 repeat TTEs, 67 (32%) were unchanged, 66 (31%) had minor changes, and 78 (37%) had major changes. The kappa statistic for agreement between extractors for “major TTE change” was 0.91, P < 0.001. The 10 most common AUC indications for TTE, which accounted for 72% of all studies, are listed in Table 1. The initial AUCs assigned by reviewers were the same in 187 of 211 TTEs (kappa 0.86, P < 0.001). Most indications were not associated with TTE outcome, although studies ordered for AUC indications 1 and 2 were less likely be associated with major changes and AUC indications 22 and 47 were more likely to be associated with major changes. Table 2 shows the comparison of the 78 patients that had repeat TTE with major changes compared with the 133 patients that did not. Nine variables were significantly different between the 2 groups; repeat TTEs with major changes were more likely to have dementia, be ordered by the surgery service, be located in an ICU, have major new ECG changes, have had prior valvular heart disease, have had an intervening AMI or cardiac surgery, or be in a high-risk AUC category. Patients with CKD were less likely to have major changes. Table 3 shows the results of the multivariate analysis; CKD, intervening AMI, prior valvular heart disease, major new ECG changes, and intervening cardiac surgery all independently predicted major changes on repeat TTE. Based on the β-coefficient for each variable, a point system was assigned to each variable and a total score calculated for each patient. Most variables had β-coefficients close to 1 and were therefore assigned a score of 1. CKD was associated with a lower risk of major TTE abnormality and was assigned a negative score. Intervening AMI was associated with a β-coefficient of 2.2 and was assigned a score of 2. Based on the points assigned to each variable and its presence or absence for each patient, a total score, which we named the CAVES score, was calculated. The acronym CAVES stands for CKD, AMI, valvular disease, ECG changes, and surgery (cardiac). Table 4 shows the frequencies of each score for each patient, ranging from patients with CKD and no other risk factors who scored −1 to patients without CKD who had all 4 of the other variables who scored 5. The prevalence of major TTE change for the full cohort was 37%. For the group with a CAVES score of −1, the probability was only 5.6%; for the group with a score of 0, the probability was 17.7%; and for the group with a score ≥1, the probability was 55.3%.
The bootstrap corrected C statistic for the model was 0.78 (95% confidence interval, 0.72-0.85), indicating good discrimination. The Hosmer and Lemeshow test showed nonsignificance, indicating good calibration (χ2 = 5.20, df = 6, P = 0.52).
DISCUSSION
In this retrospective study, we found that approximately 18% of all patients admitted to the hospital had an inpatient TTE performed, and that approximately 35% of this group had a prior TTE within the past year. Of the group with prior TTEs within the past year, 37% had a major new change and 63% had either minor or no changes. Prior studies have reported similar high rates of repeat TTE7-9 and of major changes on repeat TTE.8,14,16 On multivariate analysis, we found that 5 variables were independent predictors of new changes on TTE—absence of CKD, intervening AMI, intervening cardiac surgery, history of valvular heart disease, and major new ECG changes. We developed and internally validated a risk score based on these 5 variables, which was found to have good overall accuracy as measured by the bootstrap corrected C statistic. The simplified version of the score divides patients into low, intermediate, and high risk for major changes on TTE. The low-risk group, defined as the group with no risk factors, had an approximately 6% risk of a major TTE change; the intermediate risk group, defined as a score of 0, had an 18% risk of major TTE change; and the high-risk group, defined as a score of 1 or greater, had a 55% chance of major TTE change. We believe that this risk score, if further validated, will potentially allow hospital-based clinicians to estimate the chance of a major change on TTE prior to ordering the study. For the low-risk group, this may indicate that the study is unnecessary. Conversely, for patients at high risk, this may offer further evidence that it will be useful to obtain a repeat TTE.
In summary, we have developed a simple score to predict the likelihood of major changes on repeat TTEs for hospitalized patients. The CAVES score identified 8.5% of patients as being low risk for changed repeat TTE, 37% at intermediate risk, and 54% at high risk for major changes. We believe that the CAVES score, if further validated, may be used to risk stratify patients for ordering TTE and to potentially avoid unnecessary repeat studies.
Disclosure
The authors indicated no conflicts of interest.
1. Virnig BA, Shippee SN, O’Donnell B, Zeglin J, Parashuram S. Data point 20: echocardiography trends. In Trends in the Use of Echocardiography, 2007 to 2011. Rockville, MD: Agency for Healthcare Research and Quality; 2014. p 1-21. PubMed
2. Andrus BW, Welch HG. Medicare services provided by cardiologists in the United States: 1999-2008. Circ Cardiovasc Qual Outcomes. 2012;5(1):31-36. PubMed
3. Report to the Congress: Medicare Payment Policy. 2016; 105. http://www.medpac.gov/docs/default-source/data-book/june-2016-data-book-section-7-ambulatory-care.pdf?sfvrsn=0. Accessed on August 14, 2017.
4. American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 Appropriate use criteria for echocardiography. A report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance Endorsed by the American College of Chest Physicians. J Am Coll Cardiol. 2011;57(9):1126-1166. PubMed
5. Matulevicius SA, Rohatgi A, Das SR, Price AL, DeLuna A, Reimold SC. Appropriate use and clinical impact of transthoracic echocardiography. JAMA Intern Med. 2013;173(17):1600-1607. PubMed
6. Ioannidis JP. Appropriate vs clinically useful diagnostic tests. JAMA Intern Med. 2013;173(17):1607-1609. PubMed
7. Ghatak A, Pullatt R, Vyse S, Silverman DI. Appropriateness criteria are an imprecise measure for repeat echocardiograms. Echocardiography. 2011;28(2):131-135. PubMed
8. Koshy TP, Rohatgi A, Das SR, et al. The association of abnormal findings on transthoracic echocardiography with 2011 Appropriate Use Criteria and clinical impact. Int J Cardiovasc Imaging. 2015;31(3):521-528. PubMed
9. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transesophageal echocardiography. J Am Soc Echocardiogr. 2012;25(11):1170-1175. PubMed
10. Welch HG, Hayes KJ, Frost C. Repeat testing among Medicare beneficiaries. Arch Intern Med. 2012;172(22):1745-1751. PubMed
11. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735-736. PubMed
12. Ward RP, Mansour IN, Lemieux N, Gera N, Mehta R, Lang RM. Prospective evaluation of the clinical application of the American College of Cardiology Foundation/American Society of Echocardiography Appropriateness Criteria for transthoracic echocardiography. JACC Cardiovasc Imaging. 2008;1(5):663-671. PubMed
13. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transthoracic echocardiography in various clinical settings. J Am Soc Echocardiogr. 2012;25(11):1162-1169. PubMed
14. Mansour IN, Razi RR, Bhave NM, Ward RP. Comparison of the updated 2011 appropriate use criteria for echocardiography to the original criteria for transthoracic, transesophageal, and stress echocardiography. J Am Soc Echocardiogr. 2012;25(11):1153-1161. PubMed
15. Denes P, Larson JC, Lloyd-Jones DM, Prineas RJ, Greenland P. Major and minor ECG abnormalities in asymptomatic women and risk of cardiovascular events and mortality. JAMA. 2007;297(9):978-985. PubMed
16. Kirkpatrick JN, Ky B, Rahmouni HW, et al. Application of appropriateness criteria in outpatient transthoracic echocardiography. J Am Soc Echocardiogr. 2009;22(1):53-59. PubMed
1. Virnig BA, Shippee SN, O’Donnell B, Zeglin J, Parashuram S. Data point 20: echocardiography trends. In Trends in the Use of Echocardiography, 2007 to 2011. Rockville, MD: Agency for Healthcare Research and Quality; 2014. p 1-21. PubMed
2. Andrus BW, Welch HG. Medicare services provided by cardiologists in the United States: 1999-2008. Circ Cardiovasc Qual Outcomes. 2012;5(1):31-36. PubMed
3. Report to the Congress: Medicare Payment Policy. 2016; 105. http://www.medpac.gov/docs/default-source/data-book/june-2016-data-book-section-7-ambulatory-care.pdf?sfvrsn=0. Accessed on August 14, 2017.
4. American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, et al. ACCF/ASE/AHA/ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 Appropriate use criteria for echocardiography. A report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, American Society of Echocardiography, American Heart Association, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, Society of Critical Care Medicine, Society of Cardiovascular Computed Tomography, and Society for Cardiovascular Magnetic Resonance Endorsed by the American College of Chest Physicians. J Am Coll Cardiol. 2011;57(9):1126-1166. PubMed
5. Matulevicius SA, Rohatgi A, Das SR, Price AL, DeLuna A, Reimold SC. Appropriate use and clinical impact of transthoracic echocardiography. JAMA Intern Med. 2013;173(17):1600-1607. PubMed
6. Ioannidis JP. Appropriate vs clinically useful diagnostic tests. JAMA Intern Med. 2013;173(17):1607-1609. PubMed
7. Ghatak A, Pullatt R, Vyse S, Silverman DI. Appropriateness criteria are an imprecise measure for repeat echocardiograms. Echocardiography. 2011;28(2):131-135. PubMed
8. Koshy TP, Rohatgi A, Das SR, et al. The association of abnormal findings on transthoracic echocardiography with 2011 Appropriate Use Criteria and clinical impact. Int J Cardiovasc Imaging. 2015;31(3):521-528. PubMed
9. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transesophageal echocardiography. J Am Soc Echocardiogr. 2012;25(11):1170-1175. PubMed
10. Welch HG, Hayes KJ, Frost C. Repeat testing among Medicare beneficiaries. Arch Intern Med. 2012;172(22):1745-1751. PubMed
11. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162(10):735-736. PubMed
12. Ward RP, Mansour IN, Lemieux N, Gera N, Mehta R, Lang RM. Prospective evaluation of the clinical application of the American College of Cardiology Foundation/American Society of Echocardiography Appropriateness Criteria for transthoracic echocardiography. JACC Cardiovasc Imaging. 2008;1(5):663-671. PubMed
13. Bhatia RS, Carne DM, Picard MH, Weiner RB. Comparison of the 2007 and 2011 appropriate use criteria for transthoracic echocardiography in various clinical settings. J Am Soc Echocardiogr. 2012;25(11):1162-1169. PubMed
14. Mansour IN, Razi RR, Bhave NM, Ward RP. Comparison of the updated 2011 appropriate use criteria for echocardiography to the original criteria for transthoracic, transesophageal, and stress echocardiography. J Am Soc Echocardiogr. 2012;25(11):1153-1161. PubMed
15. Denes P, Larson JC, Lloyd-Jones DM, Prineas RJ, Greenland P. Major and minor ECG abnormalities in asymptomatic women and risk of cardiovascular events and mortality. JAMA. 2007;297(9):978-985. PubMed
16. Kirkpatrick JN, Ky B, Rahmouni HW, et al. Application of appropriateness criteria in outpatient transthoracic echocardiography. J Am Soc Echocardiogr. 2009;22(1):53-59. PubMed
© 2018 Society of Hospital Medicine