User login
Prognosticating with the hospitalized-patient one-year mortality risk score using information abstracted from the medical record
A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.
Sixteen indices that aim to predict patient death risk have been described by Yourman et al.1 They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.
In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.2 This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.3 We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.
However, we do not know whether the HOMR score can be applied to patients using primary data (ie, those taken directly from the chart). This question is important for 2 reasons. First, if HOMR accurately predicts death risk using data abstracted from the medical record, it could be used in the clinical setting to assist in clinical decision-making. Second, HOMR uses multiple administrative datasets that are difficult to access and use by most clinical researchers; it is, therefore, important to determine if HOMR is accurate for clinical research based on primary medical record review. The primary objective of this study was to determine the accuracy of the HOMR score when calculated using data abstracted from clinical notes that were available when patients were admitted to hospital. Secondary objectives included determining whether functional measures abstracted were significantly associated with death risk beyond the HOMR score and whether HOMR scores calculated from chart review deviated from those calculated from administrative data.
METHODS
Study Cohort
The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.
Primary Data Collection
For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.
Patient functional status has been shown to be strongly associated with survival4 but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.5 as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”
Analysis
These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.3 We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.6 Next, we replicated methods from our studies2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.7 This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ2 distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).
RESULTS
There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.
The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).3
A total of 563 patients (11.3%) died within 1 year of admission to hospital (Table 2). In the study cohort, each chart-HOMR component was associated with death status. People who died were older, more likely to be male, had a greater number of important comorbidities, had more ED visits and admissions by ambulance in the previous year, and were more likely to have been discharged in the previous 30 days, and were admitted urgently, directly to the intensive care unit, or with complicated diagnoses. The mean chart-HOMR score differed extensively by survival status (37.4 [SD, 7.5] in those who died vs. 19.9 [SD, 12.2] in those who survived). Three of the socio-functional variables (delirium and falls noted on admission documents, and dependent for any ADL) also varied with death status.
The chart-HOMR score was strongly associated with the likelihood of death within 1 year of admission. When included in a logistic regression model having 1-year death as the outcome, a 1-point increase in the chart-HOMR score was associated with a 19% increase in the odds of death (P < 0.0001). This model (with only the chart-HOMR score) was highly discriminative (C statistic, 0.888) and well calibrated (Hosmer-Lemeshow test, 12.9 [8 df, P = 0.11]).
Observed and expected death risks by chart-HOMR score were similar (Figure 1). The observed total number of deaths (n = 563; 11.3%) exceeded the expected number of deaths (n = 437, 8.7%). When we regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups), the Hosmer-Lemeshow test was significant, indicating that differences between observed and expected risks were beyond that expected by chance (Hosmer-Lemeshow test, 141.9, 21 df, P < 0.0001). The intercept of this model (0.035; 95% CI, 0.01-0.06) was statistically significant (P = 0.01), indicating that the observed number of cases significantly exceeded the expected; however, its calibration slope (1.02; 95% CI, 0.89-1.16) did not deviate significantly from unity, indicating that the relationship between the observed and expected death risk (based on the chart-HOMR score) remained intact (Figure 1).
The deviations between observed and expected death risks reflected deviations between the c chart-HOMR score and the administrative-HOMR score, with the former being significantly lower than the latter (Figure 2). Overall, the chart-HOMR score was 0.96 points lower (95% CI, 0.81-1.12) than the administrative-HOMR score. The HOMR score components that were notably underestimated using chart data included those for the age-Charlson Comorbidity Index interaction, living status, and admit points. Points for only 2 components (admitting service and admission urgency) were higher when calculated using chart data.
Four additional socio-functional variables collected from medical record review were significantly associated with 1-year death risk independent of the chart-HOMR score (Table 3). Admission documentation noting either delirium or falls were both associated with a significantly increased death risk (adjusted odds ratio [OR], 1.92 [95% CI, 1.24-2.96] and OR 1.96 [95% CI, 1.29-2.99], respectively). An independently increased death risk was also noted in patients who were dependent for any ADL (adjusted OR, 1.99 [95% CI, 1.24-3.19]). The presence of an ED geriatrics consultation within the previous 6 months was associated with a significantly decreased death risk of 60% (adjusted OR, 0.40 [95% CI, 0.20-0.81]). Adding these covariates to the logistic model with the chart-HOMR score significantly improved predictions (likelihood ratio statistic = 33.569, 4df, P < 0.00001).
DISCUSSION
In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.8
We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care9; ambulation difficulties10; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire11; weight loss12; functional status4; and pressure sore risk.13) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.16 Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.
Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies1 ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.
In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.
Disclosure
The authors report no financial conflicts of interest.
1. Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA. 2012;307(2):182-192. PubMed
2. van Walraven C. The Hospital-patient One-year Mortality Risk score accurately predicts long term death risk in hospitalized patients. J Clin Epidemiol. 2014;67(9):1025-1034. PubMed
3. van Walraven C, McAlister FA, Bakal JA, Hawken S, Donzé J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission. CMAJ. 2015;187(10):725-733. PubMed
4. Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285(23):2987-2994. PubMed
5. Clegg A, Young J, Iliffe S et al. Frailty in elderly people. The Lancet 2002;381:752-762. PubMed
6. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016;25(4):1692-1706. PubMed
7. Harrell FE Jr. Overview of Maximum Likelihood Estimation. Regression Modeling Strategies. New York, NY: Springer-Verlag; 2001: 179-212.
8. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130(6):515-524. PubMed
9. Fischer SM, Gozansky WS, Sauaia A, Min SJ, Kutner JS, Kramer A. A practical tool to identify patients who may benefit from a palliative approach: the CARING criteria. J Pain Symptom Manage. 2006;31(4):285-292. PubMed
10. Inouye SK, Bogardus ST, Jr, Vitagliano G, et al. Burden of illness score for elderly persons: risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments. Med Care. 2003;41(1):70-83. PubMed
11. Pilotto A, Ferrucci L, Franceschi M, et al. Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res. 2008;11(1):151-161. PubMed
12. Teno JM, Harrell FE Jr, Knaus W, et al. Prediction of survival for older hospitalized patients: the HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc. 2000;48(5 suppl):S16-S24. PubMed
13. Dramé M, Novella JL, Lang PO, et al. Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: the SAFES study. Eur J Epidemiol. 2008;23(12):783-791. PubMed
14. Kieszak SM, Flanders WD, Kosinski AS, Shipp CC, Karp H. A comparison of the Charlson comorbidity index derived from medical record data and administrative billing data. J Clin Epidemiol. 1999;52(2):137-142. PubMed
15. Quan H, Parsons GA, Ghali WA. Validity of procedure codes in International Classification of Diseases, 9th revision, clinical modification administrative data. Med Care. 2004;42(8):801-809. PubMed
16. Zhang JX, Iwashyna TJ, Christakis NA. The performance of different lookback periods and sources of information for Charlson cComorbidity adjustment in Medicare claims. Med Care. 1999;37(11):1128-1139. PubMed
A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.
Sixteen indices that aim to predict patient death risk have been described by Yourman et al.1 They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.
In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.2 This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.3 We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.
However, we do not know whether the HOMR score can be applied to patients using primary data (ie, those taken directly from the chart). This question is important for 2 reasons. First, if HOMR accurately predicts death risk using data abstracted from the medical record, it could be used in the clinical setting to assist in clinical decision-making. Second, HOMR uses multiple administrative datasets that are difficult to access and use by most clinical researchers; it is, therefore, important to determine if HOMR is accurate for clinical research based on primary medical record review. The primary objective of this study was to determine the accuracy of the HOMR score when calculated using data abstracted from clinical notes that were available when patients were admitted to hospital. Secondary objectives included determining whether functional measures abstracted were significantly associated with death risk beyond the HOMR score and whether HOMR scores calculated from chart review deviated from those calculated from administrative data.
METHODS
Study Cohort
The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.
Primary Data Collection
For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.
Patient functional status has been shown to be strongly associated with survival4 but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.5 as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”
Analysis
These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.3 We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.6 Next, we replicated methods from our studies2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.7 This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ2 distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).
RESULTS
There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.
The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).3
A total of 563 patients (11.3%) died within 1 year of admission to hospital (Table 2). In the study cohort, each chart-HOMR component was associated with death status. People who died were older, more likely to be male, had a greater number of important comorbidities, had more ED visits and admissions by ambulance in the previous year, and were more likely to have been discharged in the previous 30 days, and were admitted urgently, directly to the intensive care unit, or with complicated diagnoses. The mean chart-HOMR score differed extensively by survival status (37.4 [SD, 7.5] in those who died vs. 19.9 [SD, 12.2] in those who survived). Three of the socio-functional variables (delirium and falls noted on admission documents, and dependent for any ADL) also varied with death status.
The chart-HOMR score was strongly associated with the likelihood of death within 1 year of admission. When included in a logistic regression model having 1-year death as the outcome, a 1-point increase in the chart-HOMR score was associated with a 19% increase in the odds of death (P < 0.0001). This model (with only the chart-HOMR score) was highly discriminative (C statistic, 0.888) and well calibrated (Hosmer-Lemeshow test, 12.9 [8 df, P = 0.11]).
Observed and expected death risks by chart-HOMR score were similar (Figure 1). The observed total number of deaths (n = 563; 11.3%) exceeded the expected number of deaths (n = 437, 8.7%). When we regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups), the Hosmer-Lemeshow test was significant, indicating that differences between observed and expected risks were beyond that expected by chance (Hosmer-Lemeshow test, 141.9, 21 df, P < 0.0001). The intercept of this model (0.035; 95% CI, 0.01-0.06) was statistically significant (P = 0.01), indicating that the observed number of cases significantly exceeded the expected; however, its calibration slope (1.02; 95% CI, 0.89-1.16) did not deviate significantly from unity, indicating that the relationship between the observed and expected death risk (based on the chart-HOMR score) remained intact (Figure 1).
The deviations between observed and expected death risks reflected deviations between the c chart-HOMR score and the administrative-HOMR score, with the former being significantly lower than the latter (Figure 2). Overall, the chart-HOMR score was 0.96 points lower (95% CI, 0.81-1.12) than the administrative-HOMR score. The HOMR score components that were notably underestimated using chart data included those for the age-Charlson Comorbidity Index interaction, living status, and admit points. Points for only 2 components (admitting service and admission urgency) were higher when calculated using chart data.
Four additional socio-functional variables collected from medical record review were significantly associated with 1-year death risk independent of the chart-HOMR score (Table 3). Admission documentation noting either delirium or falls were both associated with a significantly increased death risk (adjusted odds ratio [OR], 1.92 [95% CI, 1.24-2.96] and OR 1.96 [95% CI, 1.29-2.99], respectively). An independently increased death risk was also noted in patients who were dependent for any ADL (adjusted OR, 1.99 [95% CI, 1.24-3.19]). The presence of an ED geriatrics consultation within the previous 6 months was associated with a significantly decreased death risk of 60% (adjusted OR, 0.40 [95% CI, 0.20-0.81]). Adding these covariates to the logistic model with the chart-HOMR score significantly improved predictions (likelihood ratio statistic = 33.569, 4df, P < 0.00001).
DISCUSSION
In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.8
We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care9; ambulation difficulties10; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire11; weight loss12; functional status4; and pressure sore risk.13) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.16 Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.
Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies1 ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.
In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.
Disclosure
The authors report no financial conflicts of interest.
A patient’s prognosis can strongly influence their medical care. Decisions about diagnostic modalities, treatment options, and the use of preventive therapies can all be affected by the likelihood of a patient’s death in the near future. For example, patients with severely limited survival might forego prophylactic therapy, avoid interventions for asymptomatic issues, and cease screening interventions. Knowing survival probability would also be very helpful as a controlling variable in research analyses whenever death risk might be a possible confounder.
Sixteen indices that aim to predict patient death risk have been described by Yourman et al.1 They were all created from secondary analyses of clinical and administrative datasets, were applicable to patients in a variety of settings (including the community, nursing home, or hospital), and predicted survival probabilities in time horizons ranging from 6 months to 5 years. Prognostic factors that were most commonly included in these indices were comorbidity and functional status. In validation populations, the discrimination of these indices for 1-year survival in hospitalized patients was moderate (with C statistics that ranged from 0.64 to 0.79) with good calibration for broad prognostic ranges.
In 2014, we published the Hospitalized-patient One-year Mortality Risk (HOMR) score.2 This study used health administrative data for all adult Ontarians admitted in 2011 to hospital under nonpsychiatric services (n = 640,022) to estimate the probability of dying within 1 year of admission to hospital (which happened in 11.7% of people). The HOMR score included 12 patient and hospitalization factors (Table 1). It was highly discriminative (C statistic, 0.923; [0.922-0.924]) and well calibrated (the mean relative difference between observed and expected death risk was 2.0% [range, 0.0% to 7.0%]). It was externally validated in more than 3 million adults from Ontario, Alberta, and Boston in whom the C statistic ranged from 0.89 to 0.92 and calibration was excellent.3 We concluded from these studies that the HOMR score is excellent for prognosticating a diverse group of patients using health administrative data.
However, we do not know whether the HOMR score can be applied to patients using primary data (ie, those taken directly from the chart). This question is important for 2 reasons. First, if HOMR accurately predicts death risk using data abstracted from the medical record, it could be used in the clinical setting to assist in clinical decision-making. Second, HOMR uses multiple administrative datasets that are difficult to access and use by most clinical researchers; it is, therefore, important to determine if HOMR is accurate for clinical research based on primary medical record review. The primary objective of this study was to determine the accuracy of the HOMR score when calculated using data abstracted from clinical notes that were available when patients were admitted to hospital. Secondary objectives included determining whether functional measures abstracted were significantly associated with death risk beyond the HOMR score and whether HOMR scores calculated from chart review deviated from those calculated from administrative data.
METHODS
Study Cohort
The study, which was approved by our local research ethics board, took place at the Ottawa Hospital, a 1000-bed teaching hospital that is the primary referral center in our region. We used the hospital admission registry to identify all people 18 years or older who were admitted to a nonpsychiatric service at our hospital between January 1, 2011 and December 31, 2011 (this time frame corresponds with the year used to derive the HOMR score). We excluded overnight patients in the same-day surgery or the bone-marrow transplant units (since they would not have been included in the original study) and those without a valid health card number (which was required to link to provincial data to identify outcomes). From this list, we randomly selected 5000 patients.
Primary Data Collection
For each patient, we retrieved all data required to calculate the HOMR score from the medical record (Table 1). Patient registration information in our electronic medical record was used to identify patient age, sex, admitting service, number of emergency department (ED) visits in the previous year, number of admissions in the previous year (the nursing triage note was reviewed for each admission to determine if it was by ambulance), and whether or not the patient had been discharged from hospital in the previous 30 days. The admitting service consult note was used to determine the admitting diagnosis and whether or not the patient was admitted directly to the intensive care unit. If they were present, the emergency nursing triage note, the ED record of treatment, the admission consult note, the pre-operative consult note, and consult notes were all used to determine the patient’s comorbidities, living status, and home oxygen status. Admission urgency was determined using information from the patient registration information and the ED nursing triage note. All data were abstracted from information that had been registered prior to when the patient was physically transferred to their hospital bed. This ensured that we used only data available at the start of the admission.
Patient functional status has been shown to be strongly associated with survival4 but HOMR only indirectly captures functional information (through the patient’s living status). We, therefore, collected more detailed functional information from the medical record by determining if the patient was dependent for any activities of daily living (ADL) from the emergency nursing triage note, the ED record of treatment, the admission consult note, and the pre-operative consultation. We also collected information that might indicate frailty, which we defined per Clegg et al.5 as “a state of increased vulnerability to poor resolution of homeostasis following a stress.” This information included: delirium or more than 1 fall recorded on the emergency nursing triage note, the ED record of treatment, or the admission consultation note; or whether a geriatric nursing specialist assessment occurred in the ED in the previous 6 months. Finally, we recorded possible indicators of limited social support (no fixed address [from patient registration and nursing triage note], primary contact is not a family member [from the emergency notes, consult, and patient registration], and no religion noted in system [from patient registration]). Patients for whom religion status was missing were classified as having “no religion.”
Analysis
These data were encrypted and linked anonymously to population-based databases to determine whether patients died within 1 year of admission to hospital. We calculated the chart-HOMR score using information from the chart review and determined its association with the outcome using bivariate logistic regression. We compared observed and expected risk of death within 1 year of admission to hospital for each chart-HOMR score value, with expected risks determined from the external validation study.3 We regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups to ensure adequate numbers in each group); and we gauged overall deviations from expected risk and the relationship between the observed and expected death risk (based on the chart-HOMR score) using the line’s intercept and slope, respectively.6 Next, we replicated methods from our studies2,3 to calculate the administrative-HOMR score in our study cohort using administrative databases. We compared these chart-HOMR and administrative-HOMR scores (and scores for each of its components). Finally, we determined which of the socio-functional factors were associated with 1-year death risk independent of the chart-HOMR score. We used the likelihood ratio test to determine whether these additional socio-functional factors significantly improved the model beyond the chart-HOMR score.7 This test subtracted the -2 logL value of the full model from that containing the chart-HOMR score alone, comparing its value to the χ2 distribution (with degrees of freedom equivalent to the number of additional parameters in the nested model) to determine statistical significance. All analyses were completed using SAS v9.4 (SAS Institute Inc., Cary, North Carolina).
RESULTS
There were 43,883 overnight hospitalizations at our hospital in 2011, and 38,886 hospitalizations were excluded: 1883 hospitalizations were in the same-day surgery or the bone-marrow transplant unit; 2485 did not have a valid health card number; 34,515 were not randomly selected; the records of 3 randomly selected patients had been blocked by our hospital’s privacy department; and 1 patient could not be linked with the population-based administrative datasets.
The 4996 study patients were middle-aged and predominantly female (Table 2). The extensive majority of patients was admitted from the community, was independent for ADL, had a family member as the principal contact, and had no admissions by ambulance in the previous year. Most people had no significant comorbidities or ED visits in the year prior to their admission. The mean chart-HOMR score was 22 (standard deviation [SD], 12), which is associated with a 1.2% expected risk of death within 1 year of hospital admission (Appendix 1).3
A total of 563 patients (11.3%) died within 1 year of admission to hospital (Table 2). In the study cohort, each chart-HOMR component was associated with death status. People who died were older, more likely to be male, had a greater number of important comorbidities, had more ED visits and admissions by ambulance in the previous year, and were more likely to have been discharged in the previous 30 days, and were admitted urgently, directly to the intensive care unit, or with complicated diagnoses. The mean chart-HOMR score differed extensively by survival status (37.4 [SD, 7.5] in those who died vs. 19.9 [SD, 12.2] in those who survived). Three of the socio-functional variables (delirium and falls noted on admission documents, and dependent for any ADL) also varied with death status.
The chart-HOMR score was strongly associated with the likelihood of death within 1 year of admission. When included in a logistic regression model having 1-year death as the outcome, a 1-point increase in the chart-HOMR score was associated with a 19% increase in the odds of death (P < 0.0001). This model (with only the chart-HOMR score) was highly discriminative (C statistic, 0.888) and well calibrated (Hosmer-Lemeshow test, 12.9 [8 df, P = 0.11]).
Observed and expected death risks by chart-HOMR score were similar (Figure 1). The observed total number of deaths (n = 563; 11.3%) exceeded the expected number of deaths (n = 437, 8.7%). When we regressed observed death risks on expected death risks for chart-HOMR scores (clustered into 22 groups), the Hosmer-Lemeshow test was significant, indicating that differences between observed and expected risks were beyond that expected by chance (Hosmer-Lemeshow test, 141.9, 21 df, P < 0.0001). The intercept of this model (0.035; 95% CI, 0.01-0.06) was statistically significant (P = 0.01), indicating that the observed number of cases significantly exceeded the expected; however, its calibration slope (1.02; 95% CI, 0.89-1.16) did not deviate significantly from unity, indicating that the relationship between the observed and expected death risk (based on the chart-HOMR score) remained intact (Figure 1).
The deviations between observed and expected death risks reflected deviations between the c chart-HOMR score and the administrative-HOMR score, with the former being significantly lower than the latter (Figure 2). Overall, the chart-HOMR score was 0.96 points lower (95% CI, 0.81-1.12) than the administrative-HOMR score. The HOMR score components that were notably underestimated using chart data included those for the age-Charlson Comorbidity Index interaction, living status, and admit points. Points for only 2 components (admitting service and admission urgency) were higher when calculated using chart data.
Four additional socio-functional variables collected from medical record review were significantly associated with 1-year death risk independent of the chart-HOMR score (Table 3). Admission documentation noting either delirium or falls were both associated with a significantly increased death risk (adjusted odds ratio [OR], 1.92 [95% CI, 1.24-2.96] and OR 1.96 [95% CI, 1.29-2.99], respectively). An independently increased death risk was also noted in patients who were dependent for any ADL (adjusted OR, 1.99 [95% CI, 1.24-3.19]). The presence of an ED geriatrics consultation within the previous 6 months was associated with a significantly decreased death risk of 60% (adjusted OR, 0.40 [95% CI, 0.20-0.81]). Adding these covariates to the logistic model with the chart-HOMR score significantly improved predictions (likelihood ratio statistic = 33.569, 4df, P < 0.00001).
DISCUSSION
In a large random sample of patients from our hospital, we found that the HOMR score using data abstracted from the medical record was significantly associated with 1-year death risk. The expected death risk based on the chart-HOMR score underestimated observed death risk but the relationship between the chart-HOMR score and death risk was similar to that in studies using administrative data. The HOMR score calculated using data from the chart was lower than that calculated using data from population-based administrative datasets; additional variables indicating patient frailty were significantly associated with 1-year death risk independent of the chart-HOMR score. Since the HOMR score was derived and initially validated using health administrative data, this study using data abstracted from the health record shows that the HOMR score has methodological generalizability.8
We think that our study has several notable findings. First, we found that data abstracted from the medical record can be used to calculate the HOMR score to accurately predict individual death risk. The chart-HOMR score discriminated very well between patients who did and did not die (C statistic, 0.88), which extensively exceeds the discrimination of published death risk indices (whose C statistics range between 0.69 and 0.82). It is also possible that chart abstraction for the HOMR score—without functional status—is simpler than other indices since its components are primarily very objective. (Other indices for hospital-based patients required factors that could be difficult to abstract reliably from the medical record including meeting more than 1 guideline for noncancer hospice care9; ambulation difficulties10; scales such as the Exton-Smith Scale or the Short Portable Mental Status Questionnaire11; weight loss12; functional status4; and pressure sore risk.13) Although expected risks for the chart-HOMR consistently underestimated observed risks (Figure 1), the mean deviation was small (with an absolute difference of 3.5% that can be used as a correction factor when determining expected risks with HOMR scores calculated from chart review), but it was an association between the chart-HOMR score and death risk that remained consistent through the cohort. Second, we found a small but significant decrease in the chart-HOMR score vs. the administrative-HOMR score (Figure 2). Some of these underestimates such as those for the number of ED visits or admissions by ambulance were expected since population-based health administrative databases would best capture such data. However, we were surprised that the comorbidity score was less when calculated using chart vs. database data (Figure 2). This finding is distinct from studies finding that particular comorbidities are documented in the chart are sometimes not coded.14,15 However, we identified comorbidities in the administrative databases using a 1-year ‘look-back’ period so that diagnostic codes from multiple hospitalizations (and from multiple hospitals) could be used to calculate the Charlson Comorbidity Index for a particular patient; this has been shown to increase the capture of comorbidities.16 Third, we found that variables from the chart review indicating frailty were predictive of 1-year death risk independent of the chart-HOMR score (Table 2). This illustrates that mortality risk prediction can be improved for particular patient groups by adding new covariates to the HOMR. Further work is required to determine how to incorporate these (and possibly other) covariates into the HOMR to create a unique chart-HOMR score. Finally, we found that a geriatrics assessment in the ED was associated with a significant (and notable) decrease in death risk. With these data, we are unable to indicate whether this association is causative. However, these findings indicate that the influence of emergency geriatric assessments on patient survival needs to be explored in more detail.
Several issues about our study should be considered when interpreting its results. First, this was a single-center study and the generalizability of our results to other centers is unknown. However, our study had the largest sample size of all primary data prognostic index validation studies1 ensuring that our results are, at the very least, internally reliable. In addition, our simple random sample ensured that we studied a broad assortment of patients to be certain that our results are representative of our institution. Second, we used a single abstractor for the study, which could limit the generalizability of our results. However, almost all the data points that were abstracted for our study were very objective.
In summary, our study shows that the HOMR score can be used to accurately predict 1-year death risk using data abstracted from the patient record. These findings will aid in individual patient prognostication for clinicians and researchers.
Disclosure
The authors report no financial conflicts of interest.
1. Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA. 2012;307(2):182-192. PubMed
2. van Walraven C. The Hospital-patient One-year Mortality Risk score accurately predicts long term death risk in hospitalized patients. J Clin Epidemiol. 2014;67(9):1025-1034. PubMed
3. van Walraven C, McAlister FA, Bakal JA, Hawken S, Donzé J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission. CMAJ. 2015;187(10):725-733. PubMed
4. Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285(23):2987-2994. PubMed
5. Clegg A, Young J, Iliffe S et al. Frailty in elderly people. The Lancet 2002;381:752-762. PubMed
6. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016;25(4):1692-1706. PubMed
7. Harrell FE Jr. Overview of Maximum Likelihood Estimation. Regression Modeling Strategies. New York, NY: Springer-Verlag; 2001: 179-212.
8. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130(6):515-524. PubMed
9. Fischer SM, Gozansky WS, Sauaia A, Min SJ, Kutner JS, Kramer A. A practical tool to identify patients who may benefit from a palliative approach: the CARING criteria. J Pain Symptom Manage. 2006;31(4):285-292. PubMed
10. Inouye SK, Bogardus ST, Jr, Vitagliano G, et al. Burden of illness score for elderly persons: risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments. Med Care. 2003;41(1):70-83. PubMed
11. Pilotto A, Ferrucci L, Franceschi M, et al. Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res. 2008;11(1):151-161. PubMed
12. Teno JM, Harrell FE Jr, Knaus W, et al. Prediction of survival for older hospitalized patients: the HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc. 2000;48(5 suppl):S16-S24. PubMed
13. Dramé M, Novella JL, Lang PO, et al. Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: the SAFES study. Eur J Epidemiol. 2008;23(12):783-791. PubMed
14. Kieszak SM, Flanders WD, Kosinski AS, Shipp CC, Karp H. A comparison of the Charlson comorbidity index derived from medical record data and administrative billing data. J Clin Epidemiol. 1999;52(2):137-142. PubMed
15. Quan H, Parsons GA, Ghali WA. Validity of procedure codes in International Classification of Diseases, 9th revision, clinical modification administrative data. Med Care. 2004;42(8):801-809. PubMed
16. Zhang JX, Iwashyna TJ, Christakis NA. The performance of different lookback periods and sources of information for Charlson cComorbidity adjustment in Medicare claims. Med Care. 1999;37(11):1128-1139. PubMed
1. Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA. 2012;307(2):182-192. PubMed
2. van Walraven C. The Hospital-patient One-year Mortality Risk score accurately predicts long term death risk in hospitalized patients. J Clin Epidemiol. 2014;67(9):1025-1034. PubMed
3. van Walraven C, McAlister FA, Bakal JA, Hawken S, Donzé J. External validation of the Hospital-patient One-year Mortality Risk (HOMR) model for predicting death within 1 year after hospital admission. CMAJ. 2015;187(10):725-733. PubMed
4. Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285(23):2987-2994. PubMed
5. Clegg A, Young J, Iliffe S et al. Frailty in elderly people. The Lancet 2002;381:752-762. PubMed
6. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016;25(4):1692-1706. PubMed
7. Harrell FE Jr. Overview of Maximum Likelihood Estimation. Regression Modeling Strategies. New York, NY: Springer-Verlag; 2001: 179-212.
8. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130(6):515-524. PubMed
9. Fischer SM, Gozansky WS, Sauaia A, Min SJ, Kutner JS, Kramer A. A practical tool to identify patients who may benefit from a palliative approach: the CARING criteria. J Pain Symptom Manage. 2006;31(4):285-292. PubMed
10. Inouye SK, Bogardus ST, Jr, Vitagliano G, et al. Burden of illness score for elderly persons: risk adjustment incorporating the cumulative impact of diseases, physiologic abnormalities, and functional impairments. Med Care. 2003;41(1):70-83. PubMed
11. Pilotto A, Ferrucci L, Franceschi M, et al. Development and validation of a multidimensional prognostic index for one-year mortality from comprehensive geriatric assessment in hospitalized older patients. Rejuvenation Res. 2008;11(1):151-161. PubMed
12. Teno JM, Harrell FE Jr, Knaus W, et al. Prediction of survival for older hospitalized patients: the HELP survival model. Hospitalized Elderly Longitudinal Project. J Am Geriatr Soc. 2000;48(5 suppl):S16-S24. PubMed
13. Dramé M, Novella JL, Lang PO, et al. Derivation and validation of a mortality-risk index from a cohort of frail elderly patients hospitalised in medical wards via emergencies: the SAFES study. Eur J Epidemiol. 2008;23(12):783-791. PubMed
14. Kieszak SM, Flanders WD, Kosinski AS, Shipp CC, Karp H. A comparison of the Charlson comorbidity index derived from medical record data and administrative billing data. J Clin Epidemiol. 1999;52(2):137-142. PubMed
15. Quan H, Parsons GA, Ghali WA. Validity of procedure codes in International Classification of Diseases, 9th revision, clinical modification administrative data. Med Care. 2004;42(8):801-809. PubMed
16. Zhang JX, Iwashyna TJ, Christakis NA. The performance of different lookback periods and sources of information for Charlson cComorbidity adjustment in Medicare claims. Med Care. 1999;37(11):1128-1139. PubMed
© 2017 Society of Hospital Medicine