Yvonne Schoon, MD, PhD

Anke Persoon, RN, PhD

Yvonne Schoon, MD, PhD

GM Marcel

Olde Rikkert, MD, PhD

Patient‐reported quality of care is currently an important outcome measure. Ideally, quality of care is assessed by measuring patient's experiences rather than patient satisfaction, as most patients are satisfied with the care they receive, even if the quality is poor.[1] Within the study of the CareWell in Hospital (CWH) program[2]which aims to improve quality of care for frail inpatients age 70 yearswe aimed to assess experiences using a questionnaire to determine the quality of hospital care from the perspective of elderly inpatients. This questionnaire should specifically address whether individualized, integrated care is delivered, with an emphasis on autonomy and maintaining patient independence as well as integrating well‐being into hospital care, all of which are aims of the CWH program. In this, it follows the perspective of integrated care as enabling the achievement of common goals and optimal care results from the patients' view: Care should be sensitive to the characteristics and needs of individual patients.[3]

In the Netherlands, a patient questionnaire to measure experiences with hospital care was carefully developed (partially based on the Consumer Assessment of Healthcare Providers and Systems) and is used to obtain information for national benchmarking: the Consumer Quality Index (CQI).[4] However, we considered this questionnaire containing 78 core questions as well as the time between discharge and measurement (often several months) too long for frail elderly patients, as they have complex, multidisciplinary needs and may have difficulty communicating their needs and reporting their experienced quality of care.

Here, we report on the development and validation of a questionnaire that is based on the CQI and can be used to measure the quality of individualized and integrated hospital care as experienced by inpatients age 70 years.

METHODS

Development

The predefined criteria for the questionnaire were that it should be brief, thereby reducing the burden placed on frail elderly persons; cover the aims of CWH; and measure experiences rather than satisfaction.

Ten categories were initially formulated to match CWH's goals of autonomy, independence, well‐being, individualized care, communication, coordination of care, continuity of care, patient safety, and competence of physicians and nurses. Items from the CQI questionnaire database[5] were selected for each category. Ten members of a panel representing the elderly target group were invited to select the 3 most important questions in each category (first Delphi round). This panel is an important party within a regional network of care and well‐being organizations and involved in discussing the various regional care and/or well‐being projects when it concerns their content and value for elderly persons. They represent elderly persons through their position in elderly‐care or informal care organizations or from personal experiences. During a second Delphi round, they determined whether the individual items of the concept questionnaire were clearly stated, comprehensible to frail elderly patients, represent quality of care, have appropriate answer categories, and so forth. The final questionnaire was edited to match the reading level of a 12‐year‐old and approved by the panel in a face‐to‐face meeting. By this process, content validity was ensured.[6]

Data Collection

The final questionnaire was mailed to both frail and nonfrail medical and surgical inpatients who were included in the CWH before‐after study (January 2011 to July 2012) 1 week after their discharge, by a research assistant (see Supporting Information, Appendix A, in the online version of this article for a description of the study and CWH program).

Patients in the CWH study who returned the questionnaire during the postimplementation measurement period were asked to participate in the test‐retest reliability study until a predetermined sample size of 75 was reached (March 2012 to November 2012). The target interval between returning the first and second questionnaire was 2 to 14 days.[7]

In addition, patients admitted to the geriatrics departmentand therefore assumed to be frailreceived the questionnaire upon discharge (February 2012 to April 2013). The geriatrics department administered the questionnaire anonymously for evaluation and quality‐improvement purposes, as part of usual care. The secretary included the questionnaire in all patient files, and a nurse provided the questionnaire to patients together with other important discharge documents. This questionnaire also included a question regarding goal attainment, as this reflects whether what is important to the most frail elderly patients was accomplished.

Validation and Analysis

Data were analyzed using the statistical software program SPSS version 18.0 (SPSS Inc., Chicago, IL.).

Data

Characteristics of (non)responders, levels of missing data, and measurement range were assessed using descriptive statistics.

Reliability

Internal consistency was assessed by calculating Cronbach's for all available questionnaires with complete data. The answer categories were recoded to a 010 scale; 10 represents the highest quality of care. Test‐retest reliability[6] was assessed by calculating Cohen's for individual questions and intraclass correlation (ICC) for the questionnaire's mean score.

Validity

The following hypotheses were tested in order to assess construct validity: lower scores for female patients[8] and for patients who rate their health lower,[9] and with higher education[8, 9]; higher scores for patients who had an elective admission[8] and whose treatment goals were achieved (own reasoning). Finally, whether patients answered the questionnaire independently or with help should not affect scores (own reasoning). The Spearman was calculated for nonparametric and ordinal data.

In addition, we performed a Kruskal‐Wallis analysis to test the hypothesis that patients admitted to different departments have different scores. Second, we used the Mann‐Whitney U test to detect differences before and after implementation of the CWH program.

For all these analyses, only questionnaires with complete data were included.

RESULTS

Development

The selected answers within the categories communication and competence of nurses and physicians by the panel overlapped with questions from the other 8 categories; thus, the final questionnaire contains 8 core questions (Table 1) (see Supporting Information, Appendix B, in the online version of this article).

Table 1.The 8 Core Questions of the CareWell in Hospital Questionnaire
Question
NOTE: The questionnaire for the geriatrics department included 1 additional question: Within a few days of your hospital admission, a doctor discussed the goal of the admission with you. Did you achieve your goal(s) satisfactorily? (no, not at all; yes, partially; yes, completely; don't know; doctor did not discuss my goals). See Supporting Information, Appendix A, in the online version of this article for the entire questionnaire, including the answer categories.
1. Were you informed sufficiently by your doctor regarding the various options for treating your health problems?
2. Were you able to indicate which treatment and/or care you preferred?
3. During your hospital stay, could you co‐decide what was important to your care?
4. During your hospital stay, were you supported in keeping busy and finding social contacts and activities?
5. Did you know to whom you can go within the hospital with questions, problems, or complaints?
6. Before discharge, did you talk with a member of the hospital staff regarding the care you would need after discharge?
7. Did a member of the hospital staff inform the key people and/or care providers of your discharge from the hospital?
8. During your hospital stay, did you experience 1 or more of the following events?
Did you fall?
Did you become confused?
Did you develop pressure ulcers?
Did medication errors occur?
Did you develop a urinary tract infection?
Did you develop a wound infection?
Did you experience complications with your surgery and/or treatment?

Data Collection

Figure 1 shows a flowchart of the questionnaires.

Flowchart of the available questionnaires returned by elderly inpatients. Abbreviations: CWH, CareWell in Hospital.

Table 2 presents data of responders compared with nonresponders who were included in the CWH study (N = 293). Patients were age 70 years and admitted 48 hours. Patients responded 14.8 11.3 days after discharge (n = 265). Response rate was 75.8%. From 18 responders no baseline characteristics were available, as only the questionnaire was collected from them to reach n = 75 for test‐retest purposes.

Table 2.Characteristics of the Responding (n = 293) and Nonresponding (n = 88) Patients Included in the CareWell in Hospital Before‐After Study
	No.	Responders	No.	Nonresponders	P Value
NOTE: Data on baseline characteristics from 18 patients in the post‐CWH measurement period are missing, and from those patients only the CareWell in Hospital questionnaires were gathered in order to reach n = 75 for test‐retest purposes. CIRS‐G ranging from 0 to 56 (with a higher score indicating more comorbidity).[14] MMSE ranging from 0 to 30 (with 30 representing the best score). Length of stay is defined as the time between admission to a CWH study department and discharge from a CWH study department. Abbreviations: CIRS‐G, Cumulative Illness Rating ScaleGeriatrics; CWH, CareWell in Hospital; MMSE, Mini‐Mental State Examination; SD, standard deviation.
Age, y SD	275	76.9 5.2	88	77.3 5.5	0.701
Male sex, n (%)	275	156 (56.7)	88	52 (59.1)	0.696
CIRS‐G, score SD	274	12.8 5.0	88	13.9 5.0	0.071
MMSE admission, score SD	264	26.7 3.7	82	25.1 4.8	0.001
MMSE discharge, score SD	230	26.9 3.7	66	25.8 4.4	0.026
Length of stay, days SD	275	8.2 7.4	88	9.6 9.7	0.322
Department, surgical (%)	275	170 (61.8)	88	56 (63.6)	0.759
Admission type, n (%)	275		88		0.343
Emergency		82 (29.8)		22 (25.0)
Elective		138 (50.2)		52 (59.1)
From other hospital or other department		55 (20.0)		14 (15.9)
Marital status, alone (%)	273	187 (68.5)	84	50 (59.5)	0.128
Discharge destination, n (%)	275		88		0.000
Home		197 (71.6)		54 (61.4)
Other hospital		69 (25.1)		20 (22.7)
Care facility		9 (3.3)		14 (15.9)
Readmission, n (%)	275	38 (13.8)	88	7 (8.0)	0.146
Readmission <1 mo, n (%)	275	28 (10.2)	88	14 (15.9)	0.144
Death <3 mo following discharge, n (%)	274	9 (3.3)	86	5 (5.8)	0.233
Received CWH intervention	149	43 (28.9)	33	15 (45.5)	0.064

Patients in the geriatrics department responded in 10.5 15.0 days (n = 111). Mean length of stay was 9.0 7.2 days (n = 116). Data regarding other baseline characteristics and response rate were unavailable due to privacy concerns.

Data Characteristics

Table 3 summarizes data of all 470 questionnaires. Response rates to the answer options ranged from 3.8% to 66.8%. Missing data among the questions ranged from 1.7% within question 8 to 7.0% within question 4. Upon combining the answer categories I don't know and missing, 7/8 questions had >10% missing data; the questions 2 and 3 had the highest percentage of missing data due to the I don't know answer option. The reasons stated by the respondents for why they could not answer these questions included cognitive disabilities; the perception that, because there was only one option (eg, in case of emergency admissions), the question did not apply to them; and/or that the patients preferred not to co‐decide because they felt that the physician knows best and can decide what is best.

Table 3.Data Quality and Range and Test‐Retest Reliability of All Questionnaires Received
	Data (n = 470)		Test‐Retest (n = 78)
	No.	%	No.
NOTE: For adverse events, the minimum amount of missing data was 1.7%. Sum scores range from 0 to 80. Mean scores range from 0 to 10. = Cohen's . Abbreviations: DK, don't know; ICC, intraclass correlation coefficient; Max, maximum; MIS, missing.
Sufficiently informed regarding treatment options			65	0.278
Not at all	23	4.9
Sometimes	90	19.1
Often	115	24.5
Every time	191	40.6
Don't know	29	6.2
Missing	21	4.7
Treatment and care preferences discussed			59	0.415
Not at all	89	18.9
Sometimes	78	16.6
Often	61	13.0
Every time	111	23.6
Don't know	103	21.9
Missing	28	6.0
Co‐decide regarding important issues			56	0.295
Not at all	75	16.0
Sometimes	86	18.3
Often	67	14.3
Every time	112	23.8
Don't know	98	20.9
Missing	32	6.8
Supported in finding (social) activities			73	0.533
Not at all	72	15.3
A little	66	14.0
Good	109	23.2
Very good	36	7.7
Not applicable	130	27.7
Don't know	24	5.1
Missing	33	7.0
Knows relevant person for questions, problems, complaints			77	0.652
Yes	279	59.4
No	107	22.8
Don't know	67	14.3
Missing	17	3.6
Discussed postdischarge care needs			75	0.574
Yes, sufficient	311	66.2
Yes, but insufficient	26	5.5
No	99	20.3
I don't know/I don't remember	18	3.8
Missing	19	4.0
Hospital informed other important people/providers of discharge			69	0.405
No	45	9.6
Some were informed	54	11.5
Yes	314	66.8
Don't know	38	8.1
Missing	19	4.0
Adverse events during hospital admission	DK	MIS	78	0.816
Fall, confusion, pressure ulcer, medication error, bladder infection, wound infection, complication of surgery/treatment	Max 9.1%	Max 4.3%
	Sum	Mean	No.	ICC
Mean score on the total questionnaire, complete cases (n = 222)	51.9 18.3	6.5 2.3	39	0.745

Reliability

Of the 470 questionnaires, 222 (47.2%) had complete data and were used to analyze internal consistency. Cronbach's for the 8‐item questionnaire was 0.70 (good internal consistency).

Seventy‐eight questionnaires were available to measure test‐retest reliability. The interval between test‐retest was 8.7 4.8 days; 94.7% was returned within the targeted 14 days. Thirty‐eight patients had complete data for both measurements: ICC on the mean score of the questionnaire was 0.75 (95% confidence interval [CI]: 0.56‐0.86), which indicates good test‐retest reliability (Table 3). Including patients with incomplete data (1 to 2 missing items) yielded an ICC >0.70. Among the individual questions, Cohen's ranged from 0.28 to 0.82.

Validity

The mean questionnaire score was significantly correlated with goals achieved while hospitalized (Table 4).

Table 4.Construct Validity of the CareWell in Hospital Questionnaire Based on All Questionnaires With Complete Data on Both the Variable and the Questionnaire Score
Variable	Response	No.a	Score SD	Correlation
NOTE: Mean scores range from 0 to 10. Abbreviations: F, female; M, male; SD, standard deviation. a The number differs per analysis. Education level was not known for every patient; this variable was extracted from a different questionnaire. Admission type includes only emergency admission and elective admission; patients could also be transferred from another department or hospital, but this was not included as a category as this might include emergency as well as elective admissions. Goal of admission was only available for patients from the geriatrics department, whereas educational level and admission type were not available for patients from the geriatrics department. b Correlation (Spearman ) is significant at the 0.01 level (1‐tailed for goal achieved).
Sex	M	114	6.3 2.3	0.080
	F	108	6.7 2.3
Health status	Excellent	1		0.071
	Very good	5	7.9 2.0
	Good	52	6.7 2.4
	Fair	120	6.5 2.2
	Poor	28	6.2 2.1
Education level	<6 grades primary school	4	4.9 1.2	0.068
	Primary school	19	6.4 2.5
	Higher than primary school	6	7.6 1.2
	Practical training	27	6.0 2.2
	Secondary vocational training	41	6.1 2.5
	Pre‐university education	2	7.2 4.0
	University/higher education	20	6.8 2.2
Admission type	Emergency	31	6.5 2.6	0.015
	Elective	61	6.6 2.0
Goal of admission achieved	Yes	33	7.6 1.7	0.319b
Goal of admission achieved	Partially	24	6.6 2.1
	No	6	4.7 2.8
Respondent	Patient only	117	6.7 2.2	0.063
	Patient with help	59	5.9 2.3
	Other person	41	6.7 2.4

Mean scores did not differ significantly between departments (geriatrics: 6.8 2.2, n = 88; cardiothoracic surgery and lung diseases: 6.5 2.4, n = 54; internal medicine: 6.3 2.5, n = 30; general surgery: 6.0 2.2, n = 50; P = 0.234).

In addition, mean scores did not differ significantly before (6.5 2.2, n = 53) and after (6.1 2.4, n = 67) implementation of the CWH study (P = 0.320).

DISCUSSION

The CareWell in Hospital patient questionnaire is a brief 8‐item questionnaire to assess the experiences of elderly patients regarding integrated hospital care. It showed good internal consistency and test‐retest reliability, and low responsiveness. Here we discuss some issues related to the preset criteria of the questionnaire.

First, a panel representing the elderly target population was used to develop the questionnaire in order to ensure content validity, which was confirmed by good internal consistency. Yet, with respect to individualized, integrated care for frail elderly patients, we recommend including a question regarding the involvement of informal caregivers during the hospital stay, as they are important partners in healthcare.[10]

Second, the questionnaire was kept short because it should not be a burden and feasible for frail patients to complete. Nonetheless, some of the questions had a high nonresponse rate, and many patients answered I don't know, particularly to the questions 2 and 3. It does not necessarily mean that these questions are poor in quality; it could also indicate that offering individualized care is not yet embedded in the culture of elderly patients and care professionals, such that patients consider such questions to be irrelevant.[11, 12] Nevertheless, we suggest to further explore the feasibility of the questionnaire and potential additional methods for the most frail elderly,[13] who might have been excluded from the CWH study sample at this point (Table 2).

Third, the questionnaire measures experiences rather than satisfaction. Patient‐satisfaction scores are generally tightly correlated with the age, sex, education level, health status, and the person completing the questionnaire.[8] In our study, the correlation did not reach statistical significance. Nevertheless, the achievement of preset goals was correlated significantly with mean CWH scores (Table 4). These findings may indicate that individualized care experiences can indeed be assessed better using this questionnaire. Test‐retest reliability also supports validity, as we expectedand, indeed, sawhigher reliability among the more objective questions (eg, question 8). The most valuing question is question 1, which also had the lowest reliability; the word sufficiently should perhaps be removed in the next version in order to increase its reliability and objectivity.

Finally, scores did not differ between before and after implementation of the CWH program, which suggests either that the questionnaire is unable to detect change or that the program was not sufficiently effective to invoke change yet. The latter option seems plausible, as changes in the provision of individualized care were ongoing. In addition, the items on which favorable differences can be seen for CWH are in fact the items that could be most directly influenced by the CWH interventionists, questions 4, 6, and 7 (see Supporting Information, Appendix C, in the online version of this article). Lastly, we performed an extra analysis concerning the discriminating property of the questionnaire in a subgroup of frail elderly patients; we do see a significant difference in scores between the frail patients in the geriatrics department and the frail patients who received the CWH intervention: 6.8 (n = 88) vs 4.8 (n = 13) for complete data, respectively, P = 0.013; and 6.8 (n = 155) vs 5.7 (n = 37) for incomplete data (2 items missing), P = 0.017 (Mann‐Whitney U test). This may indicate that the questionnaire can measure differences in quality of care for specifically the frail elderly patients between departments. However, these issuesincluding validity and reliability characteristics per specific patient subgroupwarrant further research using a larger sample.

CONCLUSIONS

In conclusion, the CareWell in Hospital patient questionnaire is a feasible and reliable tool for assessing experiences of frail elderly inpatients in the provision of individualized, integrated care. To improve the questionnaire, we recommend to add a question regarding the participation of informal caregivers during the hospital stay, investigate the response rate to questions regarding participation and shared decision‐making, and study responsiveness issues further.

Acknowledgements

The authors thank Gerda van Straaten, Anne Kuijpers, and Thijs Cauven for their support with data collection. We thank all members of the ZOWEL Study Group and the panel representing the elderly target group.

Disclosures: The work was made possible by grant 60‐6190‐098‐272 and grant 60‐61900‐98‐129 of the National Programme for Elderly Care, coordinated and sponsored by ZonMw, The Netherlands, Organization of Health Research and Development. The authors report no conflicts of interest.

Files

References

Kalucy L, Katterl R, Jackson‐Bowers E. Patient Experience of Health Care Performance. Adelaide, Australia: Primary Health Care Research November 2009. Available at: http://dspace.flinders.edu.au/jspui/bitstream/2328/26594/1/PIR NOV 09 Full.pdf.
Bakker FC, Persoon A, Schoon Y, Rikkert MGM. Hospital Elder Life Program integrated in Dutch hospital care: a pilot study. J Am Geriatr Soc. 2013;61(4):641–642.
Kodner DL, Spreeuwenberg C. Integrated care: meaning, logic, applications, and implications—a discussion paper. Int J Integr Care. 2002;2:e12.
Sixma H, Spreeuwenberg P, Zuidgeest M, Rademakers J. CQ‐index Ziekenhuisopname: meetinstrumentontwikkeling. Kwaliteit van de zorg tijdens ziekenhuisopnames vanuit het perspectief van patiënten. De ontwikkeling van het instrument, de psychometrische eigenschappen en het discriminerend vermogen [in Dutch]. Utrecht, The Netherlands: NIVEL (Netherlands Institute for Health Services Research), 2009.
Centrum Klantervaring Zorg.CQI vragenbank (CQI questionnaire database). Available at: http://nvl002.nivel.nl/CQI. Accessed May–June 2010.
Terwee CB, Bot SD, Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.
Streiner D, Norman G. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th ed. Oxford, UK: Oxford University Press; 2008:182–183.
Hordacre AL, Taylor A, Pirone C, Adams RJ. Assessing patient satisfaction: implications for South Australian public hospitals. Aust Health Rev. 2005;29(4):439–446.
Hekkert KD, Cihangir S, Kleefstra SM, Berg B, Kool RB. Patient satisfaction revisited: a multilevel approach. Soc Sci Med. 2009;69(1):68–75.
Wressle E, Eriksson L, Fahlander A, et al. Relatives' perspective on the quality of geriatric care and rehabilitation—development and testing of a questionnaire. Scand J Caring Sci. 2008;22(4):590–595.
Ekdahl AW, Andersson L, Wiréhn AB, Friedrichsen M. Are elderly people with co‐morbidities involved adequately in medical decision making when hospitalised? A cross‐sectional survey. BMC Geriatr. 2011;11:46.
Wilkinson C, Khanji M, Cotter PE, Dunne O, O'Keeffe ST. Preferences of acutely ill patients for participation in medical decision‐making. Qual Saf Health Care. 2008;17(2):97–100.
Goldberg SE, Harwood RH. Experience of general hospital care in older patients with cognitive impairment: are we measuring the most vulnerable patients' experience? BMJ Qual Saf. 2013;doi:10.1136/bmjqs‐2013‐001961.
Miller M, Towers A. A Manual of Guidelines for Scoring the Cumulative Illness Rating Scale for Geriatrics (CIRS‐G). Pittsburgh, PA: University of Pittsburgh School of Medicine, Department of Geriatric Psychiatry; 1991.

Article PDF

Issue

Journal of Hospital Medicine - 9(5)

Page Number

324-329

Read more about Inpatient Questionnaire for Frail Elders

Sections

Brief Reports

Author(s)

Anke Persoon, RN, PhD

Yvonne Schoon, MD, PhD

GM Marcel

Olde Rikkert, MD, PhD

Author(s)

Anke Persoon, RN, PhD

Yvonne Schoon, MD, PhD

GM Marcel

Olde Rikkert, MD, PhD

Files

Files

Article PDF

Article PDF

Patient‐reported quality of care is currently an important outcome measure. Ideally, quality of care is assessed by measuring patient's experiences rather than patient satisfaction, as most patients are satisfied with the care they receive, even if the quality is poor.[1] Within the study of the CareWell in Hospital (CWH) program[2]which aims to improve quality of care for frail inpatients age 70 yearswe aimed to assess experiences using a questionnaire to determine the quality of hospital care from the perspective of elderly inpatients. This questionnaire should specifically address whether individualized, integrated care is delivered, with an emphasis on autonomy and maintaining patient independence as well as integrating well‐being into hospital care, all of which are aims of the CWH program. In this, it follows the perspective of integrated care as enabling the achievement of common goals and optimal care results from the patients' view: Care should be sensitive to the characteristics and needs of individual patients.[3]

In the Netherlands, a patient questionnaire to measure experiences with hospital care was carefully developed (partially based on the Consumer Assessment of Healthcare Providers and Systems) and is used to obtain information for national benchmarking: the Consumer Quality Index (CQI).[4] However, we considered this questionnaire containing 78 core questions as well as the time between discharge and measurement (often several months) too long for frail elderly patients, as they have complex, multidisciplinary needs and may have difficulty communicating their needs and reporting their experienced quality of care.

Here, we report on the development and validation of a questionnaire that is based on the CQI and can be used to measure the quality of individualized and integrated hospital care as experienced by inpatients age 70 years.

METHODS

Development

The predefined criteria for the questionnaire were that it should be brief, thereby reducing the burden placed on frail elderly persons; cover the aims of CWH; and measure experiences rather than satisfaction.

Ten categories were initially formulated to match CWH's goals of autonomy, independence, well‐being, individualized care, communication, coordination of care, continuity of care, patient safety, and competence of physicians and nurses. Items from the CQI questionnaire database[5] were selected for each category. Ten members of a panel representing the elderly target group were invited to select the 3 most important questions in each category (first Delphi round). This panel is an important party within a regional network of care and well‐being organizations and involved in discussing the various regional care and/or well‐being projects when it concerns their content and value for elderly persons. They represent elderly persons through their position in elderly‐care or informal care organizations or from personal experiences. During a second Delphi round, they determined whether the individual items of the concept questionnaire were clearly stated, comprehensible to frail elderly patients, represent quality of care, have appropriate answer categories, and so forth. The final questionnaire was edited to match the reading level of a 12‐year‐old and approved by the panel in a face‐to‐face meeting. By this process, content validity was ensured.[6]

Data Collection

The final questionnaire was mailed to both frail and nonfrail medical and surgical inpatients who were included in the CWH before‐after study (January 2011 to July 2012) 1 week after their discharge, by a research assistant (see Supporting Information, Appendix A, in the online version of this article for a description of the study and CWH program).

Patients in the CWH study who returned the questionnaire during the postimplementation measurement period were asked to participate in the test‐retest reliability study until a predetermined sample size of 75 was reached (March 2012 to November 2012). The target interval between returning the first and second questionnaire was 2 to 14 days.[7]

In addition, patients admitted to the geriatrics departmentand therefore assumed to be frailreceived the questionnaire upon discharge (February 2012 to April 2013). The geriatrics department administered the questionnaire anonymously for evaluation and quality‐improvement purposes, as part of usual care. The secretary included the questionnaire in all patient files, and a nurse provided the questionnaire to patients together with other important discharge documents. This questionnaire also included a question regarding goal attainment, as this reflects whether what is important to the most frail elderly patients was accomplished.

Validation and Analysis

Data were analyzed using the statistical software program SPSS version 18.0 (SPSS Inc., Chicago, IL.).

Data

Characteristics of (non)responders, levels of missing data, and measurement range were assessed using descriptive statistics.

Reliability

Internal consistency was assessed by calculating Cronbach's for all available questionnaires with complete data. The answer categories were recoded to a 010 scale; 10 represents the highest quality of care. Test‐retest reliability[6] was assessed by calculating Cohen's for individual questions and intraclass correlation (ICC) for the questionnaire's mean score.

Validity

The following hypotheses were tested in order to assess construct validity: lower scores for female patients[8] and for patients who rate their health lower,[9] and with higher education[8, 9]; higher scores for patients who had an elective admission[8] and whose treatment goals were achieved (own reasoning). Finally, whether patients answered the questionnaire independently or with help should not affect scores (own reasoning). The Spearman was calculated for nonparametric and ordinal data.

In addition, we performed a Kruskal‐Wallis analysis to test the hypothesis that patients admitted to different departments have different scores. Second, we used the Mann‐Whitney U test to detect differences before and after implementation of the CWH program.

For all these analyses, only questionnaires with complete data were included.

RESULTS

Development

The selected answers within the categories communication and competence of nurses and physicians by the panel overlapped with questions from the other 8 categories; thus, the final questionnaire contains 8 core questions (Table 1) (see Supporting Information, Appendix B, in the online version of this article).

Table 1.The 8 Core Questions of the CareWell in Hospital Questionnaire
Question
NOTE: The questionnaire for the geriatrics department included 1 additional question: Within a few days of your hospital admission, a doctor discussed the goal of the admission with you. Did you achieve your goal(s) satisfactorily? (no, not at all; yes, partially; yes, completely; don't know; doctor did not discuss my goals). See Supporting Information, Appendix A, in the online version of this article for the entire questionnaire, including the answer categories.
1. Were you informed sufficiently by your doctor regarding the various options for treating your health problems?
2. Were you able to indicate which treatment and/or care you preferred?
3. During your hospital stay, could you co‐decide what was important to your care?
4. During your hospital stay, were you supported in keeping busy and finding social contacts and activities?
5. Did you know to whom you can go within the hospital with questions, problems, or complaints?
6. Before discharge, did you talk with a member of the hospital staff regarding the care you would need after discharge?
7. Did a member of the hospital staff inform the key people and/or care providers of your discharge from the hospital?
8. During your hospital stay, did you experience 1 or more of the following events?
Did you fall?
Did you become confused?
Did you develop pressure ulcers?
Did medication errors occur?
Did you develop a urinary tract infection?
Did you develop a wound infection?
Did you experience complications with your surgery and/or treatment?

Data Collection

Figure 1 shows a flowchart of the questionnaires.

Table 2 presents data of responders compared with nonresponders who were included in the CWH study (N = 293). Patients were age 70 years and admitted 48 hours. Patients responded 14.8 11.3 days after discharge (n = 265). Response rate was 75.8%. From 18 responders no baseline characteristics were available, as only the questionnaire was collected from them to reach n = 75 for test‐retest purposes.

Table 2.Characteristics of the Responding (n = 293) and Nonresponding (n = 88) Patients Included in the CareWell in Hospital Before‐After Study
	No.	Responders	No.	Nonresponders	P Value
NOTE: Data on baseline characteristics from 18 patients in the post‐CWH measurement period are missing, and from those patients only the CareWell in Hospital questionnaires were gathered in order to reach n = 75 for test‐retest purposes. CIRS‐G ranging from 0 to 56 (with a higher score indicating more comorbidity).[14] MMSE ranging from 0 to 30 (with 30 representing the best score). Length of stay is defined as the time between admission to a CWH study department and discharge from a CWH study department. Abbreviations: CIRS‐G, Cumulative Illness Rating ScaleGeriatrics; CWH, CareWell in Hospital; MMSE, Mini‐Mental State Examination; SD, standard deviation.
Age, y SD	275	76.9 5.2	88	77.3 5.5	0.701
Male sex, n (%)	275	156 (56.7)	88	52 (59.1)	0.696
CIRS‐G, score SD	274	12.8 5.0	88	13.9 5.0	0.071
MMSE admission, score SD	264	26.7 3.7	82	25.1 4.8	0.001
MMSE discharge, score SD	230	26.9 3.7	66	25.8 4.4	0.026
Length of stay, days SD	275	8.2 7.4	88	9.6 9.7	0.322
Department, surgical (%)	275	170 (61.8)	88	56 (63.6)	0.759
Admission type, n (%)	275		88		0.343
Emergency		82 (29.8)		22 (25.0)
Elective		138 (50.2)		52 (59.1)
From other hospital or other department		55 (20.0)		14 (15.9)
Marital status, alone (%)	273	187 (68.5)	84	50 (59.5)	0.128
Discharge destination, n (%)	275		88		0.000
Home		197 (71.6)		54 (61.4)
Other hospital		69 (25.1)		20 (22.7)
Care facility		9 (3.3)		14 (15.9)
Readmission, n (%)	275	38 (13.8)	88	7 (8.0)	0.146
Readmission <1 mo, n (%)	275	28 (10.2)	88	14 (15.9)	0.144
Death <3 mo following discharge, n (%)	274	9 (3.3)	86	5 (5.8)	0.233
Received CWH intervention	149	43 (28.9)	33	15 (45.5)	0.064

Patients in the geriatrics department responded in 10.5 15.0 days (n = 111). Mean length of stay was 9.0 7.2 days (n = 116). Data regarding other baseline characteristics and response rate were unavailable due to privacy concerns.

Data Characteristics

Table 3 summarizes data of all 470 questionnaires. Response rates to the answer options ranged from 3.8% to 66.8%. Missing data among the questions ranged from 1.7% within question 8 to 7.0% within question 4. Upon combining the answer categories I don't know and missing, 7/8 questions had >10% missing data; the questions 2 and 3 had the highest percentage of missing data due to the I don't know answer option. The reasons stated by the respondents for why they could not answer these questions included cognitive disabilities; the perception that, because there was only one option (eg, in case of emergency admissions), the question did not apply to them; and/or that the patients preferred not to co‐decide because they felt that the physician knows best and can decide what is best.

Table 3.Data Quality and Range and Test‐Retest Reliability of All Questionnaires Received
	Data (n = 470)		Test‐Retest (n = 78)
	No.	%	No.
NOTE: For adverse events, the minimum amount of missing data was 1.7%. Sum scores range from 0 to 80. Mean scores range from 0 to 10. = Cohen's . Abbreviations: DK, don't know; ICC, intraclass correlation coefficient; Max, maximum; MIS, missing.
Sufficiently informed regarding treatment options			65	0.278
Not at all	23	4.9
Sometimes	90	19.1
Often	115	24.5
Every time	191	40.6
Don't know	29	6.2
Missing	21	4.7
Treatment and care preferences discussed			59	0.415
Not at all	89	18.9
Sometimes	78	16.6
Often	61	13.0
Every time	111	23.6
Don't know	103	21.9
Missing	28	6.0
Co‐decide regarding important issues			56	0.295
Not at all	75	16.0
Sometimes	86	18.3
Often	67	14.3
Every time	112	23.8
Don't know	98	20.9
Missing	32	6.8
Supported in finding (social) activities			73	0.533
Not at all	72	15.3
A little	66	14.0
Good	109	23.2
Very good	36	7.7
Not applicable	130	27.7
Don't know	24	5.1
Missing	33	7.0
Knows relevant person for questions, problems, complaints			77	0.652
Yes	279	59.4
No	107	22.8
Don't know	67	14.3
Missing	17	3.6
Discussed postdischarge care needs			75	0.574
Yes, sufficient	311	66.2
Yes, but insufficient	26	5.5
No	99	20.3
I don't know/I don't remember	18	3.8
Missing	19	4.0
Hospital informed other important people/providers of discharge			69	0.405
No	45	9.6
Some were informed	54	11.5
Yes	314	66.8
Don't know	38	8.1
Missing	19	4.0
Adverse events during hospital admission	DK	MIS	78	0.816
Fall, confusion, pressure ulcer, medication error, bladder infection, wound infection, complication of surgery/treatment	Max 9.1%	Max 4.3%
	Sum	Mean	No.	ICC
Mean score on the total questionnaire, complete cases (n = 222)	51.9 18.3	6.5 2.3	39	0.745

Reliability

Of the 470 questionnaires, 222 (47.2%) had complete data and were used to analyze internal consistency. Cronbach's for the 8‐item questionnaire was 0.70 (good internal consistency).

Seventy‐eight questionnaires were available to measure test‐retest reliability. The interval between test‐retest was 8.7 4.8 days; 94.7% was returned within the targeted 14 days. Thirty‐eight patients had complete data for both measurements: ICC on the mean score of the questionnaire was 0.75 (95% confidence interval [CI]: 0.56‐0.86), which indicates good test‐retest reliability (Table 3). Including patients with incomplete data (1 to 2 missing items) yielded an ICC >0.70. Among the individual questions, Cohen's ranged from 0.28 to 0.82.

Validity

The mean questionnaire score was significantly correlated with goals achieved while hospitalized (Table 4).

Table 4.Construct Validity of the CareWell in Hospital Questionnaire Based on All Questionnaires With Complete Data on Both the Variable and the Questionnaire Score
Variable	Response	No.a	Score SD	Correlation
NOTE: Mean scores range from 0 to 10. Abbreviations: F, female; M, male; SD, standard deviation. a The number differs per analysis. Education level was not known for every patient; this variable was extracted from a different questionnaire. Admission type includes only emergency admission and elective admission; patients could also be transferred from another department or hospital, but this was not included as a category as this might include emergency as well as elective admissions. Goal of admission was only available for patients from the geriatrics department, whereas educational level and admission type were not available for patients from the geriatrics department. b Correlation (Spearman ) is significant at the 0.01 level (1‐tailed for goal achieved).
Sex	M	114	6.3 2.3	0.080
	F	108	6.7 2.3
Health status	Excellent	1		0.071
	Very good	5	7.9 2.0
	Good	52	6.7 2.4
	Fair	120	6.5 2.2
	Poor	28	6.2 2.1
Education level	<6 grades primary school	4	4.9 1.2	0.068
	Primary school	19	6.4 2.5
	Higher than primary school	6	7.6 1.2
	Practical training	27	6.0 2.2
	Secondary vocational training	41	6.1 2.5
	Pre‐university education	2	7.2 4.0
	University/higher education	20	6.8 2.2
Admission type	Emergency	31	6.5 2.6	0.015
	Elective	61	6.6 2.0
Goal of admission achieved	Yes	33	7.6 1.7	0.319b
Goal of admission achieved	Partially	24	6.6 2.1
	No	6	4.7 2.8
Respondent	Patient only	117	6.7 2.2	0.063
	Patient with help	59	5.9 2.3
	Other person	41	6.7 2.4

Mean scores did not differ significantly between departments (geriatrics: 6.8 2.2, n = 88; cardiothoracic surgery and lung diseases: 6.5 2.4, n = 54; internal medicine: 6.3 2.5, n = 30; general surgery: 6.0 2.2, n = 50; P = 0.234).

In addition, mean scores did not differ significantly before (6.5 2.2, n = 53) and after (6.1 2.4, n = 67) implementation of the CWH study (P = 0.320).

DISCUSSION

The CareWell in Hospital patient questionnaire is a brief 8‐item questionnaire to assess the experiences of elderly patients regarding integrated hospital care. It showed good internal consistency and test‐retest reliability, and low responsiveness. Here we discuss some issues related to the preset criteria of the questionnaire.

First, a panel representing the elderly target population was used to develop the questionnaire in order to ensure content validity, which was confirmed by good internal consistency. Yet, with respect to individualized, integrated care for frail elderly patients, we recommend including a question regarding the involvement of informal caregivers during the hospital stay, as they are important partners in healthcare.[10]

Second, the questionnaire was kept short because it should not be a burden and feasible for frail patients to complete. Nonetheless, some of the questions had a high nonresponse rate, and many patients answered I don't know, particularly to the questions 2 and 3. It does not necessarily mean that these questions are poor in quality; it could also indicate that offering individualized care is not yet embedded in the culture of elderly patients and care professionals, such that patients consider such questions to be irrelevant.[11, 12] Nevertheless, we suggest to further explore the feasibility of the questionnaire and potential additional methods for the most frail elderly,[13] who might have been excluded from the CWH study sample at this point (Table 2).

Third, the questionnaire measures experiences rather than satisfaction. Patient‐satisfaction scores are generally tightly correlated with the age, sex, education level, health status, and the person completing the questionnaire.[8] In our study, the correlation did not reach statistical significance. Nevertheless, the achievement of preset goals was correlated significantly with mean CWH scores (Table 4). These findings may indicate that individualized care experiences can indeed be assessed better using this questionnaire. Test‐retest reliability also supports validity, as we expectedand, indeed, sawhigher reliability among the more objective questions (eg, question 8). The most valuing question is question 1, which also had the lowest reliability; the word sufficiently should perhaps be removed in the next version in order to increase its reliability and objectivity.

Finally, scores did not differ between before and after implementation of the CWH program, which suggests either that the questionnaire is unable to detect change or that the program was not sufficiently effective to invoke change yet. The latter option seems plausible, as changes in the provision of individualized care were ongoing. In addition, the items on which favorable differences can be seen for CWH are in fact the items that could be most directly influenced by the CWH interventionists, questions 4, 6, and 7 (see Supporting Information, Appendix C, in the online version of this article). Lastly, we performed an extra analysis concerning the discriminating property of the questionnaire in a subgroup of frail elderly patients; we do see a significant difference in scores between the frail patients in the geriatrics department and the frail patients who received the CWH intervention: 6.8 (n = 88) vs 4.8 (n = 13) for complete data, respectively, P = 0.013; and 6.8 (n = 155) vs 5.7 (n = 37) for incomplete data (2 items missing), P = 0.017 (Mann‐Whitney U test). This may indicate that the questionnaire can measure differences in quality of care for specifically the frail elderly patients between departments. However, these issuesincluding validity and reliability characteristics per specific patient subgroupwarrant further research using a larger sample.

CONCLUSIONS

In conclusion, the CareWell in Hospital patient questionnaire is a feasible and reliable tool for assessing experiences of frail elderly inpatients in the provision of individualized, integrated care. To improve the questionnaire, we recommend to add a question regarding the participation of informal caregivers during the hospital stay, investigate the response rate to questions regarding participation and shared decision‐making, and study responsiveness issues further.

Acknowledgements

The authors thank Gerda van Straaten, Anne Kuijpers, and Thijs Cauven for their support with data collection. We thank all members of the ZOWEL Study Group and the panel representing the elderly target group.

Disclosures: The work was made possible by grant 60‐6190‐098‐272 and grant 60‐61900‐98‐129 of the National Programme for Elderly Care, coordinated and sponsored by ZonMw, The Netherlands, Organization of Health Research and Development. The authors report no conflicts of interest.

Patient‐reported quality of care is currently an important outcome measure. Ideally, quality of care is assessed by measuring patient's experiences rather than patient satisfaction, as most patients are satisfied with the care they receive, even if the quality is poor.[1] Within the study of the CareWell in Hospital (CWH) program[2]which aims to improve quality of care for frail inpatients age 70 yearswe aimed to assess experiences using a questionnaire to determine the quality of hospital care from the perspective of elderly inpatients. This questionnaire should specifically address whether individualized, integrated care is delivered, with an emphasis on autonomy and maintaining patient independence as well as integrating well‐being into hospital care, all of which are aims of the CWH program. In this, it follows the perspective of integrated care as enabling the achievement of common goals and optimal care results from the patients' view: Care should be sensitive to the characteristics and needs of individual patients.[3]

In the Netherlands, a patient questionnaire to measure experiences with hospital care was carefully developed (partially based on the Consumer Assessment of Healthcare Providers and Systems) and is used to obtain information for national benchmarking: the Consumer Quality Index (CQI).[4] However, we considered this questionnaire containing 78 core questions as well as the time between discharge and measurement (often several months) too long for frail elderly patients, as they have complex, multidisciplinary needs and may have difficulty communicating their needs and reporting their experienced quality of care.

Here, we report on the development and validation of a questionnaire that is based on the CQI and can be used to measure the quality of individualized and integrated hospital care as experienced by inpatients age 70 years.

METHODS

Development

The predefined criteria for the questionnaire were that it should be brief, thereby reducing the burden placed on frail elderly persons; cover the aims of CWH; and measure experiences rather than satisfaction.

Ten categories were initially formulated to match CWH's goals of autonomy, independence, well‐being, individualized care, communication, coordination of care, continuity of care, patient safety, and competence of physicians and nurses. Items from the CQI questionnaire database[5] were selected for each category. Ten members of a panel representing the elderly target group were invited to select the 3 most important questions in each category (first Delphi round). This panel is an important party within a regional network of care and well‐being organizations and involved in discussing the various regional care and/or well‐being projects when it concerns their content and value for elderly persons. They represent elderly persons through their position in elderly‐care or informal care organizations or from personal experiences. During a second Delphi round, they determined whether the individual items of the concept questionnaire were clearly stated, comprehensible to frail elderly patients, represent quality of care, have appropriate answer categories, and so forth. The final questionnaire was edited to match the reading level of a 12‐year‐old and approved by the panel in a face‐to‐face meeting. By this process, content validity was ensured.[6]

Data Collection

The final questionnaire was mailed to both frail and nonfrail medical and surgical inpatients who were included in the CWH before‐after study (January 2011 to July 2012) 1 week after their discharge, by a research assistant (see Supporting Information, Appendix A, in the online version of this article for a description of the study and CWH program).

Patients in the CWH study who returned the questionnaire during the postimplementation measurement period were asked to participate in the test‐retest reliability study until a predetermined sample size of 75 was reached (March 2012 to November 2012). The target interval between returning the first and second questionnaire was 2 to 14 days.[7]

In addition, patients admitted to the geriatrics departmentand therefore assumed to be frailreceived the questionnaire upon discharge (February 2012 to April 2013). The geriatrics department administered the questionnaire anonymously for evaluation and quality‐improvement purposes, as part of usual care. The secretary included the questionnaire in all patient files, and a nurse provided the questionnaire to patients together with other important discharge documents. This questionnaire also included a question regarding goal attainment, as this reflects whether what is important to the most frail elderly patients was accomplished.

Validation and Analysis

Data were analyzed using the statistical software program SPSS version 18.0 (SPSS Inc., Chicago, IL.).

Data

Characteristics of (non)responders, levels of missing data, and measurement range were assessed using descriptive statistics.

Reliability

Internal consistency was assessed by calculating Cronbach's for all available questionnaires with complete data. The answer categories were recoded to a 010 scale; 10 represents the highest quality of care. Test‐retest reliability[6] was assessed by calculating Cohen's for individual questions and intraclass correlation (ICC) for the questionnaire's mean score.

Validity

The following hypotheses were tested in order to assess construct validity: lower scores for female patients[8] and for patients who rate their health lower,[9] and with higher education[8, 9]; higher scores for patients who had an elective admission[8] and whose treatment goals were achieved (own reasoning). Finally, whether patients answered the questionnaire independently or with help should not affect scores (own reasoning). The Spearman was calculated for nonparametric and ordinal data.

In addition, we performed a Kruskal‐Wallis analysis to test the hypothesis that patients admitted to different departments have different scores. Second, we used the Mann‐Whitney U test to detect differences before and after implementation of the CWH program.

For all these analyses, only questionnaires with complete data were included.

RESULTS

Development

The selected answers within the categories communication and competence of nurses and physicians by the panel overlapped with questions from the other 8 categories; thus, the final questionnaire contains 8 core questions (Table 1) (see Supporting Information, Appendix B, in the online version of this article).

Table 1.The 8 Core Questions of the CareWell in Hospital Questionnaire
Question
NOTE: The questionnaire for the geriatrics department included 1 additional question: Within a few days of your hospital admission, a doctor discussed the goal of the admission with you. Did you achieve your goal(s) satisfactorily? (no, not at all; yes, partially; yes, completely; don't know; doctor did not discuss my goals). See Supporting Information, Appendix A, in the online version of this article for the entire questionnaire, including the answer categories.
1. Were you informed sufficiently by your doctor regarding the various options for treating your health problems?
2. Were you able to indicate which treatment and/or care you preferred?
3. During your hospital stay, could you co‐decide what was important to your care?
4. During your hospital stay, were you supported in keeping busy and finding social contacts and activities?
5. Did you know to whom you can go within the hospital with questions, problems, or complaints?
6. Before discharge, did you talk with a member of the hospital staff regarding the care you would need after discharge?
7. Did a member of the hospital staff inform the key people and/or care providers of your discharge from the hospital?
8. During your hospital stay, did you experience 1 or more of the following events?
Did you fall?
Did you become confused?
Did you develop pressure ulcers?
Did medication errors occur?
Did you develop a urinary tract infection?
Did you develop a wound infection?
Did you experience complications with your surgery and/or treatment?

Data Collection

Figure 1 shows a flowchart of the questionnaires.

Table 2 presents data of responders compared with nonresponders who were included in the CWH study (N = 293). Patients were age 70 years and admitted 48 hours. Patients responded 14.8 11.3 days after discharge (n = 265). Response rate was 75.8%. From 18 responders no baseline characteristics were available, as only the questionnaire was collected from them to reach n = 75 for test‐retest purposes.

Table 2.Characteristics of the Responding (n = 293) and Nonresponding (n = 88) Patients Included in the CareWell in Hospital Before‐After Study
	No.	Responders	No.	Nonresponders	P Value
NOTE: Data on baseline characteristics from 18 patients in the post‐CWH measurement period are missing, and from those patients only the CareWell in Hospital questionnaires were gathered in order to reach n = 75 for test‐retest purposes. CIRS‐G ranging from 0 to 56 (with a higher score indicating more comorbidity).[14] MMSE ranging from 0 to 30 (with 30 representing the best score). Length of stay is defined as the time between admission to a CWH study department and discharge from a CWH study department. Abbreviations: CIRS‐G, Cumulative Illness Rating ScaleGeriatrics; CWH, CareWell in Hospital; MMSE, Mini‐Mental State Examination; SD, standard deviation.
Age, y SD	275	76.9 5.2	88	77.3 5.5	0.701
Male sex, n (%)	275	156 (56.7)	88	52 (59.1)	0.696
CIRS‐G, score SD	274	12.8 5.0	88	13.9 5.0	0.071
MMSE admission, score SD	264	26.7 3.7	82	25.1 4.8	0.001
MMSE discharge, score SD	230	26.9 3.7	66	25.8 4.4	0.026
Length of stay, days SD	275	8.2 7.4	88	9.6 9.7	0.322
Department, surgical (%)	275	170 (61.8)	88	56 (63.6)	0.759
Admission type, n (%)	275		88		0.343
Emergency		82 (29.8)		22 (25.0)
Elective		138 (50.2)		52 (59.1)
From other hospital or other department		55 (20.0)		14 (15.9)
Marital status, alone (%)	273	187 (68.5)	84	50 (59.5)	0.128
Discharge destination, n (%)	275		88		0.000
Home		197 (71.6)		54 (61.4)
Other hospital		69 (25.1)		20 (22.7)
Care facility		9 (3.3)		14 (15.9)
Readmission, n (%)	275	38 (13.8)	88	7 (8.0)	0.146
Readmission <1 mo, n (%)	275	28 (10.2)	88	14 (15.9)	0.144
Death <3 mo following discharge, n (%)	274	9 (3.3)	86	5 (5.8)	0.233
Received CWH intervention	149	43 (28.9)	33	15 (45.5)	0.064

Patients in the geriatrics department responded in 10.5 15.0 days (n = 111). Mean length of stay was 9.0 7.2 days (n = 116). Data regarding other baseline characteristics and response rate were unavailable due to privacy concerns.

Data Characteristics

Table 3 summarizes data of all 470 questionnaires. Response rates to the answer options ranged from 3.8% to 66.8%. Missing data among the questions ranged from 1.7% within question 8 to 7.0% within question 4. Upon combining the answer categories I don't know and missing, 7/8 questions had >10% missing data; the questions 2 and 3 had the highest percentage of missing data due to the I don't know answer option. The reasons stated by the respondents for why they could not answer these questions included cognitive disabilities; the perception that, because there was only one option (eg, in case of emergency admissions), the question did not apply to them; and/or that the patients preferred not to co‐decide because they felt that the physician knows best and can decide what is best.

Table 3.Data Quality and Range and Test‐Retest Reliability of All Questionnaires Received
	Data (n = 470)		Test‐Retest (n = 78)
	No.	%	No.
NOTE: For adverse events, the minimum amount of missing data was 1.7%. Sum scores range from 0 to 80. Mean scores range from 0 to 10. = Cohen's . Abbreviations: DK, don't know; ICC, intraclass correlation coefficient; Max, maximum; MIS, missing.
Sufficiently informed regarding treatment options			65	0.278
Not at all	23	4.9
Sometimes	90	19.1
Often	115	24.5
Every time	191	40.6
Don't know	29	6.2
Missing	21	4.7
Treatment and care preferences discussed			59	0.415
Not at all	89	18.9
Sometimes	78	16.6
Often	61	13.0
Every time	111	23.6
Don't know	103	21.9
Missing	28	6.0
Co‐decide regarding important issues			56	0.295
Not at all	75	16.0
Sometimes	86	18.3
Often	67	14.3
Every time	112	23.8
Don't know	98	20.9
Missing	32	6.8
Supported in finding (social) activities			73	0.533
Not at all	72	15.3
A little	66	14.0
Good	109	23.2
Very good	36	7.7
Not applicable	130	27.7
Don't know	24	5.1
Missing	33	7.0
Knows relevant person for questions, problems, complaints			77	0.652
Yes	279	59.4
No	107	22.8
Don't know	67	14.3
Missing	17	3.6
Discussed postdischarge care needs			75	0.574
Yes, sufficient	311	66.2
Yes, but insufficient	26	5.5
No	99	20.3
I don't know/I don't remember	18	3.8
Missing	19	4.0
Hospital informed other important people/providers of discharge			69	0.405
No	45	9.6
Some were informed	54	11.5
Yes	314	66.8
Don't know	38	8.1
Missing	19	4.0
Adverse events during hospital admission	DK	MIS	78	0.816
Fall, confusion, pressure ulcer, medication error, bladder infection, wound infection, complication of surgery/treatment	Max 9.1%	Max 4.3%
	Sum	Mean	No.	ICC
Mean score on the total questionnaire, complete cases (n = 222)	51.9 18.3	6.5 2.3	39	0.745

Reliability

Of the 470 questionnaires, 222 (47.2%) had complete data and were used to analyze internal consistency. Cronbach's for the 8‐item questionnaire was 0.70 (good internal consistency).

Seventy‐eight questionnaires were available to measure test‐retest reliability. The interval between test‐retest was 8.7 4.8 days; 94.7% was returned within the targeted 14 days. Thirty‐eight patients had complete data for both measurements: ICC on the mean score of the questionnaire was 0.75 (95% confidence interval [CI]: 0.56‐0.86), which indicates good test‐retest reliability (Table 3). Including patients with incomplete data (1 to 2 missing items) yielded an ICC >0.70. Among the individual questions, Cohen's ranged from 0.28 to 0.82.

Validity

The mean questionnaire score was significantly correlated with goals achieved while hospitalized (Table 4).

Table 4.Construct Validity of the CareWell in Hospital Questionnaire Based on All Questionnaires With Complete Data on Both the Variable and the Questionnaire Score
Variable	Response	No.a	Score SD	Correlation
NOTE: Mean scores range from 0 to 10. Abbreviations: F, female; M, male; SD, standard deviation. a The number differs per analysis. Education level was not known for every patient; this variable was extracted from a different questionnaire. Admission type includes only emergency admission and elective admission; patients could also be transferred from another department or hospital, but this was not included as a category as this might include emergency as well as elective admissions. Goal of admission was only available for patients from the geriatrics department, whereas educational level and admission type were not available for patients from the geriatrics department. b Correlation (Spearman ) is significant at the 0.01 level (1‐tailed for goal achieved).
Sex	M	114	6.3 2.3	0.080
	F	108	6.7 2.3
Health status	Excellent	1		0.071
	Very good	5	7.9 2.0
	Good	52	6.7 2.4
	Fair	120	6.5 2.2
	Poor	28	6.2 2.1
Education level	<6 grades primary school	4	4.9 1.2	0.068
	Primary school	19	6.4 2.5
	Higher than primary school	6	7.6 1.2
	Practical training	27	6.0 2.2
	Secondary vocational training	41	6.1 2.5
	Pre‐university education	2	7.2 4.0
	University/higher education	20	6.8 2.2
Admission type	Emergency	31	6.5 2.6	0.015
	Elective	61	6.6 2.0
Goal of admission achieved	Yes	33	7.6 1.7	0.319b
Goal of admission achieved	Partially	24	6.6 2.1
	No	6	4.7 2.8
Respondent	Patient only	117	6.7 2.2	0.063
	Patient with help	59	5.9 2.3
	Other person	41	6.7 2.4

Mean scores did not differ significantly between departments (geriatrics: 6.8 2.2, n = 88; cardiothoracic surgery and lung diseases: 6.5 2.4, n = 54; internal medicine: 6.3 2.5, n = 30; general surgery: 6.0 2.2, n = 50; P = 0.234).

In addition, mean scores did not differ significantly before (6.5 2.2, n = 53) and after (6.1 2.4, n = 67) implementation of the CWH study (P = 0.320).

DISCUSSION

The CareWell in Hospital patient questionnaire is a brief 8‐item questionnaire to assess the experiences of elderly patients regarding integrated hospital care. It showed good internal consistency and test‐retest reliability, and low responsiveness. Here we discuss some issues related to the preset criteria of the questionnaire.

First, a panel representing the elderly target population was used to develop the questionnaire in order to ensure content validity, which was confirmed by good internal consistency. Yet, with respect to individualized, integrated care for frail elderly patients, we recommend including a question regarding the involvement of informal caregivers during the hospital stay, as they are important partners in healthcare.[10]

Second, the questionnaire was kept short because it should not be a burden and feasible for frail patients to complete. Nonetheless, some of the questions had a high nonresponse rate, and many patients answered I don't know, particularly to the questions 2 and 3. It does not necessarily mean that these questions are poor in quality; it could also indicate that offering individualized care is not yet embedded in the culture of elderly patients and care professionals, such that patients consider such questions to be irrelevant.[11, 12] Nevertheless, we suggest to further explore the feasibility of the questionnaire and potential additional methods for the most frail elderly,[13] who might have been excluded from the CWH study sample at this point (Table 2).

Third, the questionnaire measures experiences rather than satisfaction. Patient‐satisfaction scores are generally tightly correlated with the age, sex, education level, health status, and the person completing the questionnaire.[8] In our study, the correlation did not reach statistical significance. Nevertheless, the achievement of preset goals was correlated significantly with mean CWH scores (Table 4). These findings may indicate that individualized care experiences can indeed be assessed better using this questionnaire. Test‐retest reliability also supports validity, as we expectedand, indeed, sawhigher reliability among the more objective questions (eg, question 8). The most valuing question is question 1, which also had the lowest reliability; the word sufficiently should perhaps be removed in the next version in order to increase its reliability and objectivity.

Finally, scores did not differ between before and after implementation of the CWH program, which suggests either that the questionnaire is unable to detect change or that the program was not sufficiently effective to invoke change yet. The latter option seems plausible, as changes in the provision of individualized care were ongoing. In addition, the items on which favorable differences can be seen for CWH are in fact the items that could be most directly influenced by the CWH interventionists, questions 4, 6, and 7 (see Supporting Information, Appendix C, in the online version of this article). Lastly, we performed an extra analysis concerning the discriminating property of the questionnaire in a subgroup of frail elderly patients; we do see a significant difference in scores between the frail patients in the geriatrics department and the frail patients who received the CWH intervention: 6.8 (n = 88) vs 4.8 (n = 13) for complete data, respectively, P = 0.013; and 6.8 (n = 155) vs 5.7 (n = 37) for incomplete data (2 items missing), P = 0.017 (Mann‐Whitney U test). This may indicate that the questionnaire can measure differences in quality of care for specifically the frail elderly patients between departments. However, these issuesincluding validity and reliability characteristics per specific patient subgroupwarrant further research using a larger sample.

CONCLUSIONS

In conclusion, the CareWell in Hospital patient questionnaire is a feasible and reliable tool for assessing experiences of frail elderly inpatients in the provision of individualized, integrated care. To improve the questionnaire, we recommend to add a question regarding the participation of informal caregivers during the hospital stay, investigate the response rate to questions regarding participation and shared decision‐making, and study responsiveness issues further.

Acknowledgements

The authors thank Gerda van Straaten, Anne Kuijpers, and Thijs Cauven for their support with data collection. We thank all members of the ZOWEL Study Group and the panel representing the elderly target group.

Disclosures: The work was made possible by grant 60‐6190‐098‐272 and grant 60‐61900‐98‐129 of the National Programme for Elderly Care, coordinated and sponsored by ZonMw, The Netherlands, Organization of Health Research and Development. The authors report no conflicts of interest.

References

Kalucy L, Katterl R, Jackson‐Bowers E. Patient Experience of Health Care Performance. Adelaide, Australia: Primary Health Care Research November 2009. Available at: http://dspace.flinders.edu.au/jspui/bitstream/2328/26594/1/PIR NOV 09 Full.pdf.
Bakker FC, Persoon A, Schoon Y, Rikkert MGM. Hospital Elder Life Program integrated in Dutch hospital care: a pilot study. J Am Geriatr Soc. 2013;61(4):641–642.
Kodner DL, Spreeuwenberg C. Integrated care: meaning, logic, applications, and implications—a discussion paper. Int J Integr Care. 2002;2:e12.
Sixma H, Spreeuwenberg P, Zuidgeest M, Rademakers J. CQ‐index Ziekenhuisopname: meetinstrumentontwikkeling. Kwaliteit van de zorg tijdens ziekenhuisopnames vanuit het perspectief van patiënten. De ontwikkeling van het instrument, de psychometrische eigenschappen en het discriminerend vermogen [in Dutch]. Utrecht, The Netherlands: NIVEL (Netherlands Institute for Health Services Research), 2009.
Centrum Klantervaring Zorg.CQI vragenbank (CQI questionnaire database). Available at: http://nvl002.nivel.nl/CQI. Accessed May–June 2010.
Terwee CB, Bot SD, Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.
Streiner D, Norman G. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th ed. Oxford, UK: Oxford University Press; 2008:182–183.
Hordacre AL, Taylor A, Pirone C, Adams RJ. Assessing patient satisfaction: implications for South Australian public hospitals. Aust Health Rev. 2005;29(4):439–446.
Hekkert KD, Cihangir S, Kleefstra SM, Berg B, Kool RB. Patient satisfaction revisited: a multilevel approach. Soc Sci Med. 2009;69(1):68–75.
Wressle E, Eriksson L, Fahlander A, et al. Relatives' perspective on the quality of geriatric care and rehabilitation—development and testing of a questionnaire. Scand J Caring Sci. 2008;22(4):590–595.
Ekdahl AW, Andersson L, Wiréhn AB, Friedrichsen M. Are elderly people with co‐morbidities involved adequately in medical decision making when hospitalised? A cross‐sectional survey. BMC Geriatr. 2011;11:46.
Wilkinson C, Khanji M, Cotter PE, Dunne O, O'Keeffe ST. Preferences of acutely ill patients for participation in medical decision‐making. Qual Saf Health Care. 2008;17(2):97–100.
Goldberg SE, Harwood RH. Experience of general hospital care in older patients with cognitive impairment: are we measuring the most vulnerable patients' experience? BMJ Qual Saf. 2013;doi:10.1136/bmjqs‐2013‐001961.
Miller M, Towers A. A Manual of Guidelines for Scoring the Cumulative Illness Rating Scale for Geriatrics (CIRS‐G). Pittsburgh, PA: University of Pittsburgh School of Medicine, Department of Geriatric Psychiatry; 1991.

References

Kalucy L, Katterl R, Jackson‐Bowers E. Patient Experience of Health Care Performance. Adelaide, Australia: Primary Health Care Research November 2009. Available at: http://dspace.flinders.edu.au/jspui/bitstream/2328/26594/1/PIR NOV 09 Full.pdf.
Bakker FC, Persoon A, Schoon Y, Rikkert MGM. Hospital Elder Life Program integrated in Dutch hospital care: a pilot study. J Am Geriatr Soc. 2013;61(4):641–642.
Kodner DL, Spreeuwenberg C. Integrated care: meaning, logic, applications, and implications—a discussion paper. Int J Integr Care. 2002;2:e12.
Sixma H, Spreeuwenberg P, Zuidgeest M, Rademakers J. CQ‐index Ziekenhuisopname: meetinstrumentontwikkeling. Kwaliteit van de zorg tijdens ziekenhuisopnames vanuit het perspectief van patiënten. De ontwikkeling van het instrument, de psychometrische eigenschappen en het discriminerend vermogen [in Dutch]. Utrecht, The Netherlands: NIVEL (Netherlands Institute for Health Services Research), 2009.
Centrum Klantervaring Zorg.CQI vragenbank (CQI questionnaire database). Available at: http://nvl002.nivel.nl/CQI. Accessed May–June 2010.
Terwee CB, Bot SD, Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.
Streiner D, Norman G. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th ed. Oxford, UK: Oxford University Press; 2008:182–183.
Hordacre AL, Taylor A, Pirone C, Adams RJ. Assessing patient satisfaction: implications for South Australian public hospitals. Aust Health Rev. 2005;29(4):439–446.
Hekkert KD, Cihangir S, Kleefstra SM, Berg B, Kool RB. Patient satisfaction revisited: a multilevel approach. Soc Sci Med. 2009;69(1):68–75.
Wressle E, Eriksson L, Fahlander A, et al. Relatives' perspective on the quality of geriatric care and rehabilitation—development and testing of a questionnaire. Scand J Caring Sci. 2008;22(4):590–595.
Ekdahl AW, Andersson L, Wiréhn AB, Friedrichsen M. Are elderly people with co‐morbidities involved adequately in medical decision making when hospitalised? A cross‐sectional survey. BMC Geriatr. 2011;11:46.
Wilkinson C, Khanji M, Cotter PE, Dunne O, O'Keeffe ST. Preferences of acutely ill patients for participation in medical decision‐making. Qual Saf Health Care. 2008;17(2):97–100.
Goldberg SE, Harwood RH. Experience of general hospital care in older patients with cognitive impairment: are we measuring the most vulnerable patients' experience? BMJ Qual Saf. 2013;doi:10.1136/bmjqs‐2013‐001961.
Miller M, Towers A. A Manual of Guidelines for Scoring the Cumulative Illness Rating Scale for Geriatrics (CIRS‐G). Pittsburgh, PA: University of Pittsburgh School of Medicine, Department of Geriatric Psychiatry; 1991.

Issue

Journal of Hospital Medicine - 9(5)

Issue

Journal of Hospital Medicine - 9(5)

Page Number

324-329

Page Number

324-329

Article Type

Article

Display Headline

The carewell in hospital questionnaire: A measure of frail elderly inpatient experiences with individualized and integrated hospital care

Display Headline

The carewell in hospital questionnaire: A measure of frail elderly inpatient experiences with individualized and integrated hospital care

Sections

Brief Reports

Article Source

Disallow All Ads

Corresponding Author

Correspondence Location

Address for correspondence and reprint requests: Franka Bakker, MSc, Department of Geriatric Medicine 925, Radboud University Medical Center, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands; Telephone: +31 24 3616772; Fax: +31 24 3617408; E‐mail: [email protected]

Content Gating

Gated (full article locked unless allowed per User)

Gating Strategy

First Peek Free

Teaser Media

Article PDF Media

Media Files