Quality of acute internal medicine: A patient-centered approach. Validation and usage of the Patient Reported Measure-acute care in the Netherlands

Abstract

Background

Providing high quality care is important and has gained more attention since the introduction of value-based healthcare. Value should be measured by outcomes achieved, relevant for patients. Patient-centeredness is one domain for quality improvement determined by the Institute of Medicine, aiming to deliver care responsive to the patient. The development and implementation of patient reported outcome- and experience measures can be used for this goal. Recently, we developed the Patient Reported Measure (PRM)-acute care, based on five relevant domains to evaluate and improve the quality of care in the Emergency Department (ED).

Objective

To validate the PRM-acute care, in order to evaluate and improve patient-centered care in the ED.

Methods

We performed a prospective questionnaire-based study. Patients ≥18 years presenting for internal medicine in the ED were eligible. The validity of the PRM-acute care was evaluated according to the COSMIN-criteria. We performed hypotheses testing to evaluate construct validity. The perceived quality of care was evaluated by statistical analysis.

Results

Face- and content validity was evaluated based on previously performed research and deemed good. Construct validity was supported by demonstrated differences between subgroups; patients with severe symptoms had a higher perceived quality of care. The correlation between overall satisfaction and the total mean score of the PRM-acute care (r = 0,447, p = 0.01) was significant. Overall, patients reported a mean perceived quality of care of 4.67/6.0.

Conclusion

The PRM-acute care is a valid instrument to measure the perceived quality of care in an acute setting for internal medicine patients. Additionally, patients reported a good perceived quality of care in the ED with scores ranging from moderate to well for each of the relevant domains. Therefore, we believe that the PRM-acute care can be implemented in daily practice to evaluate the perceived quality of care and to improve the quality of acute care.

About the Authors:

Marjolein N. T. Kremers

Roles Conceptualization, Formal analysis, Methodology, Project administration, Validation, Writing – original draft

* E-mail: [email protected]

Affiliations Department of Health Services Research, and CAPHRI School for Public Health and Primary Care, Aging and Long Term Care, Maastricht, The Netherlands, Department of Internal Medicine, Máxima MC, Veldhoven/Eindhoven, The Netherlands

ORCID logo https://orcid.org/0000-0002-3791-8961

Elsemieke E. M. Mols

Roles Data curation, Formal analysis, Validation, Writing – review & editing

Affiliation: Department of Internal Medicine, Máxima MC, Veldhoven/Eindhoven, The Netherlands

Yvonne A. E. Simons

Roles Data curation, Formal analysis, Validation, Writing – review & editing

Affiliation: Department of Internal Medicine, Máxima MC, Veldhoven/Eindhoven, The Netherlands

Sander M. J. van Kuijk

Roles Formal analysis, Methodology, Validation, Writing – review & editing

Affiliation: Department of KEMTA, Maastricht University Medical Center+, Maastricht, The Netherlands

Frits Holleman

Roles Conceptualization, Supervision, Writing – review & editing

Affiliation: Department of Internal Medicine, Amsterdam UMC, Location Academic Medical Center, Amsterdam, The Netherlands

Prabath W. B. Nanayakkara

Roles Conceptualization, Methodology, Supervision, Writing – review & editing

Affiliation: Section General and Acute Internal Medicine, Amsterdam Public Health Research Institute, Department of Internal Medicine, Amsterdam UMC, Location VU University Medical Center, Amsterdam, The Netherlands

Harm R. Haak

Roles Conceptualization, Methodology, Supervision, Writing – review & editing

Affiliations Department of Health Services Research, and CAPHRI School for Public Health and Primary Care, Aging and Long Term Care, Maastricht, The Netherlands, Department of Internal Medicine, Máxima MC, Veldhoven/Eindhoven, The Netherlands, Department of Internal Medicine, Division of General Internal Medicine, Maastricht University Medical Center+, Maastricht, The Netherlands

Introduction

The increasing attention that providing high quality care receives since the introduction of value-based healthcare creates a need for transparency of healthcare quality [1]. Value in health care should always be defined around the customer in a well-functioning care system, therefore value is measured by the outcomes achieved that are relevant for patients [2]. The Institute of Medicine determined patient-centeredness, defined as providing care that is respectful of and responsive to individual patient preferences, needs and values, and ensuring that patients`values guide all clinical decisions, to be one of six domains for measuring and improving healthcare quality [3].

Patient-centeredness is important for all healthcare domains, including the field of acute medicine. The emergency department (ED) is a busy environment characterised by rapid triage, acute conditions and a high turnover. As a consequence, an ED visit can be considered as a stressful life event and associated with adverse effects on the patient’s emotional state [4]. It is important that patients in the ED receive high-quality care and experience it as such. A way to assess the perceived quality of care by patients is the routine use of Patient-Reported Measures (PRMs), which consist of measures of satisfaction with outcomes of care and measures of experiences of care [5, 6]. So, Patient-Reported Outcomes (PROs) and patients perceptions of experiences whilst receiving care, known as Patient-Reported Experiences (PREs), are combined in PRMs. PROs and PREs are directly obtained from the patient without the interference of a clinician and pertain to the patients’ health, quality of life, and functional status associated with healthcare [7].

Many Patient-Reported Outcome Measures have been developed over the past years, being either generic, disease specific or patient specific. Their measurement properties often restrict successful use in different settings. Since the patient population in the ED is heterogeneous and often lacks a diagnosis, many disease and patient specific PROMs are not suitable for use in this setting [8]. In particular, patients presenting for internal medicine often suffer from multiple chronic conditions and present with non-specific complaints, which may explain why commonly used indicators do not reflect relevant outcomes for this specific group of patients [9]. In a previous study, we determined the PROs relevant to internal medicine patients in an acute setting. Five core domains were identified, namely relief of symptoms, understanding the diagnosis, understanding the treatment plan, reassurance and patient experiences [9]. Experiences were determined as a relevant domain, because in the patients’ perception of quality of care, both outcomes and experiences play an important role. While researchers and doctors try to distinguish Patient Reported Outcomes from Patient Reported Experiences, patients do not. So, based on these five domains, we developed a PRM-acute care in order to assess the perceived quality of acute care for internal medicine patients.

Acute care in the Netherlands is mainly provided by General Practitioners (GPs) in primary care and via EDs. To gain access to hospital care, including EDs, patients are required to have a referral from a GP or be directly transferred by an ambulance [10]. Due to increasing numbers of patients presenting in the ED with a higher age, comorbidity and therefore often a greater case complexity, the speciality acute medicine was recognised in 2012 as a subspecialty of internal medicine. The number of patients presenting for internal medicine increased slightly from 2013 to 2016. In 2013, on average 3,824 patients (range: 1,227–10,403) presented per hospital to the ED for internal medicine, compared to 4,343 patients (range: 1,418–29,426) in 2016 [11].

In this study, we primarily aim to assess the validity of the PRM-acute care in internal medicine patients and secondly, to gain insight into the current perceived quality of acute care, with the overarching goal to use the PRM-acute care in daily practice and improve patient-centered care in the ED.

Methods

Study design

We performed a prospective questionnaire-based study in two hospitals in the Netherlands (Máxima MC, Veldhoven and Amsterdam UMC, location VUmc), as part of the PROMESQUE trial. Within this trial, a baseline measurement and a consecutive intervention study was planned. This study concerns the baseline measurement. The study protocol was approved by the Medical Ethics Committees of the participating hospitals.

Selection of participants

Patients were included between January 5 and March 12, 2020. Due to regulations during the COVID-19 crisis, the inclusion of patients was terminated in both hospitals. All patients ≥18 years presenting for internal medicine in the ED were eligible for inclusion. Patients were asked by their treating physician to participate in the study. Patients who were unable to participate due to a language barrier, inability to understand the questionnaire, or severity of illness, were excluded at the discretion of the treating physician. Patients willing to participate were approached by a researcher and written informed consent was obtained. The PRM-acute care was presented to participants during admission or at home by phone, 12 to 72 hours after arrival in the ED. This timeframe was selected with the planned intervention study in mind, because the presence of researchers in the ED and visible measurements might influence the daily practice in the ED (Hawthorne effect) and as a consequence the perceived quality of care.

Development of the PRM-acute care

The development of the PRM-acute care was based on the previously determined five relevant domains for patients presenting for internal medicine in the ED, namely relief of symptoms, understanding the diagnosis, understanding the therapeutic plan, reassurance, and patient experiences [9]. A concept questionnaire was developed and presented in three focus groups to 16 different experts in the field of acute medicine by a trained interviewer (MK) to explore the face- and content validity. Thereafter, 15 patients were interviewed to identify face- and content validity of the questionnaire and subsequently a cognitive interview was performed by a researcher trained in qualitative interviewing (MK) in order to ensure that the questions were considered relevant and understood by patients, and to determine whether each question generated the information intended by the researchers. Patients were selected by purpose sampling. For all interviews a topic list was established and audio records were made. As a result, the PRM-acute care was finalised and consisted of 11 questions covering the five domains (Questionnaire in Dutch in S1 Appendix, Questionnaire in English in S2 Appendix). Answers to questions were scored on a Likert scale with a range of 1 to 6, except questions concerning symptoms, these were scored on a Numeric Rating Scale (NRS) with a range from 0 to 10.

The scoring model for the PRM-acute care was based on qualitative interviews with 10 patients assessing the importance of the five domains, which resulted in a total score consisting of the mean of all the reported domains. A maximum of one missing domain was accepted for analysing the questionnaire. The domain score was calculated as the mean score of the questions of that domain. One missing question per domain was accepted. Domain scores were defined as missing when the domain had more than one missing question, or if the domain score could not be calculated (in the case of domain ‘relief of symptoms’). Most domain scores were adopted from the results on the Likert-scales ranging 1–6. However, two exceptions were made. Firstly, within the domain ‘understanding the diagnosis’ a grade 0 was given when a patient reported not receiving any explanation about their diagnosis. Secondly, to establish the score of the domain ‘symptom relief’, we calculated the percentage difference between the severity of symptoms at arrival and discharge from the ED, and converted these into a 1–6 score, as is shown in the table in S1 Table. This grading system was based on an expert discussion and on literature, which defined the minimum clinically important difference in acute pain as 30% [12, 13].

Measurements

Perceived quality of care.

Participants were asked to recall their ED visit and complete the PRM-acute care. Hospitalised patients completed the PRM-acute care on paper, preferably by themselves. If necessary, caregivers or a researcher assisted without interfering in the interpretation of the patient. Patients at home received a link to the online PRM-acute care. Overall satisfaction was scored as a report mark, ranging from 1 to 10.

Visit and patient characteristics.

Destination after ED visit, arrival and discharge date and time were extracted from the electronic patient file. Length of stay in the ED (LOS-ED) was calculated. Baseline characteristics, such as gender, age, living situation and educational level were obtained from all included patients. During the first month of the study, baseline characteristics such as gender, age and destination after ED, were also collected from patients who were not included in the study. All data were stored in a web-based database (Castor EDC).

Data analysis

Validity testing.

The PRM-acute care is based on a formative model, as perceived quality of care is determined by the five relevant domains [14, 15]. The validation of the PRM-acute care was executed in accordance with the COSMIN-criteria applying to a formative model [14, 16–18]. Firstly, face and content validity was evaluated based on our previous study and strengthened by additional cognitive interviews in this study, as mentioned in ‘development of the PRM-acute care’. Face validity is the extent to which a test is subjectively viewed as covering the concept it purports to measure. Content validity refers to the extent to which a measure represents all facets of a given construct.

Furthermore, hypotheses testing was used to assess construct validity. Construct validity refers to the degree to which a measure actually measures the theory it purports. Hypotheses were proposed with the objective to demonstrate differences in scores between subgroups, which would establish construct validity. Previous research indicates that differences related to scores regarding quality of care are present [19]. Amongst others, a positive relationship was found between patient satisfaction of care and higher age, lesser education, trust in the medical care centre and good communication between patients and healthcare professionals [19–21]. Moreover, a negative correlation between the experience of pain and patient satisfaction is demonstrated [21, 22]. However, due to mandatory guidelines on pain management [23], early pain recognition has gained much attention in the Netherlands which leads to prompt treatment and patient satisfaction. Lastly, a negative correlation between LOS-ED and patient satisfaction, has been reported [24, 25]. Considering these findings, we propose several hypotheses regarding differences in perceived quality of care between subgroups, namely: 1) Older patients perceive a higher quality of care, 2) Patients with lower education (middle-level applied education or lower) perceive a higher quality of care, 3) Patients arriving in the ED with severe complaints (graded as 8–10), will perceive higher quality of care than patients with mild complaints (graded as 0–4), 4) Patients with a LOS-ED ≥ 4 hours, perceive lower quality of care than patients with a LOS-ED <4 hours. In addition, we explored whether differences in perceived quality of care existed between hospitalised and patients discharged directly from the ED and between patients presenting during weekdays or weekends. Moreover, while the PRM-acute care is based on a formative model, we expect that differences in perceived quality in subgroups may only be present in specific domains. Therefore, statistical analysis of differences between subgroups regarding domain scores were executed.

Lastly, with the aim of strengthening the construct, we analysed whether the total score of the PRM-acute care and the individual domains correlates with the overall satisfaction of the ED-care, graded using a report mark (range 1–10).

Statistical analysis

Patient and ED characteristics were analysed using descriptive statistics. Total- and domain scores were reported using the mean, standard deviation (SD) and 95% confidence interval (CI). To asses differences in the perceived quality of care between subgroups, unpaired T-tests and linear regression tests were used. To evaluate differences between subgroups on domain scores, the Mann-Whitney U test was used for the following domains: relief of symptoms, understanding the diagnosis, understanding the treatment plan and reassurance. An unpaired T-test was used to analyse the domain ‘experiences’. A Spearman’s rho was used to analyse the correlation between the overall satisfaction and relief of symptoms, understanding the diagnosis, understanding the treatment plan and reassurance. The correlation between the overall satisfaction and the domain ‘experiences’ was analysed using a Pearson’s rho. A p-value of 0.05 was considered significant. All analyses were performed using IBM SPSS version 26.0 for Windows.

Results

Patient characteristics

We included 81 patients, of which 47 were men (58%) as is shown in Table 1. The response rate, measured during the first three weeks of the study, was 86.6%. The mean age of the study sample was 68 years (range 26–93). All patients lived at home, 27 (33.3%) patients lived alone and 54 (66.7%) lived together. Of all patients, 57 patients (70.4%) were treated in MMC and 24 (29.6%) patients in A-UMC. Seventy patients (86.4%) were hospitalised after their ED visit and 11 patients (13.4%) were directly discharged from the ED. Seventy patients (86.4%) were seen during week-days, whereas 11 patients were seen during weekends (13.6%).

The study participants differed in gender and destination after ED visit compared to the patients who were not included. More men participated in the study (58% vs. 42%, p = 0.019) and more patients were hospitalised in the study group (86.4% vs. 13.6%, p <0.001). Patients were mostly not included because they were not asked by the treating physician to participate.

[Figure omitted. See PDF.]

Table 1. Patient characteristics.

https://doi.org/10.1371/journal.pone.0242603.t001

Face- and content validity.

Face- and content validity was partly established in our previous study [9]. In this study, professionals in acute care recognised all domains and questions as relevant. No new themes came up during the cognitive interviews with patients and all questions were deemed relevant. Minor adjustments in the questionnaire were made based on these interviews. Additionally, we observed that all five domains were equally important to patients.

Construct validity.

Patients experiencing severe symptoms had a higher mean total score as was shown by linear regression. The total mean score increased on average by 0.08 for each point increase in severity of symptoms (p = 0.006). The associations between age, gender, educational level, LOS-ED, discharge, and day of presentation with perceived quality were not statistically significant (Table 2). Subsequently, we evaluated differences between subgroups in each domain. Most differences were found among the subgroups in the domain ‘understanding the diagnosis’ as presented in Table 2. Patients who received less education had a greater perceived understanding of the diagnosis (mean 4.94, SD 1.39) than patients with a higher education (mean 4.14, SD 1.9) as shown by a Mann-Whitney U test (p = 0.01). Furthermore, patients with a LOS-ED < 4hours (mean 5.02, SD 1.41) scored on average higher in this domain than patients with a LOS-ED ≥ 4 hours (mean 4.18, SD 1.74, p = 0.003). Additionally, a linear regression model showed a significant association between the degree of understanding the diagnosis and the initial severity of symptoms (p = 0.005). Other significant differences were found in the domain ‘relief of symptoms’. Patients who experienced more severe symptoms on arrival in the ED, reported the biggest relief of symptoms. The domain score for relief of symptoms increases with 0.24 with each point increase on the NRS-scale, p = 0.001. Lastly, in the domain ‘patient experiences’ differences were found between admitted and discharged patients as admitted patients (mean 5.36, SD 0.53) reported higher scores than the discharged patients (mean 4.97, SD 0.45), p = 0.02.

[Figure omitted. See PDF.]

Table 2. Results of hypotheses testing on total score and separate domains.

https://doi.org/10.1371/journal.pone.0242603.t002

In order to strengthen the construct, the correlation between the mean total score of the perceived quality of care and the overall satisfaction of the ED-care was tested. The mean total score was correlated to the graded overall satisfaction as is shown in Fig 1 (r = 0.447, p = 0.01). Additionally, the scores of all domains except the domain ‘relief of symptoms’ were correlated with the overall satisfaction of ED-care, as analysed using Spearman’s rho and Pearson’s rho as presented in Table 3.

[Figure omitted. See PDF.]

Fig 1. Correlation between total mean score and overall satisfaction in the ED.

https://doi.org/10.1371/journal.pone.0242603.g001

[Figure omitted. See PDF.]

Table 3. Correlation between overall satisfaction and individual domains.

https://doi.org/10.1371/journal.pone.0242603.t003

Perceived quality.

The total score of the PRM-acute care was calculated in all patients (n = 81). The mean total score for all patients was 4.67 (95% CI 4.53–4.82) with a range from 2.66 to 6.00 (Table 4). The overall satisfaction of ED-care was 8.4/10, (range 6 to 10). Evaluation of scores per domain showed a mean score in the domain ‘relief of symptoms’ of 3.03 (95% CI 2.68–3.35) as presented in Table 4. Seventy-four out of 81 patients responded to both questions within this domain. Two patients did not experience any symptoms during arrival and discharge, whereas five patients did not answer one of the two questions. These records were excluded. The domain ‘understanding the diagnosis’ was scored by 79 patients and had a mean score of 4.66 (95% CI 4.30–5.02). The domain ‘understanding the treatment plan was scored with a mean of 5.33 (95% CI 5.17–5.49) and answered by all patients. All patients reported on the domain ‘patient experiences’ and revealed a mean score of 5.31 (95% CI 5.19–5.43). Patients graded the domain ‘reassurance’ (n = 81) with a mean score of 4.93 (95% CI 4.67–5.18). A graphic overview of the distribution of scores within the domains is presented in the figure in S1 Fig.

[Figure omitted. See PDF.]

Table 4. Mean scores per domain.

https://doi.org/10.1371/journal.pone.0242603.t004

Discussion

This study is the first use of PRMs in the ED in the Netherlands, consisting of both outcome and experience measures. We examined the validity of the PRM-acute care following the COSMIN-criteria for a formative construct-model. Intensive previous research formed the basis of the PRM-acute care [9], including semi-structured interviews with both healthcare professionals and patients, which was followed by cognitive testing in this study. Therefore, we deem the face- and content validity as good. Additionally, we conclude that the construct validity is adequate. This is supported by the demonstrated differences in perceived quality of care between subgroups and the correlation between the overall satisfaction of the ED-care and total score of the PRM-acute care.

The most notable difference in perceived quality of care exists between patients experiencing severe symptoms on arrival int the ED and patients with less severe symptoms. The severity of symptoms appears to be positively correlated with the total mean score of the PRM-acute care. Boudreaux et al also showed that the satisfaction level of ED-care was higher in those with serious illnesses or emergency needs [26]. We believe that our findings can be explained by the increased attention of healthcare professionals for patients who are obviously suffering and the perception of a more favourable throughput time in these patients.

Furthermore, we found differences between subgroups in specific domains of the PRM-acute care, which is important as in a formative model all domains determine the perceived quality of care. Firstly, patients who received less education did perceive a better understanding of the diagnosis. These results are in line with previous findings, showing that patients with less education tend to have a higher perceived quality of care and patient satisfaction [27–29]. Secondly, patients with a LOS-ED <4 hours had a better understanding of the diagnosis. This could be due to the complexity of the situation of patients with a LOS-ED ≥4 hours and the number of consultants involved. Research shows that the complexity of the case and the number of consultants involved are correlated with the LOS-ED [30, 31]. Lastly, admitted patients were more satisfied with their ED experiences than discharged patients. This seems to be caused by a lower satisfaction with the waiting time in discharged patients. An association between perceived waiting time vs expected waiting time on patient satisfaction has been indicated previously [32].

Moreover, we found a positive correlation between the overall satisfaction with the ED-care and the total mean score of the PRM-acute care. A positive correlation between the overall score of the ED and the domains ‘understanding of the diagnosis’, ‘understanding of the treatment plan’ and ‘experiences’ was also found. These correlations show that an increase in understanding the diagnosis or treatment plan, as well as better experiences, may induce an increase in overall ED-rating, which is in accordance with previous research [28, 33]. The domain ‘relief of symptoms’ did not significantly correlate with overall satisfaction, which may be due to the selected scoring method, which was based on literature regarding only pain instead of heterogeneous symptoms. Moreover, the found correlations endorse the underlying formative model as the rated perceived quality of care increases even if only one of the domains shows an increase. However, as the correlation between overall satisfaction and the total score of the PRM-acute care knows a wide distribution, grading overall satisfaction by a report mark cannot fathom the complexity of perceived quality of care. Therefore, a more elaborate model is needed, such as the PRM-acute care model.

Due to the study design and construct model we were not able to evaluate the reliability and thus were limited to the evaluation of the face-, content- and construct validity. The validating measurements are less well-known for a formative model and therefore might seem limited. However, this does not imply that the methods we used to validate the PRM-acute care are less reliable or validating [18].

Following a demonstrated validity of the PRM-acute care, we evaluated the perceived quality of ED-care for internal medicine patients. Overall, the perceived quality of care in the EDs was good, with a mean score of 4.67/6.0. As the Dutch healthcare system is known as outstanding in Europe, with the Netherlands being the only country consistently among the top 3 of the European Health Consumer Index [34], these results may be an example of the high quality of care in the Netherlands. Performing this study internationally would be of interest in order to evaluate the association between the perceived quality of care and the ranking in the European Health Consumer Index.

Within the specific domains, the most remarkable findings concern the domain ‘understanding the treatment plan’. In our study, patients perceive their understanding of the treatment plan good to very well. However, many studies have shown that patients regularly do not understand their treatment plan or discharge instructions [35, 36]. More importantly, most patients appear to be unaware of their lack of understanding, which might be also the case in our study and an explanation for the high scores [37, 38]. So, based on our results and the literature, it is important for physicians in the ED to be aware of the possible dissimilarity between perceived understanding and real understanding. The teach-back method could be used as a tool to confirm understanding and improve recall, especially in discharged patients [39].

Evaluating the use of the PRM-acute care, we believe implementing this questionnaire into daily practice is feasible. Our study did not reveal major problems during the inclusion process, besides the challenge of reaching discharged patients. Almost all of the included patients filled out all questions and did not report any difficulties. The questionnaire is short, consisting of only 11 questions, which is not time consuming (around 10 minutes). Another study in the Netherlands also showed the feasibility of using a PROM in an acute medical unit. Patients especially appreciated the fact that their view was taken into account [40].

Limitations

On account of the spread of COVID-19, patient based research was suspended indefinitely. Subsequently, the smaller sample size could have contributed to the inability to demonstrate differences between subgroups. One might also suggest differences are simply not there, because the healthcare system in the Netherlands is known to be outstanding for several years [34, 41]. This could also be the cause for high scores of perceived quality of care among various subgroups. Especially the distribution between hospitalised and discharged patients is not optimal to identify differences between these groups, even though this distribution represents daily practice. We experienced that physicians were prone to forget to approach patients in the ED, when not reminded by a researcher. Since the admitted patients could be approached on a later moment in time, this has led to a skewed distribution between discharged and admitted patients and has contributed to selection bias. Additionally, it is possible that during busy periods in the ED physicians were more prone to forget to include patients. This may have led to a greater perceived quality of care amongst the included patients, as busy periods may affect the length of stay and communication negatively.

Secondly, patients were asked to fill in the questionnaire within 12–72 hours after their ED visit, because of an intended future intervention study. This delay can affect the memory of the patient and cause recall bias [42]. To limit recall bias, patients should preferably fill out the questionnaire immediately after their ED visit.

Lastly, patients’ opinions about the concept questionnaire were inventoried during individual interviews. Organising focus groups aiming to select relevant questions could have resulted in a more profound discussion of this topic.

Conclusion

The PRM-acute care is a valid instrument to measure the perceived quality of healthcare in an acute setting for internal medicine patients. Additionally, patients reported a good perceived quality of care in the ED and a score ranging from moderate to well was given for each of the relevant domains.

Recommendations

We recommend the use of the PRM-acute care in the ED to evaluate the perceived quality of care for internal medicine patients in order to improve the quality of care. As the PRM-acute care is able to indicate within which domain(s) improvements are needed, tailor-made adjustments can be directly implemented for every single patient and in the ED as a whole. Cross-cultural validation should be executed to validate this instrument for use in international settings. When it is not possible to execute the PRM-acute care, the use of an overall satisfaction score of the ED-care can be considered as a screening tool for the perceived quality of care. We only recommend this for severely time constrained situations, as patients who perceived a low quality of care can be missed and it will remain unclear in which domain improvements could be beneficial.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. Questionnaire in Dutch.

https://doi.org/10.1371/journal.pone.0242603.s001

(DOCX)

S2 Appendix. Questionnaire in English.

https://doi.org/10.1371/journal.pone.0242603.s002

(DOCX)

S1 Table. Scoring domain relief of symptoms.

https://doi.org/10.1371/journal.pone.0242603.s003

(DOCX)

S1 Fig. Overview of distributions of scores per domain.

https://doi.org/10.1371/journal.pone.0242603.s004

(DOCX)

S1 Dataset. Dataset PRM-acute care.

https://doi.org/10.1371/journal.pone.0242603.s005

(XLSX)

Acknowledgments

We would like to thank Myriam Blokland, Eva Groot, Sani Kreca, Lara Pladet, Kaithlyn Tjong and Eline Velker, all medical students of Amsterdam UMC, for their help in recruiting participants.

Citation: Kremers MNT, Mols EEM, Simons YAE, van Kuijk SMJ, Holleman F, Nanayakkara PWB, et al. (2020) Quality of acute internal medicine: A patient-centered approach. Validation and usage of the Patient Reported Measure-acute care in the Netherlands. PLoS ONE 15(12): e0242603. https://doi.org/10.1371/journal.pone.0242603

References

1. Delnoij D, Hendriks M. De CQ-index: het meten van klantervaringen in de zorg. TSG 2008;86:440–446.

2. Porter ME. What is value in health care? NEJM 2010 December 23; 363(26): 2477–2481. pmid:21142528

3. Institute of Medicine (US) Committee on Quality of Health Care. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US); 2001.

4. Faessler L, Brodbeck J, Schuetz P, Haubitz S, Mueller B, Perrig-Chiello P. Medical patients' affective well-being after emergency department admission: The role of personal and social resources and health-related variables. PloS one 2019;14(3):e0212900. pmid:30893347

5. Crow R, Gage H, Hampson S, Hart J, Kimber A, Storey L, et al. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess 2002;6(32):1–244. pmid:12925269

6. Cleary PD, Edgman-Levitan S. Health care quality. Incorporating consumer perspectives. JAMA 1997 November 19;278(19):1608–1612. pmid:9370508

7. Weldring T, Smith SMS. Article Commentary: Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Health Services Insights 2013 Jan;6:HSI.S11093. https://doi.org/10.4137/HSI.S11093 pmid:25114561

8. Inspectie voor de Gezondheidszorg en Jeugd. Basisset medisch specialistische zorg kwaliteitsindicatoren 2018. 2017 August.

9. Kremers MNT, Zaalberg T, van den Ende ES, van Beneden M, Holleman F, Nanayakkara PWB, et al. Patient's perspective on improving the quality of acute medical care: determining patient reported outcomes. BMJ Open Qual 2019 Sep 29;8(3):e000736–000736. eCollection 2019. pmid:31637327

10. Kremers MNT, Nanayakkara PWB, Levi M, Bell D, Haak HR. Strengths and weaknesses of the acute care systems in the United Kingdom and the Netherlands: what can we learn from each other? BMC Emerg Med 2019 19:1–7 pmid:30606124

11. Kremers MNT, Wachelder JJH, Nanayakkara PWB, HR Haak. Organisation of internal medicine in acute care in the Netherlands: a detailed overview. Neth J Med 2020 78(5):251–260. pmid:33093250

12. Olsen MF, Bjerre E, Hansen MD, Hilden J, Landler NE, Tendal B, et al. Pain relief that matters to patients: systematic review of empirical studies assessing the minimum clinically important difference in acute pain. BMC Med 2017 February 20;15(1):35–3. pmid:28215182

13. Gridley L, van den Dolder P A. The percentage improvement in Pain Scale as a measure of physiotherapy treatment effects. Aust J Physiother 2001;47(2):133–138. pmid:11552868

14. de Vet H. C. W., Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine. 1st edition: Cambridge University Press; 2011.

15. Edwards JR, Bagozzi RP. On the nature and direction of relationships between constructs and measures. Psychological methods 2000 Jun;5(2):155–174. pmid:10937327

16. Mokkink LM, Prinsen CA. COSMIN Study Design checklist for Patient-reported outcome measurement instruments. 2019 July,.

17. Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet CW, Terwee CB. COSMIN Methodology for systematic reviews of Patient‐Reported Outcome Measures (PROMs). Qual Life Res 2018 Feb,;27(5):1147–1157. pmid:29435801

18. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality of life research: an international journal of quality of life aspects of treatment, care and rehabilitation 2010;19(4):539–549.

19. Sixma HJ, Kerssens JJ, van Campen C, Peters L. Quality of care from the patients' perspective: from theoretical concept to a new measuring instrument. Health Expectations 1998 Nov;1(2):82–95. pmid:11281863

20. Travaline JM, Ruchinskas R, D'Alonzo GE Jr. Patient-Physician Communication: Why and How. J Am Osteopath Assoc 2005;105(1):13–18. pmid:15710660

21. Otani K, Chumbler NR, Herrmann PA, Kurz RS. Impact of Pain on Patient Satisfaction Integration Process: How Patients With Pain Combine Their Health Care Attribute Reactions. Health services research and managerial epidemiology 2015;2:2333392815615103. pmid:28462270

22. Wells N, Pasero C, McCaffery M. Improving the Quality of Care Through Pain Assessment and Management. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses: Rockewell MD: Agency for Healthcare Research and Quality (US); 2008. p. 469–489.

23. Nederlandse vereniging voor ziekenhuizen, NVZ, Nederlandse Federatie van Universitair Medische Centra, NFU. Vroege herkenning en behandeling van pijn. 2009. [Cited June 6 2020] Available from: https://www.vmszorg.nl/wp-content/uploads/2017/11/web_2009.0109_praktijkgids_pijn.pdf

24. Parker BT, Marco C. Emergency department length of stay: accuracy of patient estimates. The western journal of emergency medicine 2014;15(2):170–175. pmid:24672606

25. Bos N, Sizmur S, Graham C, van Stel HF. The accident and emergency department questionnaire: a measure for patients' experiences in the accident and emergency department. BMJ Qual Saf 2013 Feb;22(2):139–146. pmid:22942399

26. Boudreaux ED, Friedman J, Chansky ME, Baumann BM. Emergency department patient satisfaction: examining the role of acuity. Acad Emerg Med 2004 February 01;11(2):162–168. pmid:14759959

27. Rahmqvist M. Patient satisfaction in relation to age, health status and other background factors: A model for comparisons of care units. International journal for quality in health care: journal of the International Society for Quality in Health Care / ISQua 2001;13:385–90.

28. Hall JA, Dornan MC. Patient sociodemographic characteristics as predictors of satisfaction with medical care: A meta-analysis. Social Science & Medicine 1990;30(7):811–818. pmid:2138357

29. Alberti G. Transforming emergency care in England. Department of Health 2004 Oct 26,:1–48.

30. Brouns SH, Stassen PM, Lambooij SL, Dieleman J, Vanderfeesten IT, Haak HR. Organisational Factors Induce Prolonged Emergency Department Length of Stay in Elderly Patients—A Retrospective Cohort Study. PLoS One 2015 Aug 12;10(8):e0135066. pmid:26267794

31. Driesen BEJM, van Riet BHG, Verkerk L, Bonjer J, Merten H, Nanayakkara PWB. Long length of stay at the emergency department is mostly caused by organisational factors outside the influence of the emergency department: A root cause analysis. PLoS One 2018 Sep 14;13(9):e0202751. pmid:30216348

32. Thompson DA, Yarnold PR. Relating Patient Satisfaction to Waiting Time Perceptions and Expectations: The Disconfirmation Paradigm. Acad Emerg Med 1995;2(12):1057–1062. pmid:8597916

33. Rapport F, Hibbert P, Baysari M, Long JC, Seah R, Zheng WY, et al. What do patients really want? An in-depth examination of patient experience in four Australian hospitals. BMC health services research 2019 Jan 15,;19(1):38. pmid:30646962

34. 2017 Euro Health Consumer Index. PharmacoEcon Outcomes News 2018 Feb;796(1):31.

35. Alberti TL, Nannini A. Patient comprehension of discharge instructions from the emergency department: a literature review. J Am Assoc Nurse Pract 2013 April 01;25(4):186–194. pmid:24218236

36. Hoek AE, Anker SCP, van Beeck EF, Burdorf A, Rood PPM, Haagsma JA. Patient Discharge Instructions in the Emergency Department and Their Effects on Comprehension and Recall of Discharge Instructions: A Systematic Review and Meta-analysis. Ann Emerg Med 2020 March 01;75(3):435–444. pmid:31439363

37. Lin MJ, Tirosh AG, Landry A. Examining patient comprehension of emergency department discharge instructions: Who says they understand when they do not? Intern Emerg Med 2015 December 01;10(8):993–1002. pmid:26358466

38. Engel KG, Heisler M, Smith DM, Robinson CH, Forman JH, Ubel PA. Patient comprehension of emergency department care and instructions: are patients aware of when they do not understand? Ann Emerg Med 2009 April 01;53(4):454-461.e15.

39. Slater BA, Huang Y, Dalawari P. The Impact of Teach-Back Method on Retention of Key Domains of Emergency Department Discharge Instructions. J Emerg Med 2017 Nov;53(5):e59–e65. pmid:28939399

40. Galen LS, der Schors W, Damen NL, Kramer M, Wagner C, Nanayakkara PWB. Measurement of generic patient reported outcome measures (PROMs) in an acute admission unit: A feasibility study. Acute Med 2016;15(1):13–19. pmid:27116582

41. Watson R. Netherlands tops European healthcare league, with UK coming in at 12th. BMJ 2012;344:e3430. pmid:22589527

42. Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. Journal of multidisciplinary healthcare 2016;9:211–217. pmid:27217764