Implementing paper-based patient-reported outcome collection within outpatient integrative health and medicine

Abstract

Objective

To investigate the feasibility of pre- and post-encounter patient-reported outcome (PRO) measure collection within an outpatient integrative health and medicine (IHM) clinic and to characterize factors associated with successful completion.

Methods

We conducted a retrospective review of 27,464 outpatient IHM encounters including 9,520 chiropractic; 8,237 acupuncture; 5,847 massage; 2,345 IHM consultation; and 1,515 osteopathic manipulation treatment encounters at four clinics offering IHM over 18 months. Patients were asked to complete paper questionnaires rating pain, anxiety, and stress from 0–10 immediately pre- and post-encounter. Generalized linear mixed effect regression models were used to examine the relationship between demographic, clinical, and operational covariates and completing (1) pre-encounter and (2) paired (i.e., pre and post) PROs.

Results

Patients (N = 5587, mean age 49 years, 74% white, 77% female) generally presented for musculoskeletal conditions (81.7%), with a chief complaint of pain (55.1%). 21,852 (79.6%) encounters were among patients who completed pre-encounter PROs; 11,709/21,852 (53.6%) completed subsequent post-encounter PROs. Odds of PRO completion were more impacted by provider, operational, and clinical-level factors than patient factors. Covariates associated with increased odds of pre-encounter PRO completion included being female, having additional IHM encounters, and having a pain or anxiety complaint. Covariates associated with increased odds of paired PRO completion included being aged 31–40 vs. 51–60 years and having additional IHM encounters.

Conclusion

Implementing a paper-based PRO collection system in outpatient IHM is feasible; however, collecting post-encounter PROs was challenging. Future endeavors should leverage the electronic health record and patient portals to optimize PRO collection and engage patients and clinical providers.

1.0 Introduction

Integrative health and medicine (IHM) is a whole-person paradigm of healthcare utilizing traditional, non-pharmacologic approaches to address the numerous physical and emotional factors which may impact an individual’s health [1]. Such an approach often combines therapeutic modalities from various ancient healing traditions [2], such as acupuncture, yoga, and massage, with conventional medical approaches, including pharmacotherapy and surgery, to provide care that addresses the mind, body, and spirit.

Applications of IHM in the ambulatory setting have been well-studied in the medical literature, from uses in the treatment of chronic pain [3] and symptom palliation [4, 5] to stress and anxiety reduction [6]. For example, a 2018 meta-analysis of 39 trials involving 20,827 patients found that acupuncture was superior to sham and/or no acupuncture treatment controls (p < 0.001) in the treatment of nonspecific musculoskeletal pain, osteoarthritis, chronic headache, or shoulder pain, with only a 15% decrease in effect at one year after treatment [7]. Similarly, results of a 2019 systematic review and meta-analysis of 47 randomized controlled trials (RCTs) suggest that chiropractic spinal manipulation therapy yields similar pain reduction as other recommended therapies (mean difference -3.17, 95% confidence interval [CI] -7.85 to 1.51) [8], and a lesser risk of benzodiazepine (OR 0.67; 95% CI 0.62 to 0.74) [9] or gabapentin (OR 0.53; 95% CI 0.40 to 0.71) prescription [10]. With regards to anxiety reduction, numerous systematic reviews support the effectiveness of acupuncture [11], massage [12], and yoga [13], as well as the safety and acceptability of these modalities.

Perhaps most notably, IHM has been widely utilized in the service of improving quality of life across populations and disease states [14]. A multi-site, prospective observational study of 409 patients undergoing an IHM program for chronic pain demonstrated significant improvements in pain (mean change -1.20; 95% CI − 1.60 to − 0.80) and quality of life (mean change -1.29; 95% CI − 1.54 to − 1.04) [15]. In a single-site prospective observational study, Crocker et al. reported a significant improvement (p = 0.002) in health-related quality of life from baseline to 12 months among 177 adults seeking ambulatory IHM care [16], as measured by numerous patient-reported outcome (PRO) instruments. PROs are subjective measures that quantify perceptions of health-related quality of life from the patient’s perspective [17]. Notably, these self-reported data are obtained directly from patients based upon their own appraisal of treatment impact [18], unlike physiological or laboratory assessments collected by clinicians or investigators. PROs are particularly vital in evaluating the effectiveness of health interventions for chronic conditions, where primary treatment goals may include improving quality of life, day-to-day function, and symptomatic treatment as well as direct disease management/reversal [19]. By providing data from the patient perspective, PROs capture holistic assessments of treatment effect that may otherwise not be obtained through traditional data collection procedures [20]. From a study design standpoint, routine collection of PROs in the clinical setting can highlight areas in need of methodological improvement [21].

Existing literature indicates that routine PRO collection and monitoring potentiates enhanced patient-clinician communication, clinician understanding of symptoms, symptom management, and quality of life [22]; thus, these inherently patient-centered measures complement conventional, objective outcomes, with comparable reliability between PROs and clinical measurements [23]. Routine implementation throughout the treatment course can help clinicians tailor and personalize care to each individual’s unique needs [24], while offering avenues for shared decision making, evaluating novel therapies and provider/treatment performance, and insight into systems-level quality improvement [25]. PRO collection has drastically increased over the last decade, with numerous professional societies encouraging their use in clinical care and research [26], especially in the midst of a shift towards value-based care reimbursement [25]. Within the realm of IHM, such measures may even constitute primary outcomes and the main indicators of treatment success, given IHM’s wholistic paradigm and focus upon quality of life [27, 28]: Thus, efforts to incorporate PRO collection into routine IHM care are vital in appropriately measuring care outcomes.

The utility of and implementation strategies for PRO collection have been widely studied in the outpatient setting, particularly in oncology and pain care [29], with a recent shift towards electronic data capture [30–32]. While electronic and paper-based methods have been shown to be comparable [33], electronic approaches pose certain limitations ranging from the individual to systems level. These limitations include the need for technology skills among patients and providers, internet access, significant financial investment, robust electronic health record (EHR) integration, and dedicated information technology support [30, 34]. In addition, challenges in technical accessibility may further widen the “digital divide,” contributing to existing disparities in PRO collection among diverse and underrepresented populations [35]. In contrast, though they require additional time and resources to print and transfer to electronic databases, paper-based PRO collection methods offer a uniquely low-tech, low-investment, accessible option.

Facilitators of successful outpatient PRO implementation across diverse fields of medicine have included: selecting clinically relevant PROs, integrating procedures within routine workflow, and minimizing complexity, while time constraints, poor communication around importance of PROs, language and/or technological literacy, and lack of administrative support have been barriers to completion [36, 37]. In an outpatient upper hand and extremity clinic, patient-level factors associated with completion of PRO measures included patient literacy, physical ability, and questionnaire design, among others [38, 39]. Prior research upon large-scale, longitudinal PRO collection within both the outpatient and inpatient IHM settings underscores its feasibility, acceptability, and potential benefit following routine implementation [16, 39–43]. A 2023 process-improvement study by Rodgers-Melnick et al. [44] highlighted efforts to optimize PRO collection by inpatient music therapists, including offering resources and guidelines for data collection as well as opportunities for staff feedback. However, to our knowledge, there is limited research on patient, clinical-, and operational-level factors associated with PRO collection across IHM modalities in the outpatient setting.

Successful integration of PRO measurement models into the outpatient IHM setting represents a critically important endeavor in measuring the real-world clinical effectiveness of IHM delivery. Thus, the present study reports on the pilot implementation of a paper-based PRO collection endeavor across four IHM clinics. The objectives of this study were to: (1) describe the implementation of a paper-based PRO collection system and (2) examine demographic, clinical, and operational characteristics associated with completing any pre-encounter PRO and completing a paired PRO (i.e., a post-encounter PRO following the completion of a pre-encounter PRO) among adult patients receiving outpatient IHM care.

2.0 Materials and methods

2.1 Participants and design

The present study is a retrospective review of all encounters among adult patients (ages 18 and older) receiving care at one of four outpatient IHM clinics between January 1, 2019 and July 31, 2020 which met the following criteria: (1) the encounter was for acupuncture, chiropractic, massage, integrative medicine consultation (IMC), or osteopathic manipulation treatment (OMT) and (2) the encounter was at an IHM clinic where patient service representatives (PSRs) had been instructed to administer the paper PRO measures. IHM encounters not meeting these criteria were excluded from the sample. Data access via the EHR began on March 16, 2020; raw data were extracted on May 17, 2023. Authors were able to access individually identifiable health data at time of data extraction, as approved by the hospital’s ethics committee.

2.2 Setting

University Hospitals (UH) is a not-for-profit health system in Northeast Ohio serving the needs of more than 1.2 million unique patients annually. University Hospitals Connor Whole Health (UHCWH) is a center offering IHM embedded within the larger UH health system. In accordance with prior established tenets of whole health, UHCWH partners with UH providers, departments, and institutes to provide IHM modalities and empower patients in managing health and wellbeing while centering patient goals [45]. In efforts to meet the growing demand for comprehensive strategies to manage a wide variety of health concerns, UHCWH services are integrated into the UH health system across primary care, multispecialty outpatient care [46], hematology/oncology [47–49], fertility, rehabilitation, and orthopedic divisions spanning a central academic medical center [50] and eight community hospitals [51].

IHM providers active during the study period included six acupuncturists, ten massage therapists, four chiropractors, four integrative medicine physicians and physician assistants, and one physician providing OMT across four clinical locations. Additionally, UHCWH had eleven expressive therapists (board-certified music and art therapists) and four mind-body program instructors who facilitated yoga therapy, mindfulness, meditation, and stress management and resilience training offerings.

2.3 Ethics and permissions

This study was approved by the UH Cleveland Medical Center Institutional Review Board (STUDY20200308) as a retrospective chart review with a waiver of informed consent by the hospital system’s research ethics committee. This study was conducted in accordance with the Declaration of Helsinki.

2.4 Data collection

2.4.1 Questionnaires.

The numeric rating scale (NRS) is commonly used for measuring pain, stress, and anxiety in clinical practice [52–54] and research [44, 48, 51], with demonstrated validity across these constructs. Owing to its simplicity, ease of use across populations, focus on symptoms in the present moment, and generalizability across various age and education levels [52], the NRS was selected as the instrument of choice in the present study.

A 3-item pre-encounter questionnaire was developed to measure pain, anxiety, and stress with space to rate each symptom on a numeric rating scale (NRS) from 0 (“none”) to 10 (“worst possible”), printed on a yellow-colored sheet of paper. An identical post-encounter questionnaire was also developed and printed on a blue-colored paper to be distinguished from the pre-encounter assessment. Brief acute NRS measures of pain, anxiety, and stress were specifically chosen given their successful collection in prior studies [51] and short completion time, minimizing response burden for patients. In addition, the selection of maximally relevant measures was prioritized in accordance with clinical practice guidelines [55], so as to integrate their collection within clinic workflow [38]. Furthermore, prior research has demonstrated that patients often seek IHM care for acute pain or psychosocial complaints, informing construct choice [56, 57]. As part of routine clinical care, the PSRs presented the pre-encounter questionnaires to patients as they checked in for their appointments, alongside any relevant intake information. Patients completed questionnaires by hand and subsequently presented these to the IHM provider. Following the IHM visit, providers presented patients with the post-encounter questionnaires and instructed patients to leave the completed questionnaire in the treatment room prior to leaving the clinic. Pre- and post-encounter NRS scores were then collected and entered into the EHR by the IHM clinical providers at the conclusion of the encounter. This workflow prevented patients from having to wait for PSRs to collect their completed questionnaires before leaving and promoted efficiency as the provider typically had the treatment note open to quickly complete data entry. Questionnaires were collected over an 18-month timespan.

The questionnaire collection processes were generally overseen by clinical operations directors (CK and DV), though there were no onsite PRO collection coordinators. Providers whose patients completed PRO measures in lower volumes were occasionally encouraged to increase collection by these coordinators. PRO collection efforts were de-prioritized, though not entirely stopped, at the onset of the coronavirus-2019 (COVID-19) pandemic in March 2020 and resumed in June 2020 so as to minimize the exchange of paperwork. During this time, acupuncture and massage operations paused entirely for six weeks, and chiropractic operated at a much lesser volume.

2.4.2 Patient characteristics.

The following data were extracted from all records which met eligibility criteria: (1) demographics including age, sex, race, and ethnicity; (2) International Classification of Diseases (ICD)-10 codes for all diagnoses and chief complaints listed in provider documentation; and (3) IHM documentation data of modality (i.e., acupuncture, chiropractic, massage, IMC, or OMT) and PROs (i.e., NRS scores of pain, anxiety, and stress) that were collected pre- and post-encounter. ICD-10 codes were only available for patients who had at least one encounter with an acupuncturist, chiropractor, physician, or advanced practice provider. Demographic data including sex, race, and ethnicity were extracted exactly as they were entered into the EHR by medical staff and may not have reflected the gender, racial, and/or ethnic identities of the patients included in this study [58]. All data, including full note documentation, were extracted via a single structured query language script from the UH Electronic Data Warehouse. Regular expressions functions including str_extract_all from the stringr package [59] and regmatches, gregepr, sub, and gsub from base R version 4.3.0 were used to extract clinical information from the free-text note.

2.5 Statistical analysis methods

Descriptive statistics were tabulated to characterize the study population, while run charts were constructed to visualize PRO questionnaire completion rates per week. Generalized linear mixed effect regression models with binary outcome distribution were used to examine the relationship between demographic, clinical, and operational characteristics with each of the two outcomes: (1) completion of any pre-encounter PRO, and (2) paired PRO completion which was defined as completing a post-encounter PRO following the completion of a respective pre-encounter PRO (e.g., pre-encounter stress and post-encounter stress). Both models included the following covariates: (1) demographics including age (18–30, 31–40, 41–50, 51–60, 61–70, or 71+), sex (male or female), ethnicity (non-Hispanic [NH], Hispanic/Latino, or declined/missing), and race (White, American Indian, Asian, Black/African American, other/multi-racial, or declined/missing; (2) clinical characteristics including chief complaints (yes or no) of pain, headache, or anxiety and number of visits during the study period; and (3) operational characteristics including clinic location and time period (2019 quarter [Q] 1 through 2023Q3). Both models included a random intercept for patient and a random residual effect for provider nested within modality. The second model was limited to only those who provided a pre-encounter score and included a covariate for whether any pre-encounter PRO was rated ≥4 (i.e., moderate-to-severe pain, stress, or anxiety). The general linear mixed model was generated using proc GLMIMMIX in SAS software, Version 9.4 of the SAS System for Windows (Cary, NC).

3.0 Results

3.1 Sample

Between January 2019 and July 2020, there were 27,464 IHM encounters among 5,587 patients receiving outpatient care at UHCWH. These 27,464 included encounters for chiropractic (34.7%), acupuncture (30.0%), massage (21.3%), IMC (8.5%), and OMT (5.5%).

3.2 Demographics, chief complaints, and utilization

Table 1 summarizes demographics, chief complaints, and utilization characteristics. Patients (mean age: 48.95 ± 15.86 years at first encounter) were mostly White (74.0%) or Black/African American (17.0%), non-Hispanic (85.8%), and female (77.0%). Over half of patients reported a chief complaint of pain (55.1%) at some point during the study, with specific pain complaints located in the back (34.3%), neck (23.4%), and shoulder (23.4%). Other common chief complaints included stress (12.7%), anxiety (11.3%), and headache/migraine (8.9%). Median [IQR] encounter engagement volume among patients was 2 [1–6] encounters, with the most common IHM modality being massage (35.8%) followed by acupuncture (28.2%), chiropractic (27.5%), IMC (23.1%), and OMT (13.0%).

[Figure omitted. See PDF.]

3.3 Clinical characteristics

S1 Table summarizes the Major Expanded Diagnosis Clusters (MEDCs) among 4,194 patients who attended at least one IHM encounter that was not massage. Most patients presented for musculoskeletal diagnoses (81.7%), followed by neurologic diagnoses (50.5%) such as headaches, sleep problems, and migraines. Many patients also presented for general signs and symptoms (49.5%), including fatigue (9.9%), administrative concerns (25.3%) such as non-specific lab abnormalities (22.1%) and preventive care (6.7%), and psychosocial and mental health (23.6%) conditions.

3.4 Pre-encounter PRO completion data

Table 2 summarizes pre-encounter PRO completion characteristics among all encounters including (1) the count and percentage of encounters with complete pre-encounter PROs, (2) the count and percentage of encounters with pre-encounter PROs rated 0/10 (i.e., no symptom), and (3) the count, percentage, mean, and standard deviation (SD) of encounters with pre-encounter PROs rated ≥1/10 (i.e., mild to severe symptoms). Of the 27,464 encounters included in this sample, 21,852 (79.6%) were among patients who completed at least one PRO. Of the 21,852 encounters in which any pre-encounter PRO was completed, the majority were obtained from chiropractic encounters (41.2%), followed by acupuncture (25.8%), massage (20.4%), IMC (7.5%), and OMT (5.1%) encounters.

[Figure omitted. See PDF.]

Of the 27,464 encounters, 21,798 (79.4%) were among patients who completed a pre-encounter pain questionnaire. Across 19,752 encounters in which patients reported pain ≥1, patients reported mean (± SD) pain scores of 4.25 ± 2.25 units on the NRS. Similarly, 19,749 (71.9%) encounters were among patients who completed a pre-encounter questionnaire for stress with mean ± SD scores of 4.02 ± 2.40 among patients reporting stress ≥1. Lastly, 19,767 (72%) encounters were among patients who completed a pre-encounter questionnaire for anxiety with mean ± SD scores of 3.76 ± 2.32 among patients reporting anxiety ≥1. Table 2 provides additional details regarding pre-encounter pain, stress, and anxiety scores by modality.

A run chart of pre-encounter PRO completion over the 18-month (83-week) period is shown in Fig 1. Notable features include a decrease in net pre-encounter PRO completion at week 64 due to de-prioritizing data collection in the wake of COVID-19-related infection control concerns, as well as an increase at week 79 when operational emphasis upon paper PRO collection resumed. The median [IQR] percentage of pre-encounter PRO completion per week was 84.6% [74.3–87.5] over 83 weeks. Pre-encounter PRO completion was maintained at above 75% of all encounters per week for 61/83 (73.5%) weeks.

[Figure omitted. See PDF.]

Fig 2 and S2 Table depict adjusted odds ratios (aOR) for completing any pre-encounter PRO questionnaire. Chief complaints of pain, headache, and anxiety were selected due to their high clinical relevance and prevalence so as to display in the model. For this generalized linear mixed model with binary distribution, the covariance parameter estimate ± standard error (SE) for the random effect of provider nested within modality was 3.25 ± 0.96, while the covariance parameter estimate for the random effect of patient was 0.96 ± 0.06. Among all 27,464 encounters, the following were associated with increased odds (aOR [95% CI]) of completing a pre-encounter PRO: (1) being female as compared to male (1.178 [1.028, 1.349]); (2) attending an additional IHM encounter beyond two encounters during the study period (1.016, [1.009, 1.022]); (3) being seen in 2019Q2 (4.193 [3.584, 4.904]), 2019Q3 (3.223 [2.752, 3.774]), 2019Q4 (3.668 [3.115, 4.318]) or 2020Q1 (2.072 [1.750, 2.454]) as compared to 2019Q1; and (4) having a pain (1.365 [1.215, 1.534]) or anxiety complaint (3.188 [2.601, 3.908]). Covariates associated with decreased odds of completing a pre-encounter PRO included (1) being seen in clinic 2 (0.060 [0.047, 0.075]) or clinic 3 (0.729 [0.605, 0.877]) as compared to clinic 1 and (2) being seen in 2020Q2 (0.148 [0.124, 0.177]) or 2020Q3 (0.163 [0.133, 0.199]) as compared to 2019Q1.

[Figure omitted. See PDF.]

3.5 Paired PRO completion data

Of the 21,852 encounters in which patients completed at least one PRO, 11,709 (53.6%) were among patients who completed a subsequent post-encounter PRO.

Fig 3 depicts a run chart of paired PRO completion. Notable features include a sharp increase in paired PRO completion at week 30 when the chiropractic service began increasing efforts to obtain post-encounter PROs, a decrease at week 64 due to infection control concerns related to COVID-19, and an increase at week 79 when paper PRO collection efforts were resumed. The median [IQR] percentage of paired PRO completion per week was 45.6 [29.7–49.7]. Across 83 weeks, only 20 (24.1%) were weeks in which paired PRO completion constituted ≥50% of all encounters.

[Figure omitted. See PDF.]

3.6 Model 2: Predicting post-encounter PRO completion

Fig 4 and S3 Table depict adjusted odds ratios (aOR) for completing any post-encounter PRO questionnaire following the completion of a pre-encounter PRO questionnaire. For this model, the covariance parameter estimates for the random effect of provider nested within modality was 8.80 ± 2.77, while the covariance parameter estimate for the random effect of patient was 1.36 ± 0.08. Among 21,852 encounters, the following were associated with increased odds (aOR [95% CI]) of completing a post-encounter PRO: (1) being aged 31–40 years as compared to 51–60 years (1.279 [1.041, 1.572]); (2) having an additional IHM encounter beyond 2 encounters (1.039, [1.031, 1.047]); (3) visiting clinic 3 as compared to clinic 1 (1.322 [1.088, 1.607]); and (4) being seen in 2019Q4 as compared to 2019Q1 (2.505 [2.087, 3.007]). Covariates associated with decreased odds of completing a post-encounter PRO included (1) visiting clinic 2 as compared to clinic 1 (0.377 [0.262, 0.543]); (2) being seen in 2019Q2 as compared to 2019Q1 (0.711 [0.600, 0.843]); and (3) having a pain complaint (0.791 [0.692, 0.904]).

[Figure omitted. See PDF.]

4.0 Discussion

The purpose of this study was to describe the implementation of a paper-based PRO collection system within an outpatient IHM center and determine characteristics associated with completion of pre-encounter and paired (e.g., pre- and post-encounter) measures. Overall, pre-encounter PRO collection was feasible and successful, while a smaller proportion of paired PROs were collected. Provider- and organization-level factors appeared to impact odds of PRO completion more than patient-level factors.

The demographic distribution of patients in this study is consistent with previous literature on IHM utilization [60], reflecting predominantly white, middle-aged females seeking IHM care; however, 17% of patients in this study identified as Black, representing a higher proportion as compared to other IHM literature [16, 61]. Black or African-American individuals represented 29% of the total population of the county in which the IHM clinics were located per the 2020 United States census [62]. Thus, this work continues to underscore the need to diversify IHM clientele through innovative outreach and addressing barriers, both individual and systemic, to equitable IHM access and participation.

The observed prevalence of pain complaints and musculoskeletal conditions is consistent with the literature that such complaints, including back and neck pain, are the most common reasons cited to seek IHM care [63, 64]. Thus, further investigation of PROs is warranted to demonstrate both immediate and longitudinal impacts of IHM for these conditions.

The most prevalent IHM modality sought among all patients was massage (35.8%), which may be due to initial massage therapy treatments being offered free-of-charge to employees within the UH health system during a period within the study timeframe. These rates are consistent with previous work demonstrating that massage is one of the most common IHM modalities sought across demographic groups [56]. Notably, at the time of study, massage was an exclusively self-pay modality, where others had some level of insurance reimbursement depending upon patient type and associated diagnoses. It may be worthwhile for future endeavors to examine the relationship between insurance status and modalities sought among patients seeking IHM care.

Among the key observations of this pilot study was the sheer volume able to be captured using a low-tech, paper-based PRO collection system. The large sample size of 27,464 encounters within this timeframe, from which 21,852 pre-encounter PRO questionnaires were completed, is akin to that of Hui et al. [30], who surveyed 6631 patients across 25,767 visits to an outpatient palliative care clinic. In their study, PROs were completed by 100% of patients, attributable to mandatory completion via integration of both paper and electronic versions into clinic workflow, follow-up by clinic nurses, and clinician utilization of PROs during the visit. In contrast, patients presenting to any of the four IHM clinics where PSRs were collecting these data were not required to complete PROs; our observed median 84.6% pre-encounter PRO collection exceeds the average 71% response rates of PROs captured by clinical registries [57]. Within the realm of outpatient PRO implementation, our pre-encounter collection rate falls within the range of other studies where it was not mandatory, ranging from 54% to 88% [65–67], and exceeds the 59% collection rate across 12,207 visits within an ambulatory IHM clinic [39]. However, to our knowledge, this study is one of the first to explore PRO collection immediately pre- and post-encounter within IHM modalities. From our perspective, the large volume of data capture, particularly pre-encounter, is attributable to support from operations staff and training of PSRs, the low-tech, ease of disseminating paper questionnaires, and responsiveness of clinical staff to feedback regarding increasing collection efforts.

Demographic and clinical covariates associated with increased odds of pre-encounter PRO collection included female sex, attending more than two IHM encounters, and having a pain or anxiety complaint. This is consistent with previous research in which females were more likely to engage with health information, such as contributing to survey responses [68, 69].

Additionally, patients presenting with pain or anxiety complaints may have been more likely to complete pre-encounter PRO measures out of a willingness to communicate the intensity of their needs to providers, as well as the relative heterogeneity of symptoms appraised as such [70].

Decreases in both pre-encounter and paired PRO collection at week 64 corresponded to operational pauses due to the onset of COVID-19: clinical operations were greatly reduced, and less emphasis was placed upon PRO collection, though it was not eliminated entirely. Anecdotally, PSRs noted that hesitance towards touching clinic writing instruments due to perceived risk of infection was a major deterrent to PRO questionnaire completion, though infection has been shown to be unlikely from day-to-day use of droplet-contaminated fomites [71]. However, these observations contextualize findings of decreased odds of pre-encounter completion during 2020Q2 and 2020Q3, when the COVID-19 pandemic first began to affect clinical operations within the health system.

In contrast, paired PRO completion rates (53.6% of those in which pre-encounter PROs were collected) were lower than pre-encounter PRO completion rates (79.6% of all encounters). To our knowledge, limited existing work describes immediate post-encounter PRO collection in the IHM setting; however, using Sisodia et al.’s [72] threshold of 50% completion as a successful collection rate, we achieved successful paired PRO collection during just 24.1% of the study period. This observation corroborates the need for future, targeted efforts to optimize immediate post-encounter PRO collection, particularly leveraging electronic data capture methods delivered via text message.

Covariates associated with increased paired PRO collection included being aged 31–40 years versus 51–60 years, attending more visits to the IHM center, and presenting to IHM care at Clinic 3. Though patients with a pain complaint were more likely to complete a pre-encounter, they were less likely to complete a paired PRO. Though the exact reasoning for this is unclear, possible explanation include pain taking more time to resolve than anxiety, it being a comparatively physical measure compared to anxiety or stress, or electing to verbally share pain relief with the provider rather than completing an NRS.

Clinic-to-clinic differences in rates of PRO collection are most likely attribute to operational differences in PSR staff. In particular, clinics 3 and 4 were locations shared by many departments in addition to the whole health center. Thus, PSRs at these locations, were managing patients from numerous providers, and may not have had direct, consistent communications from the whole health center’s policies or initiatives, such as reminders or emphasis upon questionnaire administration. In addition, these locations typically had low-patient-volume clinical operations. While real-time process monitoring was not conducted in this study, providing a comprehensive PRO collection training, managerial expectation, and monitoring process represents an important opportunity to unify administrative efforts to optimize and equalize collection across sites [38, 39, 44].

Notably, while massage was the most prevalent modality, followed by acupuncture, then chiropractic, pre-encounter PROs were collected most from chiropractic, then acupuncture, then massage therapy encounters. Differences in PRO collection by modality may be attributed to various factors. First, while we did not have a designated PRO collection coordinator, we did have administrative champions driving these efforts within the acupuncture (CK) and chiropractic (DV) modalities. Further, the responsivity of clinical staff to encouragement to increase PRO collection, evidenced in the increase in paired PRO completion at 30 weeks, underscores the importance of real-time monitoring and coordination to maximize data collection. In addition, other factors such as differences in workflows between providers and modalities may contribute to these observed differences in pre-encounter PRO collection. Thus, efforts to optimize PRO collection must account for modality-specific considerations.

The covariance parameter estimates from both logistic regression models suggest that provider- and operational-level factors, such as provider behavior, clinic location, and timing, were more influential than patient-level factors on PRO collection outcomes. This is consistent with prior findings that physician and administrative engagement were most strongly associated with PRO collection and program success [72]. Increased odds of completion of any PRO questionnaire at later quarters of 2019 may reflect the “adjustment time” to successfully integrate a PRO collection system into clinical operations [30]. Notably, neither race nor age had a detectable impact on odds of pre-encounter PRO completion.

Variation in PRO completion rates between clinic locations may be attributable to different PSR staffing each site, as well as each clinic’s unique operational procedures when patients checked in for their visit, underscoring the pivotal role of PSR and operations engagement in successful administration. In addition, this observation highlights the integral role of administrative and clinical staff engagement in PRO collection endeavors, and that clinic-specific factors must be considered with designing strategies for PRO implementation–i.e., there may not be a “one size fits all” approach [22].

This study had several limitations. These included potential response bias (i.e., non-assessment of other variables that may account for response rates) and lack of privacy for patients completing paper-based measures, which may have influenced patients who may have otherwise not completed a PRO to do so, or vice versa. In addition, the fact that we did not closely monitor PRO collection in real-time limited our ability to identify real-time barriers to PRO collection and generate solutions. Future implementation strategies should ideally include efforts to increase patient privacy while having dedicated personnel to manage and monitor real-time PRO collection efforts and aid patients in completing forms as needed. This latter point may be crucial as PRO collection migrates to an electronic format, especially to avoid worsening existing disparities.

In addition, the studied characteristics were just a snapshot of the myriad factors that may be associated with PRO completion, and it is near impossible to capture all attributes. However, this research provides an important initial framework for characterizing at least some factors associated with PRO completion in the IHM setting. While creating an abbreviated questionnaire measuring pain, anxiety, and stress allowed for an easy-to-implement instrument, our inability to measure other constructs (e.g., nausea or fatigue) for which patients also commonly seek IHM constitutes another study limitation.

Strengths of this study included its large sample size, implementation across four IHM clinic settings within a large healthcare system, and the novel collection effort of PROs immediately before and after IHM encounters. This represents a unique, large-scale effort to characterize PRO completion (and subsequently, perceived treatment impact) immediately following IHM modalities. The utilization of real-world data from the EHR allowed us to extract variables from many clinical encounters without having to enter this information into a separate database during the study. In addition, the relatively low-tech paper-based data collection method allowed for easy integration into clinic workflow, avoiding the logistical and access-related issues that electronic collection may pose.

With high rates of completion for pre- and paired measures, the low-cost paper questionnaires proved feasible (e.g., established proof of concept) from the lens of a single-encounter collection. However, electronic collection would be better suited for longitudinal collection and data analysis, especially if integrated into the EHR. Thus, future research will ideally explore routine electronic PRO collection and implementation in the outpatient IHM setting. Such efforts would align with current trends, especially in the post/chronic COVID-19 pandemic era, to move toward electronic data capture and potentiate real-time collection and visualization for both patients and providers [30]. If issues of implementation are surmounted, electronic PRO collection can improve data accuracy and integrity [73], maximizing compliance while minimizing secondary data entry errors [74] and administrative burden [75, 76]. In fact, the present work has inspired the subsequent piloting of efforts at the authors’ own institution to move toward similar methods of electronic data collection.

5.0 Conclusions

Implementation of a paper-based immediate PRO collection system in the outpatient IHM setting is feasible. Overall, collection of pre-encounter PROs was successful, while a smaller proportion of paired PROs were collected. Provider- and organization-level factors appeared to impact rates of any (pre- or paired) PRO completion more than patient-level factors.

Supporting information

S1 Table. Major Expanded Diagnosis Clusters (MEDCs) of patients attending at least one non-massage IHM encounter.

https://doi.org/10.1371/journal.pone.0303985.s001

(DOCX)

S2 Table. Adjusted odds ratios (aOR) for completing any pre-encounter PRO questionnaire.

https://doi.org/10.1371/journal.pone.0303985.s002

(DOCX)

S3 Table. Adjusted odds ratios (aOR) for completing any post-encounter PRO questionnaire following the completion of a pre-encounter PRO questionnaire.

https://doi.org/10.1371/journal.pone.0303985.s003

(DOCX)

Acknowledgments

We extend our sincere gratitude to all the UHCWH patients who received the integrative health and medicine modalities analyzed in this study. We would also like to thank the following collaborators on this study: The Information Technology Senior Developers Carl Langdon and Ryan Jaskolka; study coordinators Tracy Segall, MSHS and Jessica Surdam, MPH, CCRC; Operations Manager Vanessa Lapohn, MPH; the patient service representatives; and all participating integrative health and medicine practitioners. We especially appreciate the generous philanthropic support of Chris and Sarah Connor.

Citation: Srinivasan R, Rodgers-Melnick SN, Rivard RL, Kaiser C, Vincent D, Adan F, et al. (2024) Implementing paper-based patient-reported outcome collection within outpatient integrative health and medicine. PLoS ONE 19(5): e0303985. https://doi.org/10.1371/journal.pone.0303985

About the Authors:

Roshini Srinivasan

Roles: Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America, Duke University School of Medicine, Durham, NC, United States of America

ORICD: https://orcid.org/0000-0003-1967-4630

Samuel N. Rodgers-Melnick

Roles: Data curation, Formal analysis, Visualization, Writing – original draft, Writing – review & editing

Affiliations: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America, Department of Population and Quantitative Health Sciences, Case Western Reserve University School of Medicine, Cleveland, OH, United States of America

ORICD: https://orcid.org/0000-0003-0886-253X

Rachael L. Rivard

Roles: Data curation, Formal analysis, Writing – review & editing

Affiliations: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America, Center for Evaluation Survey and Research, HealthPartners Institute, Minneapolis, MN, United States of America

Christine Kaiser

Roles: Project administration

Affiliation: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America

David Vincent

Roles: Project administration

Affiliation: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America

Francoise Adan

Roles: Resources, Supervision

Affiliations: Connor Whole Health, University Hospitals of Cleveland, Cleveland, OH, United States of America, Department of Psychiatry, Case Western Reserve University School of Medicine, Cleveland, OH, United States of America

Jeffery A. Dusek

Roles: Conceptualization, Investigation, Methodology, Project administration, Software, Writing – review & editing

Affiliations: Susan Samueli Integrative Health Institute, University of California Irvine, Irvine, CA, United States of America, Department of Medicine, University of California Irvine, Irvine, CA, United States of America

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

References

1. Clarke TC, Black LI, Stussman BJ, Barnes PM, Nahin RL. Trends in the use of complementary health approaches among adults: United States, 2002–2012. Natl Health Stat Report. 2015;(79):1–16. pmid:25671660; PubMed Central PMCID: PMC4573565.

2. Health NCfCaI. Complementary, Alternative, or Integrative Health: What’s In a Name?2021 5 April 2024. Available from: https://www.nccih.nih.gov/health/complementary-alternative-or-integrative-health-whats-in-a-name.

3. Dyer NL, Surdam J, Dusek JA. A Systematic Review of Practiced-Based Research of Complementary and Integrative Health Therapies as Provided for Pain Management in Clinical Settings: Recommendations for the Future and a Call to Action. Pain Med. 2022;23(1):189–210. pmid:34009391; PubMed Central PMCID: PMC8932298.

4. Kligler B, Teets R, Quick M. Complementary/Integrative Therapies That Work: A Review of the Evidence. Am Fam Physician. 2016;94(5):369–74. pmid:27583423.

5. David A, Hausner D, Frenkel M. Cancer-Related Fatigue-Is There a Role for Complementary and Integrative Medicine? Curr Oncol Rep. 2021;23(12):145. Epub 20211107. pmid:34743258.

6. Brown ML. Integrative Approaches to Stress, Anxiety, and Resilience. Pediatr Ann. 2019;48(6):e226–e30. pmid:31185113.

7. Vickers AJ, Vertosick EA, Lewith G, MacPherson H, Foster NE, Sherman KJ, et al. Acupuncture for Chronic Pain: Update of an Individual Patient Data Meta-Analysis. J Pain. 2018;19(5):455–74. Epub 20171202. pmid:29198932; PubMed Central PMCID: PMC5927830.

8. Rubinstein SM, de Zoete A, van Middelkoop M, Assendelft WJJ, de Boer MR, van Tulder MW. Benefits and harms of spinal manipulative therapy for the treatment of chronic low back pain: systematic review and meta-analysis of randomised controlled trials. BMJ. 2019;364:l689. Epub 20190313. pmid:30867144; PubMed Central PMCID: PMC6396088.

9. Trager RJ, Cupler ZA, DeLano KJ, Perez JA, Dusek JA. Association between chiropractic spinal manipulative therapy and benzodiazepine prescription in patients with radicular low back pain: a retrospective cohort study using real-world data from the USA. BMJ Open. 2022;12(6):e058769. Epub 20220613. pmid:35697464; PubMed Central PMCID: PMC9196200.

10. Trager RJ, Cupler ZA, Srinivasan R, Casselberry RM, Perez JA, Dusek JA. Association between chiropractic spinal manipulation and gabapentin prescription in adults with radicular low back pain: retrospective cohort study using US data. BMJ Open. 2023;13(7):e073258. Epub 20230721. pmid:37479505; PubMed Central PMCID: PMC10364168.

11. Amorim D, Amado J, Brito I, Fiuza SM, Amorim N, Costeira C, et al. Acupuncture and electroacupuncture for anxiety disorders: A systematic review of the clinical research. Complement Ther Clin Pract. 2018;31:31–7. Epub 20180131. pmid:29705474.

12. Miri S, Hosseini SJ, Ghorbani Vajargah P, Firooz M, Takasi P, Mollaei A, et al. Effects of massage therapy on pain and anxiety intensity in patients with burns: A systematic review and meta-analysis. Int Wound J. 2023;20(6):2440–58. Epub 20230121. pmid:36680488; PubMed Central PMCID: PMC10333016.

13. Cramer H, Lauche R, Anheyer D, Pilkington K, de Manincor M, Dobos G, et al. Yoga for anxiety: A systematic review and meta-analysis of randomized controlled trials. Depress Anxiety. 2018;35(9):830–43. Epub 20180426. pmid:29697885.

14. Dyer NL, Surdam J, Srinivasan R, Agarwal A, Dusek JA. The Impact of Individualized Complementary and Integrative Health Interventions Provided in Clinical Settings on Quality of Life: A Systematic Review of Practice-Based Research. J Integr Complement Med. 2022;28(8):618–40. Epub 20220714. pmid:35834609; PubMed Central PMCID: PMC9419987.

15. Abrams DI, Dolor R, Roberts R, Pechura C, Dusek J, Amoils S, et al. The BraveNet prospective observational study on integrative medicine treatment approaches for pain. BMC Complementary and Alternative Medicine. 2013;13(1):1–9. pmid:23800144

16. Crocker RL, Hurwitz JT, Grizzle AJ, Abraham I, Rehfeld R, Horwitz R, et al. Real-world evidence from the Integrative Medicine Primary Care Trial (IMPACT): assessing patient-reported outcomes at baseline and 12-month follow-up. Evidence-Based Complementary and Alternative Medicine. 2019;2019. pmid:31346339

17. Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol. 2010;63(11):1179–94. Epub 20100804. pmid:20685078; PubMed Central PMCID: PMC2965562.

18. Weldring T, Smith SM. Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Health Serv Insights. 2013;6:61–8. Epub 20130804. pmid:25114561; PubMed Central PMCID: PMC4089835.

19. McAllister M, Dunn G, Payne K, Davies L, Todd C. Patient empowerment: the need to consider it as a measurable patient-reported outcome for chronic conditions. BMC Health Serv Res. 2012;12:157. Epub 20120613. pmid:22694747; PubMed Central PMCID: PMC3457855.

20. Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167. Epub 20130128. pmid:23358487.

21. Au HJ, Ringash J, Brundage M, Palmer M, Richardson H, Meyer RM, et al. Added value of health-related quality of life measurement in cancer clinical trials: the experience of the NCIC CTG. Expert Rev Pharmacoecon Outcomes Res. 2010;10(2):119–28. pmid:20384559.

22. Basch E, Barbera L, Kerrigan CL, Velikova G. Implementation of Patient-Reported Outcomes in Routine Medical Care. Am Soc Clin Oncol Educ Book. 2018;38:122–34. pmid:30231381.

23. Hahn EA, Cella D, Chassany O, Fairclough DL, Wong GY, Hays RD, et al. Precision of health-related quality-of-life data compared with other clinical measures. Mayo Clin Proc. 2007;82(10):1244–54. pmid:17908530.

24. Field J, Holmes MM, Newell D. PROMs data: can it be used to make decisions for individual patients? A narrative review. Patient Relat Outcome Meas. 2019;10:233–41. Epub 20190729. pmid:31534379; PubMed Central PMCID: PMC6681163.

25. Squitieri L, Bozic KJ, Pusic AL. The Role of Patient-Reported Outcome Measures in Value-Based Payment Reform. Value Health. 2017;20(6):834–6. Epub 20170322. pmid:28577702; PubMed Central PMCID: PMC5735998.

26. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas. 2018;9:353–67. Epub 20181101. pmid:30464666; PubMed Central PMCID: PMC6219423.

27. Cella D, Choi S, Garcia S, Cook KF, Rosenbloom S, Lai JS, et al. Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment. Qual Life Res. 2014;23(10):2651–61. Epub 20140618. pmid:24938431; PubMed Central PMCID: PMC4710358.

28. Eton DT, Beebe TJ, Hagen PT, Halyard MY, Montori VM, Naessens JM, et al. Harmonizing and consolidating the measurement of patient-reported information at health care institutions: a position statement of the Mayo Clinic. Patient Relat Outcome Meas. 2014;5:7–15. Epub 20140210. pmid:24550683; PubMed Central PMCID: PMC3926456.

29. Silveira A, Sequeira T, Gonçalves J, Lopes Ferreira P. Patient reported outcomes in oncology: changing perspectives-a systematic review. Health Qual Life Outcomes. 2022;20(1):82. Epub 20220521. pmid:35597948; PubMed Central PMCID: PMC9124403.

30. Hui D, Mastroleo GS, Rozman De Morales A, Peek A, Reddy A, Zhukovsky DS, et al. Implementation of Patient Reported Outcomes in Outpatient Palliative Care: From Paper to Computer. J Pain Symptom Manage. 2023;66(2):e197–e203. Epub 20230418. pmid:37080480.

31. Karamanidou C, Natsiavas P, Koumakis L, Marias K, Schera F, Schäfer M, et al. Electronic Patient-Reported Outcome-Based Interventions for Palliative Cancer Care: A Systematic and Mapping Review. JCO Clin Cancer Inform. 2020;4:647–56. pmid:32697604; PubMed Central PMCID: PMC7397776.

32. Nguyen MP, Rivard RL, Blaschke B, Vang S, Schroder LK, Cole PA, et al. Capturing patient-reported outcomes: paper versus electronic survey administration. OTA Int. 2022;5(3):e212. Epub 20220908. pmid:36349121; PubMed Central PMCID: PMC9580259.

33. Muehlhausen W, Doll H, Quadri N, Fordham B, O’Donohoe P, Dogar N, et al. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes. 2015;13:167. Epub 20151007. pmid:26446159; PubMed Central PMCID: PMC4597451.

34. Meirte J, Hellemans N, Anthonissen M, Denteneer L, Maertens K, Moortgat P, et al. Benefits and Disadvantages of Electronic Patient-reported Outcome Measures: Systematic Review. JMIR Perioper Med. 2020;3(1):e15588. Epub 20200403. pmid:33393920; PubMed Central PMCID: PMC7709853.

35. Hyland CJ, Guo R, Dhawan R, Kaur MN, Bain PA, Edelen MO, et al. Implementing patient-reported outcomes in routine clinical care for diverse and underrepresented patients in the United States. J Patient Rep Outcomes. 2022;6(1):20. Epub 20220307. pmid:35254556; PubMed Central PMCID: PMC8901833.

36. Foster A, Croot L, Brazier J, Harris J, O’Cathain A. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews. J Patient Rep Outcomes. 2018;2:46. Epub 20181003. pmid:30363333; PubMed Central PMCID: PMC6170512.

37. Glenwright BG, Simmich J, Cottrell M, O’Leary SP, Sullivan C, Pole JD, et al. Facilitators and barriers to implementing electronic patient-reported outcome and experience measures in a health care setting: a systematic review. J Patient Rep Outcomes. 2023;7(1):13. Epub 20230214. pmid:36786914; PubMed Central PMCID: PMC9928985.

38. Long C, Beres LK, Wu AW, Giladi AM. Patient-level barriers and facilitators to completion of patient-reported outcomes measures. Qual Life Res. 2022;31(6):1711–8. Epub 20210917. pmid:34533759.

39. Dusek JA, JaKa M, Wallerius S, Fairchild S, Victorson D, Rivard RL, et al. Rationale for routine collection of patient reported outcomes during integrative medicine consultation visits. Complementary therapies in medicine. 2018;37:43–9. Epub 20180131. pmid:29609936.

40. Dusek JA, Gao Q, Kim RS, Abrams DI, Kligler B, Dyer NL, et al. Patients Receiving Integrative Medicine Effectiveness Registry (PRIMIER) of the BraveNet practice-based research network: Outcomes of the PRIMIER cohort. Complementary therapies in medicine. 2022;71:102904. Epub 20221123. pmid:36435299.

41. Johnson JR, Crespin DJ, Griffin KH, Finch MD, Rivard RL, Baechler CJ, et al. The effectiveness of integrative medicine interventions on pain and anxiety in cardiovascular inpatients: a practice-based research evaluation. BMC Complement Altern Med. 2014;14:486. Epub 20141213. pmid:25494710; PubMed Central PMCID: PMC4301797.

42. Johnson JR, Crespin DJ, Griffin KH, Finch MD, Dusek JA. Effects of integrative medicine on pain and anxiety among oncology inpatients. J Natl Cancer Inst Monogr. 2014;2014(50):330–7. pmid:25749600; PubMed Central PMCID: PMC4411536.

43. Dusek JA, Griffin KH, Finch MD, Rivard RL, Watson D. Cost Savings from Reducing Pain Through the Delivery of Integrative Medicine Program to Hospitalized Patients. J Altern Complement Med. 2018;24(6):557–63. Epub 20180223. pmid:29474095; PubMed Central PMCID: PMC6006422.

44. Rodgers-Melnick SN, Block S, Rivard RL, Dusek JA. Optimizing Patient-Reported Outcome Collection and Documentation in Medical Music Therapy: Process-Improvement Study. JMIR Hum Factors. 2023;10:e46528. Epub 20230727. pmid:37498646; PubMed Central PMCID: PMC10415937.

45. Bokhour BG, Haun JN, Hyde J, Charns M, Kligler B. Transforming the Veterans Affairs to a Whole Health System of Care: Time for Action and Research. Med Care. 2020;58(4):295–300. pmid:32044866.

46. Trager RJ, Brewka EC, Kaiser CM, Patterson AJ, Dusek JA. Acupuncture in Multidisciplinary Treatment for Post-COVID-19 Syndrome. Med Acupunct. 2022;34(3):177–83. Epub 20220616. pmid:35821795; PubMed Central PMCID: PMC9248328.

47. Rodgers-Melnick SN, Rivard RL, Block S, Dusek JA. Clinical Delivery and Effectiveness of Music Therapy in Hematology and Oncology: An EMMPIRE Retrospective Study. Integr Cancer Ther. 2022;21(21):15347354221142538. pmid:36510393; PubMed Central PMCID: PMC9751180.

48. Rodgers-Melnick SN, Bartolovich M, Desai NJ, Vasanna SH, Rivard RL, Owusu-Ansah A, et al. Massage therapy for children, adolescents, and young adults: Clinical delivery and effectiveness in hematology and oncology. Pediatric Blood & Cancer. 2023;70(4):e30243. Epub 20230201. pmid:36726036.

49. Shi Y, Nguyen T, Farrell M, Rodgers-Melnick S, Moss G, Kaiser C, et al. Retrospective Outcomes of a New Acupuncture Service at a Comprehensive Cancer Center. J Integr Complement Med. 2023;29(10):674–82. Epub 20230526. pmid:37252748.

50. Rodgers-Melnick SN, Rivard RL, Block S, Dusek JA. Effectiveness of Medical Music Therapy Practice: Integrative Research Using the Electronic Health Record: Rationale, Design, and Population Characteristics. J Integr Complement Med. 2024;30(1):57–65. Epub 20230711. pmid:37433198; PubMed Central PMCID: PMC10795501.

51. Rodgers-Melnick SN, Rivard RL, Block S, Dusek JA. Effectiveness of music therapy within community hospitals: an EMMPIRE retrospective study. PAIN Reports. 2023;8(3):e1074. pmid:37731473

52. Alghadir AH, Anwer S, Iqbal A, Iqbal ZA. Test-retest reliability, validity, and minimum detectable change of visual analog, numerical rating, and verbal rating scales for measurement of osteoarthritic knee pain. J Pain Res. 2018;11:851–6. Epub 20180426. pmid:29731662; PubMed Central PMCID: PMC5927184.

53. Prokopowicz A, Stanczykiewicz B, Uchmanowicz I. Validation of the Numerical Anxiety Rating Scale in postpartum females: a prospective observational study. Ginekol Pol. 2022;93(9):686–94. Epub 20220124. pmid:35072220.

54. Karvounides D, Simpson PM, Davies WH, Khan KA, Weisman SJ, Hainsworth KR. Three studies supporting the initial validation of the stress numerical rating scale-11 (Stress NRS-11): A single item measure of momentary stress for adolescents and adults. Pediatr Dimens. 2016;1(4):105–9.

55. Kluetz PG, Slagle A, Papadopoulos EJ, Johnson LL, Donoghue M, Kwitkowski VE, et al. Focusing on Core Patient-Reported Outcomes in Cancer Clinical Trials: Symptomatic Adverse Events, Physical Function, and Disease-Related Symptoms. Clin Cancer Res. 2016;22(7):1553–8. Epub 20160112. pmid:26758559.

56. Robles B, Upchurch DM, Kuo T. Comparing Complementary and Alternative Medicine Use with or without Including Prayer as a Modality in a Local and Diverse United States Jurisdiction. Front Public Health. 2017;5:56. Epub 20170321. pmid:28377918; PubMed Central PMCID: PMC5359242.

57. Ruseckaite R, Mudunna C, Caruso M, Ahern S. Response rates in clinical quality registries and databases that collect patient reported outcome measures: a scoping review. Health Qual Life Outcomes. 2023;21(1):71. Epub 20230711. pmid:37434146; PubMed Central PMCID: PMC10337187.

58. Klinger EV, Carlini SV, Gonzalez I, Hubert SS, Linder JA, Rigotti NA, et al. Accuracy of race, ethnicity, and language preference in an electronic health record. Journal of general internal medicine. 2015;30:719–23. pmid:25527336

59. Wickham H. stringr: Simple, Consistent Wrappers for Common String Operations. 2022.

60. Bishop FL, Lewith GT. Who Uses CAM? A Narrative Review of Demographic Characteristics and Health Factors Associated with CAM Use. Evid Based Complement Alternat Med. 2010;7(1):11–28. Epub 20080313. pmid:18955327; PubMed Central PMCID: PMC2816378.

61. Saper R. Integrative Medicine and Health Disparities. Glob Adv Health Med. 2016;5(1):5–8. Epub 20160101. pmid:26937308; PubMed Central PMCID: PMC4756775.

62. QuickFacts Cuyahoga County, Ohio; United States. United States Census Bureau. 2022.

63. Corp N, Jordan JL, Croft PR. Justifications for using complementary and alternative medicine reported by persons with musculoskeletal conditions: A narrative literature synthesis. PLoS One. 2018;13(7):e0200879. Epub 20180719. pmid:30024952; PubMed Central PMCID: PMC6053199.

64. Murthy V, Sibbritt DW, Adams J. An integrative review of complementary and alternative medicine use for back pain: a focus on prevalence, reasons for use, influential factors, self-perceived effectiveness, and communication. Spine J. 2015;15(8):1870–83. Epub 20150509. pmid:25962340.

65. Schamber EM, Takemoto SK, Chenok KE, Bozic KJ. Barriers to completion of Patient Reported Outcome Measures. J Arthroplasty. 2013;28(9):1449–53. Epub 20130725. pmid:23890831.

66. Stehlik J, Rodriguez-Correa C, Spertus JA, Biber J, Nativi-Nicolau J, Zickmund S, et al. Implementation of Real-Time Assessment of Patient-Reported Outcomes in a Heart Failure Clinic: A Feasibility Study. J Card Fail. 2017;23(11):813–6. Epub 20171002. pmid:28982635.

67. Gelkopf M, Mazor Y, Roe D. A systematic review of patient-reported outcome measurement (PROM) and provider assessment in mental health: goals, implementation, setting, measurement characteristics and barriers. Int J Qual Health Care. 2021;34(Suppl 1):ii13–ii27. pmid:32159763.

68. Rice RE. Influences, usage, and outcomes of Internet health information searching: multivariate results from the Pew surveys. Int J Med Inform. 2006;75(1):8–28. Epub 20050824. pmid:16125453.

69. Glass DC, Kelsall HL, Slegers C, Forbes AB, Loff B, Zion D, et al. A telephone survey of factors affecting willingness to participate in health research surveys. BMC Public Health. 2015;15:1017. Epub 20151005. pmid:26438148; PubMed Central PMCID: PMC4594742.

70. Rose M, Devine J. Assessment of patient-reported symptoms of anxiety. Dialogues Clin Neurosci. 2014;16(2):197–211. pmid:25152658; PubMed Central PMCID: PMC4140513.

71. Meister TL, Dreismeier M, Blanco EV, Brüggemann Y, Heinen N, Kampf G, et al. Low Risk of Severe Acute Respiratory Syndrome Coronavirus 2 Transmission by Fomites: A Clinical Observational Study in Highly Infectious Coronavirus Disease 2019 Patients. J Infect Dis. 2022;226(9):1608–15. pmid:35512326; PubMed Central PMCID: PMC9129150.

72. Sisodia RC, Dankers C, Orav J, Joseph B, Meyers P, Wright P, et al. Factors Associated With Increased Collection of Patient-Reported Outcomes Within a Large Health Care System. JAMA Netw Open. 2020;3(4):e202764. Epub 20200401. pmid:32286657; PubMed Central PMCID: PMC7156989.

73. Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H, Malizia W. Capturing Patient-Reported Outcome (PRO) Data Electronically: The Past, Present, and Promise of ePRO Measurement in Clinical Trials. Patient. 2015;8(4):301–9. pmid:25300613; PubMed Central PMCID: PMC4529477.

74. Ganser AL, Raymond S. A., & Pearson J. DsdData quality and power in clinical trials: a comparison of ePRO and paper in a randomized trial: Routledge; 2016.

75. Dale O, Hagen KB. Despite technical problems personal digital assistants outperform pen and paper when collecting patient diary data. J Clin Epidemiol. 2007;60(1):8–17. Epub 20060830. pmid:17161749.

76. Greenwood MC, Hakim AJ, Carson E, Doyle DV. Touch-screen computer systems in the rheumatology clinic offer a reliable and user-friendly means of collecting quality-of-life and outcome data from patients with rheumatoid arthritis. Rheumatology (Oxford). 2006;45(1):66–71. Epub 20051101. pmid:16263782.