Correspondence to Dr Sofia F Garcia; [email protected]
Strengths and limitations of this study
This study uses a type 2 hybrid effectiveness-intervention design, allowing for simultaneous systemwide implementation of a quality improvement initiative and an effectiveness evaluation.
This study is implementing a patient-reported symptom monitoring system that is integrated within the electronic health record and clinical workflows in order to minimise staff burden and promote sustainability.
This study evaluates implementation and impact of an electronic symptom-monitoring programme in one healthcare system, which may limit the generalisability of our findings to comparable high-volume, well-resourced academic health systems.
The stepped wedge study design, while practical and highly acceptable to the healthcare system, limits the ability to mask conditions, as implementers and patients are aware of the change to screening, and a delayed intervention effect in any cluster could reduce power.
Introduction
Context
Advances in screening and early detection, and more successful treatment options, have led to an unprecedented number of people surviving cancer. There are currently almost 17 million cancer survivors in the USA, and that number is expected to exceed 22 million by 2030.1 Cancer is now characterised as a chronic, manageable condition, requiring specific and targeted comprehensive efforts to address long-term challenges and late effects of treatment. Despite advances in early detection and treatment success that extends longevity, survival benefit is often offset by debilitating cancer-related and treatment-related symptoms and psychosocial sequelae that compromise health-related quality of life (HRQoL).2
A growing body of literature has documented the needs of oncology patients, providing evidence that psychological and physical concerns are both prevalent and persistent.3 About 32% of patients with cancer have been shown to meet criteria for mental health conditions.2 4 In a meta-analysis of 70 studies with over 10 000 oncology patients in ambulatory settings, 16.3%, 10.3% and 19.4% met Diagnostic and Statistical Manual of Mental Disorders (DSM 5) criteria for major depression, adjustment and anxiety disorders, respectively; 38.2% met criteria for any psychological diagnosis.5 6 Physical symptoms such as fatigue, pain and poor physical function are among the most common and debilitating reported in oncology settings.7–9 On treatment completion, physical needs (eg, pain and nutrition) are among the top unmet needs.10 Other concerns such as practical needs (eg, transportation, childcare, stress management) are also common.11
In recognition of these challenges, key leadership organisations have prioritised the need to address and embed symptom screening with a referral process in ambulatory cancer care.12–14 This includes standards to better identify, monitor and manage patients’ health needs, including referral to supportive oncology care.15–19 However, work evaluating clinical management and intervention programmes that address the unique needs of oncology patients remain limited and poorly integrated within most institutions.20–23
Related Research
Patient-reported symptom screening has been found to be feasible and efficacious in ambulatory oncology. In a randomised trial of 286 patients with cancer, ongoing monitoring of HRQoL prior to clinical encounters, relative to no monitoring, was associated with better HRQoL over time and improved patient–physician communication.24 Our team has documented high patient compliance (92%) in a technology-based monitoring system developed for patients with lung cancer starting chemotherapy that assessed relevant patient-reported outcomes (PROs) such as fatigue, dyspnoea, cough, weight loss, anorexia, pain, insomnia, change in mental status and psychological distress.25 The majority (69%) of patients felt the questionnaire helped them focus on issues to be discussed with their physicians and, similarly, physicians indicated the reports from the monitoring system helped track and compare symptom burden over time. Recently, a web-based programme that allowed patients to report symptoms to their clinicians was associated with improved HRQoL and longer survival within a randomised single-centre trial.26–28 Patients in the self-reporting arm (relative to usual care) reported greater HRQoL at 6 months postbaseline assessment and had a 5-year absolute survival benefit of 5 months. While promising, most of the work evaluating the efficacy of systematically capturing and addressing PROs via the electronic health record (EHR) remains limited to controlled trials, with limited generalisability to health systemwide application.29 In general, most previous studies evaluating the efficacy of symptom monitoring: (a) are limited in their scalability, generalisability and implementation; (b) implemented measures limited in regards to sensitivity and specificity; (c) did not evaluate the impact of symptom monitoring on clinic workflows or system-level outcomes or (d) did not evaluate or address implementation of the programme as a standard of care.
Preliminary Work
To answer the need for comprehensive symptom assessment that leverages health information technology to reach patients in a feasible manner, we developed and piloted an electronic PRO assessment specific to cancer (‘cPRO’ for cancer patient-reported outcomes) within Northwestern Medicine’s primary EHR, specifically, Epic Systems medical record software (Epic, Verona, Wisconsin).
cPRO development
The cPRO system was custom designed to electronically administer validated PROs from the Patient-reported Outcomes Measurement Information System (PROMIS)30 31 that assess key health outcomes in patients with cancer (depression, anxiety, fatigue, pain interference and physical function) across the trajectory of care and a checklist to identify practical and supportive care needs (eg, financial and transportation concerns; nutritional support). PROMIS measures are administered as computer adaptive tests (CATs), allowing for assessment efficiency and precision.32 Assessment invitations are automated and launched 72 hours prior to scheduled medical oncology appointments (limited to once a month) and completed by patients via the EHR patient portal (Epic MyChart) prior to their visits.33 Results are scored and immediately available in the EHR to inform clinical communications and decision-making. Severe symptoms trigger notifications to clinicians (via the Epic inbox) who can then communicate with patients and make necessary referrals and care decisions in real time (figure 1). Alerts are addressed by clinicians via MyChart, telephone or in-person contacts.
Enlarge this image.Figure 1. EHR-integrated symptom and needs assessment (cPRO) system. cPRO, cancer patient-reported outcomes; EHR, Electronic health-record.
cPRO pilot studies
Two clinical quality improvement initiatives were conducted to assess the feasibility of implementing the cPRO system as a standard of oncology care at Northwestern Medicine. In the first, 636 women receiving gynaecologic oncology outpatient care received invitations and completed at least one symptom and need assessment through their Epic MyChart portal.34 In the second, 6825 adult oncology outpatients received invitations to complete an earlier version of the cPRO assessment through their Epic MyChart portal; 3526 (51%) completed at least one assessment.33 Assessments were launched 72 hours prior to scheduled medical oncology appointments (limited to once a month). Patients whose symptoms scored in the ‘severe’ range or endorsed supportive care needs (psychosocial and nutrition-related concerns) triggered alerts via Epic in-box messaging to the oncology care team for intervention according to clinic workflow. Together, these pilot studies demonstrated a successful integration of PRO and need assessment administration, scoring and reporting within an EHR system, implemented within a specialised oncology clinic and then more broadly across medical oncology clinics at one geographic site. EHR integration enabled standardised routine assessment and real-time reporting of patient-reported symptoms and needs within ambulatory cancer care, towards the goal of improving care quality and efficiency.
Current Work
The current study seeks to expand on our previous cPRO assessment implementation by expanding to medical oncology patients at all regions of a healthcare system and to patients who do not regularly use the EHR patient portal, by conducting in-clinic assessments using eHealth tools (eg, iPads). We will also conduct a formal evaluation of the programme’s implementation and effectiveness.
Study Aims
Our approach to the expansion, implementation and evaluation of a patient-reported symptom and supportive care need monitoring programme specific to cancer (cPRO) across all adult ambulatory cancer care clinics in a large urban healthcare system (Northwestern Medicine; serving >8000 new oncology patients yearly) is informed by the Framework for Spread35 and the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework.36 Specific aims include:
Aim 1
Expansion and Implementation Evaluation: the Framework for Spread will guide the implementation process to expand cPRO to reach patients at all Northwestern Medicine-affiliated ambulatory cancer clinics and allow for both at-home and in-clinic symptom assessment prior to medical visits. A mixed methods evaluation of implementation success will adhere to RE-AIM36 and its extension to enhance health equity and sustainment.37 The consolidated framework for implementation research (CFIR)38 will be used to assess and characterise implementation determinants at multiple levels of the system.
Aim 2
Evaluation of Effectiveness: we will evaluate the effectiveness of systemwide cPRO implementation on outcomes at the system and patient levels over 12 months via a quality improvement study (estimated n=4000 cases) and a human subjects substudy (n=1000 patients), respectively.
Aim 3
Identify Implementation Facilitators and Barriers: we will identify implementation facilitators and barriers to systemwide expansion of cPRO via qualitative research, gathering feedback from clinicians, administrators and patients participating in the symptom monitoring programme expansion.
Methods and analysis
Overall study design
This 4-year study (anticipated September 2018 to May 2022) uses a cluster randomised stepped wedge trial with a type two hybrid effectiveness-implementation design to test the expansion of cPRO across oncology care practices in a large healthcare system (figure 2).
Enlarge this image.Figure 2. A 4 year, type 2 hybrid effectiveness-implementation design to test the expansion and implementation of cPRO across oncology care practices in a large healthcare system. cPRO, cancer patient-reported outcomes.
The first aim (aim 1) will focus on the expansion of the symptom monitoring programme (cPRO) using health information technology (configuration and enhancement of technical workflows for the symptom assessment to allow for at-home and in-clinic assessment). Work will focus on the execution of the plan for implementation spread across all network medical oncology clinics using the Framework for Spread, which provides key strategic considerations and goals for implementing a systemwide change that will be evaluated using EHR and stakeholder survey data aligning with the domains of RE-AIM.
The second aim of the study will centre on two evaluations of the effects of implementation: a quality improvement protocol (aim 2a) to compare the impact of cPRO use on EHR-documented healthcare usage and patient satisfaction at the system level, using a stepped wedge design, in which clusters of study sites will sequentially be assigned to cross from serving as a control setting (preimplementation) to implementing cPRO; and a human subjects substudy (aim 2b) with patients who will complete the symptom screener and a battery of measures at baseline, 6 and 12 months to evaluate the impact of cPRO on patient-reported healthcare utilisation, quality, symptoms and HRQoL.
In aim 3, qualitative methods will be used to identify facilitators and barriers to systemwide expansion and adoption of cPRO. Patients will be invited to participate in focus groups or individual semistructured interviews designed to solicit feedback on experiences with cPRO. Key stakeholders (clinicians, administrators) will be invited to complete surveys designed to evaluate key measures of successful implementation (eg, acceptability, appropriateness and perceived sustainability of the intervention).
Study setting
Research will occur at outpatient oncology settings across multiple hospitals that are part of a single healthcare system, Northwestern Medicine. Existing regional clusters (Central, North and West) within Northwestern Medicine serve as the clusters for the stepped wedge trial. The central region includes a single, large, urban-based medical centre; the North and West regions are each comprised of smaller hospitals (two and four, respectively) in suburban communities. All regions include specialty clinics for the diagnosis and management of cancer.
Study population
For the implementation component of this study (aim 1: cPRO administration within clinical care), the study population includes any adult outpatient receiving or clinician (physician, nurse, social worker, dietician) administering, cancer care at a participating medical oncology clinic as well as clinic administrative staff. For the evaluation component of this study, inclusion criteria vary by aim and participant population. For aims 2a and 2b and aim 3, eligible patients must have a confirmed cancer diagnosis and have received oncology services within the past 12 months. Additional criteria for aim 2b, patient eligibility includes recent completion of a cPRO assessment and authorisation for access to the patients’ disease and treatment information in the EHR. For aim 3, participants must have received at least four invitations to complete cPRO. An additional criterion regarding actual number of completed screeners defines focus group assignment: patients who have had one or more cPRO clinical alerts are assigned to participate in an individual interview. For aim 3, healthcare clinicians and administrators must be working at a site participating in the cPRO implementation.
Sample selection
For aim 2a, Enterprise Data Warehouse (EDW) queries of the EHR system will be performed on all cases with a completed symptom assessment within the year prior to the go-live date for implementation across each region. The rationale is to have sufficient data for comparison with our implementation period for both the number of patients as well as across a calendar year to examine the presence of seasonal trends.
For aim 2b and aim 3, patients will be recruited via e-mail invitation from among those receiving invitations to complete the cPRO assessment. For aim 3, clinicians and administrators will be recruited from the pool participating in the cPRO implementation, at the point of their in-person cPRO training and/or via e-mail invitation. Recruitment for both patients and healthcare system stakeholders will happen across regions to ensure representation from each geographic site.
Study measures related to effectiveness and implementation outcomes
The cPRO assessment consists of PROMIS CATs30 31 39 of (1) Depression (PROMIS Item Bank V.1.0—Depression); (2) Anxiety (PROMIS Item Bank V.1.0—Anxiety); (3) Fatigue (PROMIS Item Bank—Fatigue V.1.0); (4) Pain Interference (PROMIS Item Bank V.1.1—Pain Interference) and (5) Physical Function (PROMIS Item Bank V.1.1—Physical Function), along with supportive care checklist items (covering psychosocial and nutritional needs). Cancer centre patients are asked to complete a screener before each oncology visit (but no more than once a month). Data related to cPRO completion, scores and alerts will be used in all study aims, primarily to evaluate the effects of the symptom monitoring system on severity of patient-reported symptoms related to cancer and cancer treatment.
For aim 2a, we will obtain data collected independently from a hospital-based Press Ganey Patient Experience survey40 and a Medallia customer experience questionnaire41 to assess patient satisfaction with their care experience. Patients are provided the opportunity to complete the survey after a care experience (appointment or hospitalisation). Patient-level healthcare utilisation data (eg, clinical notes, emergency room visits and hospitalisations) will be extracted from the EHR to help evaluate effectiveness of the intervention in terms of resource utilisation. For aim 2b, participants will complete questionnaires to assess intervention outcomes related to HRQoL, healthcare quality, symptom experience, financial toxicity, healthcare utilisation (including healthcare utilisation outside of the Northwestern Medicine (NM) health system, shared decision-making and health literacy (see table 1 for Patient-reported Effectiveness Outcomes and Measures).
Table 1Patient-reported effectiveness outcomes and measures (aim 2b)
| Outcome | Measure | Items | Measure details | Assessment |
| Health-related quality of life: Effects of the symptom monitoring system on patient-reported quality of life related to cancer | Functional Assessment of Cancer Therapy—General—7 Item Version (FACT-G7)65 | 7 items | The FACT-G-7 is a brief validated measure of patient-reported priority symptoms in cancer; The FACT-G7 has demonstrated internal consistency reliability, convergence and known-groups validity and is highly correlated with the parent measure (FACT-G) total score.65 | Baseline, 6 and 12 months |
| Healthcare quality: Impact of the symptom monitoring system on patient experiences with their cancer care team | Select items from the Consumer Assessment of Healthcare Providers and Systems CAHPS Cancer Care Survey66 67 | 12 items (if endorsed up to 3 more) | CAHPS is a survey system designed to capture patient experiences with their cancer care team; a rigorously developed, well-tested, reliable and valid survey of patient experiences with their cancer care.67 | Baseline, 6 and 12 months |
| Healthcare utilisation: Impact of the symptom monitoring system on healthcare services used by patients | Custom measure designed to measure assess healthcare utilisation outside of the Northwestern system | 3 items | Baseline, 6 and 12 months | |
| Symptom burden: Effects of the symptom monitoring system on patient-reported adverse events related to cancer | Select items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)68 | 6 items (if endorsed, up to 4 more) | PRO-CTCAE is a compendium of PRO items uniquely targeted to symptomatic treatment-related toxicity assessment in oncology care; Published data substantiates content and construct validity, reliability and responsiveness.69 70 | Baseline, 6 and 12 months |
| Financial toxicity of cancer care | Summary item from FACIT Measure of Financial Toxicity (FACIT-COST)71 | 1 item | The last (overall summary) item of an 11-item questionnaire that measures personal financial burden of care. | Baseline, 6 and 12 months |
| Reading ability (component of health literacy) | Single Item Literacy Screener (SILS)72 | 1 item | A simple assessment of a person’s ability to read and understand printed health material; The SILS ‘performs moderately well at ruling out limited reading ability in adults.’72 | Baseline |
| Shared decision-making | CollaboRATE survey73 | 3 items | A brief patient survey designed to assess the perceived extent of shared decision-making in a given clinical encounter; The measure has demonstrated discriminative and concurrent validity, interrater reliability and sensitivity to change.73 | Baseline, 6 and 12 months |
Patients will be asked to complete the battery assessment at baseline, 6 and 12 months via an electronic survey administered using the Research Electronic Data Capture (REDCap) platform.42 REDCap43 is a secure and flexible web application that is available both online and offline, supports longitudinal data collection and allows for data exports to common data analysis packages.
For aim 3, carefully designed focus group and individual interview guides and self-administered surveys, informed by the CFIR Interview Guide,38 will be developed and used to facilitate data collection from clinicians, administrators and patients. To assess the related barriers and facilitators to using cPRO, patients will be invited to provide targeted feedback about their experience with cPRO (ease of navigation and completion, comprehension of purpose and goals, general experience with care team and communication related to the symptom and needs assessment) at one point in time across two waves of qualitative data collection (short-term or long-term postimplementation). Most will be invited to participate in a focus group; patients who had one or more ‘alerts’ will be individually interviewed. For aim 3, surveys with clinicians and administrators, we will assess targeted implementation barriers and facilitators and implementation process domains from the Framework for Spread, including salient constructs of implementation leadership support, implementation climate and sustainability (see table 2 for clinician-reported and administrator-reported Implementation Outcomes and Measures). Surveys will be administered electronically at three time points: baseline (preimplementation) and then 3-month and 7 month postimplementation.
Table 2Implementation outcomes and measures (aim 3)
| Outcome variable(s) | Measure | Items | Measure details | Assessment |
| Organisational culture (Do clinicians, researchers and staff believe that implementing cancer patient-reported outcome (cPRO) is appropriate and beneficial for the patients, the practice, and themselves?) | Organisational Change Recipients’ Beliefs Scale (OCRBS)74 | Five items from the ‘Appropriateness’ subscale | The OCRBS has good to excellent reported internal consistency reliability (eg, α=0.89–0.95 reported across several studies)74 and includes item indicators such as ‘the change we implemented was correct for our organisation’. | Baseline (at the point of regional intervention), and 3 and 7 months postimplementation |
| Leadership support (Does staff feel supported by Northwestern Medicine (NM) leadership to implement cPRO in their practice? and Is the leadership proactive, supportive, knowledgeable and perseverant? | Implementation Leadership Scale (ILS) to assess the degree to which a leader exhibits specific supportive behaviors75 | Three items from the ‘Supportive Leadership’ subscale | The ILS has excellent reported internal consistency reliability (eg, α=0.95; Aarons et al) and includes item indicators such as ‘supports employee efforts to use evidence-based practice’. | Baseline (at the point of regional intervention), and 3 and 7 months postimplementation |
| Acceptability, appropriateness and feasibility (Do physicians, staff and leaders find cPRO acceptable, appropriate and feasible for their practice?) | The Normalisation MeAsure Development questionnaire (NoMAD)76 | 23 items | The NoMAD is anticipated to have acceptable internal consistency reliability (ie, α≥0.70)77 and includes item indicators such as ‘I can see the potential value of cPRO for my work’. | Baseline (at the point of regional intervention), and 3 and 7 months postimplementation |
| Training experience related to cPRO (Do physicians, staff and leaders find cPRO training experience effective) | CBH Post-Training Survey78 | 6 items | The Training Survey is anticipated to have acceptable internal consistency reliability (ie, α≥0.70) and includes item indicators such as (‘the training prepared me for my role in cPRO’). | Post-training and 3 months postimplementation |
| Sustainability | Clinical Sustainability Assessment Tool (CSAT)-Short Form79 80 | 21 items | The CSAT includes items related to seven domains perceived by stakeholders to determine sustained implementation. It has shown to be reliable, usable and valid in a pilot study (n=126). | 7 months postimplementation |
Interventions to be measured
In addition to the cPRO intervention, this trial will measure the impact of a multicomponent implementation strategy. Implementation strategies are the methods and approaches used to support adoption and delivery of healthcare interventions in practice.44 Our approach comprises a number of discrete strategies that target multiple levels of the delivery system, including oncologists, implementation leaders/operational managers, workflows and internal monitoring of use. These include developing stakeholder interrelationships, training and educating stakeholders, engaging consumers, using evaluative and iterative strategies, and changing infrastructure. While the majority of these discrete strategies were prospectively proposed, we will carefully and comprehensively track strategy use over time using the Longitudinal Implementation Strategy Tracking System (LISTS).45 LISTS involves a time-line follow back procedure, in which members of the research and implementation teams meet at least quarterly throughout the project period to report on all dimensions of the Proctor, Powell and McMillen46 reporting standards for implementation strategies being used in the trial. This repeated evaluation of strategies allows for specifying when strategies are modified from their original planned usage (eg, discontinued, changed) using dimensions from the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS)47 and when new strategies are added, either as planned or in response to emergent barriers or effects of other strategies in use. Reporting also specifies with which cluster of the study the strategy is used/modified. This evaluation will aid in the interpretation of the trial results and inform replication efforts in other healthcare systems.
Sample size calculation
For aim 2 a, we plan to include all EHR ‘cases’ with completed symptom screener data during the study period of postimplementation, estimated at 4000 based on data from our pilot initiative. With 4000 cases, we will have the power to compare patient groups by specific trigger alerts (eg, PROMIS Anxiety) versus those who do not receive that alert. All cases will be analysed within the context of the proposed stepped wedge design.48 For aim 2b, we plan to enrol 1000 patients. These participants will be stratified by region, with site-specific accrual goals based on region size (oncology patient population). The primary outcome measures for this aim will be EHR and self-reported healthcare utilisation and patient satisfaction (see table 1 for a complete list of patient-reported outcome measures). We hypothesise that patients with triggered alerts will use more services and will be more satisfied with their care. Assuming 80% power, alpha (two tailed) of 5%, two equal-sized groups and a small effect (0.20), our proposed sample (n=1000) exceeds the required sample size of 785.
For aim 3, we plan to invite all clinicians and administrators involved in the cPRO implementation to complete the survey and expect to enrol a minimum of 40 across cancer clinics. Also, for aim 3, we plan to enrol at least 50 patients recruited from among those completing the symptom assessment. Focus groups will include roughly 6–10 participants, which is considered an appropriate size to maximise group dynamics and information sharing.49
Data analysis plan
Aim 2: evaluate the impact of cPRO
Aims 2 a/2b—descriptive statistics, reliability and CAT analyses. For all PRO measures, we will compute descriptive statistics at the item level (mean, SD, range, skewness, kurtosis, per cent of responses at floor/ceiling), flagging items exhibiting problems, for example, little or no variability. At the scale level, we will compute interitem and item-total correlations, checking for items with low construct validity, for example, item-total correlations <0.40. We will estimate Cronbach’s alpha to determine internal consistency reliability, using alpha ≥0.70 as a standard for making group-level comparisons. For CATs, we will examine how many participants reach the set SE stopping criterion for highly reliable scores (ie, SE <0.30 on the theta metric; reliability ≥0.91). If a substantial number of scores are unreliable (eg, 20% at reliability <0.70), we will investigate whether the issue is score level-specific, that is, associated with very low or very high (extreme) domain scores, which would represent a domain-targeting issue.
Aim 2 a—analysis of cPRO implementation impact. To evaluate the impact of cPRO on EHR-documented healthcare usage and patient satisfaction, we will use a stepped wedge48 (or dynamic waitlist) design,50 in which clusters of study sites will be assigned to cross over from serving as a control to implementing cPRO in a prospective, staggered manner. A priori, we created three clusters corresponding to the three NM regions; order of stepped cluster enrolment will be determined randomly. Our preliminary data analysis will include exploratory data inspection to identify general relationships as well as potential problems with asymmetry, extreme observations or outliers. We will examine differences in patient characteristics (eg, cancer type, cancer stage, demographic variables) to determine the comparability of intervention and control patients. Analysis details. We will use generalised linear mixed models (GLMM) with an appropriate link function specific to a dependent variable’s distribution (eg, an identity link for continuous outcomes; a logit link for binary outcomes). These models will include random effects to account for clusters, while simultaneously examining the intervention’s fixed effects. We will also include in our models covariates of specific interest (eg, age, race/ethnicity, gender, cancer stage/type); however, covariates will not be assessed for inferential purposes. We will run GLMM models to compare intervention to control groups on the primary outcomes of EHR-documented health service usage and Press Ganey/Medallia patient satisfaction, both of which will be treated as continuous variables.
Aim 2b: ePRO longitudinal analyses to explore trajectories of ePRO scores
Analysis 1—individual change trajectories. We will estimate individual trajectories of PROMIS Depression, Anxiety, Fatigue, Pain Interference and Physical Function scores as collected at three time points (baseline, 6, 12 months). We will graphically display the individual trajectories and determine the amount of variability associated with their intercepts (start parameter) and slopes (change parameter). Analysis 2—latent growth curves. Using growth mixture modelling (GMM), we will characterise the overall trajectory of each PROMIS domain score set by estimating its start and change parameters. We will then investigate the potential model impacts of gender, region and cPRO alert status (triggered alert vs did not) by including them as trajectory model explanatory variables in order to answer the following questions: Do gender and/or region statistically significantly (p<0.05) explain start or change parameter variability? Does cPRO alert status statistically significantly explain change parameter variability? In secondary analyses, we will investigate potential trajectory impacts of major clinical status variables such as cancer site. Analysis 3—latent classes. We will estimate potential subpopulations with distinct trajectories per set of PROMIS domain scores. For each identified subpopulation or ‘class’, we will characterise membership by (a) demographic, clinical and regional status and (b) healthcare utilisation and satisfaction with healthcare. Using multifactor analysis of variance, we will test whether characterisation variables statistically significantly (p<0.05) explain identified class membership differences. We hypothesise that (1) cPRO alert status will impact change parameters; (2) geographic region will not impact start or change parameters and (3) there will be sufficient heterogeneity in our studied sample to identify and define two or more latent patient classes per PROMIS domain. Analysis details. GMM assumes the population of interest consists of heterogeneous subpopulations with varying growth parameters.51 A GMM framework allows for unobserved heterogeneity in a sample and different individuals belonging to distinct subpopulations52; it enables identifying and characterising such subpopulations, known or unknown.53–55 Latent growth curve change parameter(s) and model assessment strategies include: (a) assessment of growth curve form, (b) determination of the statistical significance of the mean and variance of linear and non-linear slope terms, (c) inspection of overall model fit improvement when adding non-linear slope parameters, using χ2-based fit statistics (eg, the log likelihood ratio) and their associated p values and the root mean square error of approximation fit statistic.56 A bootstrap likelihood ratio test (BLRT) has been evaluated and appears promising for determining the number of latent classes to model.57 We will, therefore, use the BLRT, as well as the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), and the sample size-adjusted BIC (ABIC), with decreasing values of AIC, BIC and ABIC, indicating improved model performance; we will emphasise ABIC findings.58 We will also use the entropy measure as a summary of latent class classification success (the degree to which obtained latent classes can be distinguished). Its values range from 0 to 1, with values closer to 1 indicating greater classification clarity.59 We will use Mplus (V.8.6) to conduct GMM analyses.60
Aim 3: identify facilitators and barriers to systemwide cPRO implementation
Aim 3—implementation battery measure descriptive statistics and reliability analyses. As with our PRO measures (aims 2 a/2b), for all implementation battery measures, we will compute descriptive statistics at the item and scale level to identify poorly functioning items (see above for details). We will also estimate internal consistency reliability (Cronbach’s alpha); alpha ≥0.70 will serve as a minimum standard for making group-level comparisons. For measures with acceptable internal consistency reliability, we will graphically depict baseline, 3-month and 7-month status for (a) organisational culture, (b) leadership support, (c) use of research in practice, (d) acceptability, appropriateness and feasibility and (e) Collaborative Behavioral Health (CBH) training experience. Although our aim 3 sample size will not support conducting inferential statistical comparisons, graphical depictions of longitudinal status will allow us to visualise overall trends as well as trends within region and clinic. For each graph constructed, we will plot mean and median values per assessment time point as well as individual values per survey participant. We will summarise and share (in aggregate form) group-based descriptive statistics of the healthcare stakeholder survey measures with respondents. Using this sequential mixed methods (quantitative and qualitative) approach for implementation research,61 62 we will next use the implementation survey results themselves (combined with EHR data on patient engagement) to develop focus group guides to obtain additional information on barriers and facilitators of specific aspects of implementation.
Aim 3—focus group data analysis. We will code focus group data using a directed content analysis approach63 that involves use of existing implementation research theory, prior results and empirical frameworks to develop a manualised coding scheme and hypothesise the relationship between codes. Coders will meet to discuss initial thoughts, insights and observations for the development of preliminary coding categories. Through a systematic process, analysts will next begin investigating coded focus group data and refine emergent themes (eg, collapsing redundant themes, removing irrelevant themes); they will develop a coding dictionary for the remaining analyses. They will summarise coded data and identify the most important themes in terms of prevalence and/or impact. Twenty per cent of the focus group data will be double coded in order to calculate coder agreement reliability.64 Disagreements in coding will be resolved via expert consensus.
Patient and public involvement
No patients or public were directly involved in the design or implementation of this study protocol. However, this work builds on prior initiatives informed by the priorities, experiences and feedback of patients and the medical community at large regarding many aspects of the symptom monitoring process, including value and functionality. During the study, we plan to collect feedback formally via qualitative interviews and surveying and informally via ongoing feedback from patients and clinicians actively using cPRO to guide the implementation process and facilitate system adoption. Finally, at the point of dissemination, we plan to include relevant study findings in ongoing communication with current and future patients (eg, via brochures and website information and in personal communication from clinicians) to help define value.
Ethics and dissemination
Ethical and safety considerations
This study has undergone rigorous scientific evaluation via the Agency for Healthcare Research and Quality peer-review process, and the protocol has been approved by the Social and Behavioral Research Panel of Northwestern University’s Institutional Review Board (IRB; study number STU00207807). All study sites fall under a single IRB reliance agreement. All component human subjects’ research has been deemed ‘low risk’, and participation requires signed informed consent of an IRB-approved consent form. The study will not advertently recruit special patient populations, including adults with cognitive impairments, pregnant women, prisoners or other detained individuals.
Dissemination
This protocol details the rationale, design, materials and methods of an innovative quality improvement initiative that permits implementation and expansion of a patient-reported e-symptom screening system in a multisite cancer centre and a formal mixed-methods evaluation of the process and initial results. The outcomes of this study will not only help define the success of enhanced symptom monitoring in improving the quality of patient care and experiences in our study population but may also contribute knowledge to the field of implementation science to accelerate innovation and research. Data will be curated and findings from this research will be disseminated via professional scientific meetings and in peer-reviewed journals to the extent possible. Anonymised data from this study will be shared in a publicly accessible way with the research community at large to advance science and health, including patient-centred care.
We would like to thank September Cahue, MPH and Victoria Morken, MPH for their contributions to protocol development and Karen Kaiser, PhD for her guidance on qualitative methods.
Ethics statements
Patient consent for publication
Not applicable.
Contributors SFG: study conceptualisation; study design; qualitative analysis plan; quantitative analysis plan; protocol writing; critical review of drafts. JDS: study design; qualitative analysis plan; quantitative analysis plan; protocol writing; critical review of drafts. MK: quantitative analysis plan; protocol writing; critical review of drafts. KAW: qualitative analysis plan; protocol writing; critical review of drafts. ML: qualitative analysis plan; protocol writing; critical review of drafts. SK: study conceptualisation; critical review of drafts. MB: study design; critical review of drafts. DC: study conceptualisation; critical review of drafts. FJP: study conceptualisation; study design; quantitative analysis plan; critical review of drafts.
Funding This work is supported by the Agency for Healthcare Research and Quality (AHRQ) grant number R18HS026170-01.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; peer reviewed for ethical and funding approval prior to submission.
1 Miller KD, Nogueira L, Mariotto AB, et al. Cancer treatment and survivorship statistics, 2019. CA Cancer J Clin 2019; 69: 363–85. doi:10.3322/caac.21565 http://www.ncbi.nlm.nih.gov/pubmed/31184787
2 Caruso R, Nanni MG, Riba MB, et al. The burden of psychosocial morbidity related to cancer: patient and family issues. Int Rev Psychiatry 2017; 29: 389–402. doi:10.1080/09540261.2017.1288090 http://www.ncbi.nlm.nih.gov/pubmed/28753076
3 Penedo FJ, Cella D. Responding to the quality imperative to embed mental health care into ambulatory oncology. Cancer 2017; 123: 382–6. doi:10.1002/cncr.30402 http://www.ncbi.nlm.nih.gov/pubmed/27859012
4 Mehnert A, Brähler E, Faller H, et al. Four-week prevalence of mental disorders in patients with cancer across major tumor entities. J Clin Oncol 2014; 32: 3540–6. doi:10.1200/JCO.2014.56.0086 http://www.ncbi.nlm.nih.gov/pubmed/25287821
5 Kanani R, Davies EA, Hanchett N, et al. The association of mood disorders with breast cancer survival: an investigation of linked cancer registration and hospital admission data for South East England. Psychooncology 2016; 25: 19–27. doi:10.1002/pon.4037 http://www.ncbi.nlm.nih.gov/pubmed/26619290
6 Mitchell AJ, Chan M, Bhatti H, et al. Prevalence of depression, anxiety, and adjustment disorder in oncological, haematological, and palliative-care settings: a meta-analysis of 94 Interview-Based studies. Lancet Oncol 2011; 12: 160–74. doi:10.1016/S1470-2045(11)70002-X http://www.ncbi.nlm.nih.gov/pubmed/21251875
7 Cella D, Peterman A, Passik S, et al. Progress toward guidelines for the management of fatigue. Oncology 1998; 12: 369–77. http://www.ncbi.nlm.nih.gov/pubmed/10028520
8 Deshields TL, Potter P, Olsen S, et al. The persistence of symptom burden: symptom experience and quality of life of cancer patients across one year. Support Care Cancer 2014; 22: 1089–96. doi:10.1007/s00520-013-2049-3 http://www.ncbi.nlm.nih.gov/pubmed/24292095
9 Pearman T, Garcia S, Penedo F, et al. Implementation of distress screening in an oncology setting. J Community Support Oncol 2015; 13: 423–8. doi:10.12788/jcso.0198 http://www.ncbi.nlm.nih.gov/pubmed/26863872
10 Burg MA, Adorno G, Lopez EDS, et al. Current unmet needs of cancer survivors: analysis of open-ended responses to the American cancer Society study of cancer survivors II. Cancer 2015; 121: 623–30. doi:10.1002/cncr.28951 http://www.ncbi.nlm.nih.gov/pubmed/25581252
11 Wang T, Molassiotis A, Chung BPM, et al. Unmet care needs of advanced cancer patients and their informal caregivers: a systematic review. BMC Palliat Care 2018; 17: 1–29. doi:10.1186/s12904-018-0346-9 http://www.ncbi.nlm.nih.gov/pubmed/30037346
12 Cancer ACoSCo. Cancer program standards 2016: ensuring patient-centered care, 2016. Available: https://www.facs.org/quality-programs/cancer/coc/standards/2016 [Accessed 20 Sep 2017 ].
13 Page AE, Adler NE. Cancer care for the whole patient: meeting psychosocial health needs, 2008.
14 Riba MB, Donovan KA, Andersen B, et al. Distress management, version 3.2019, NCCN clinical practice guidelines in oncology. J Natl Compr Canc Netw 2019; 17: 1229–49. doi:10.6004/jnccn.2019.0048 http://www.ncbi.nlm.nih.gov/pubmed/31590149
15 Cancer ACoSCo. Cancer program standards: ensuring patient-centered care. 2016, 2016.
16 Swarm RA, Paice JA, Anghelescu DL, et al. Adult cancer pain, version 3.2019, NCCN clinical practice guidelines in oncology. J Natl Compr Canc Netw 2019; 17: 977–1007. doi:10.6004/jnccn.2019.0038 http://www.ncbi.nlm.nih.gov/pubmed/31390582
17 NCCN guidelines. Available: https://www.nccn.org/guidelines/guidelines-detail?category=3&id=1424
18 Holland JC, Andersen B, Breitbart WS, et al. Distress management. J Natl Compr Canc Netw 2013; 11: 190–209. doi:10.6004/jnccn.2013.0027 http://www.ncbi.nlm.nih.gov/pubmed/23411386
19 Jacobsen PB, Donovan KA, Trask PC, et al. Screening for psychologic distress in ambulatory cancer patients. Cancer 2005; 103: 1494–502. doi:10.1002/cncr.20940 http://www.ncbi.nlm.nih.gov/pubmed/15726544
20 Jensen RE, Snyder CF, Abernethy AP, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract 2014; 10: e215–22. doi:10.1200/JOP.2013.001067 http://www.ncbi.nlm.nih.gov/pubmed/24301843
21 Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol 2014; 32: 1480–501. doi:10.1200/JCO.2013.53.5948 http://www.ncbi.nlm.nih.gov/pubmed/24711559
22 Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res 2008; 17: 179–93. doi:10.1007/s11136-007-9295-0 http://www.ncbi.nlm.nih.gov/pubmed/18175207
23 Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res 2013; 13: 1–24. doi:10.1186/1472-6963-13-211
24 Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004; 22: 714–24. doi:10.1200/JCO.2004.06.078 http://www.ncbi.nlm.nih.gov/pubmed/14966096
25 Davis K, Yount S, Del Ciello K, et al. An innovative symptom monitoring tool for people with advanced lung cancer: a pilot demonstration. J Support Oncol 2007; 5: 381–7. http://www.ncbi.nlm.nih.gov/pubmed/17944147
26 Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017; 318: 197–8. doi:10.1001/jama.2017.7156
27 Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017; 318: 197-198. doi:10.1001/jama.2017.7156 http://www.ncbi.nlm.nih.gov/pubmed/28586821
28 Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016; 34: 557–65. doi:10.1200/JCO.2015.63.0830 http://www.ncbi.nlm.nih.gov/pubmed/26644527
29 Penedo FJ, Oswald LB, Kronenfeld JP, et al. The increasing value of eHealth in the delivery of patient-centred cancer care. Lancet Oncol 2020; 21: e240–51. doi:10.1016/S1470-2045(20)30021-8 http://www.ncbi.nlm.nih.gov/pubmed/32359500
30 Cella D, Riley W, Stone A, et al. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol 2010; 63: 1179–94. doi:10.1016/j.jclinepi.2010.04.011 http://www.ncbi.nlm.nih.gov/pubmed/20685078
31 Jensen RE, Moinpour CM, Potosky AL, et al. Responsiveness of 8 patient-reported outcomes measurement information system (PROMIS) measures in a large, community-based cancer study cohort. Cancer 2017; 123: 327–35. doi:10.1002/cncr.30354 http://www.ncbi.nlm.nih.gov/pubmed/27696377
32 Bass M, Morris S, Neapolitan R. Utilizing multidimensional computer adaptive testing to mitigate burden with patient reported outcomes. Paper presented at: AMIA Annual Symposium Proceedings, 2015.
33 Garcia SF, Wortman K, Cella D, et al. Implementing electronic health record-integrated screening of patient-reported symptoms and supportive care needs in a comprehensive cancer center. Cancer 2019; 125: 4059–68. doi:10.1002/cncr.32172 http://www.ncbi.nlm.nih.gov/pubmed/31373682
34 Wagner LI, Schink J, Bass M, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer 2015; 121: 927–34. doi:10.1002/cncr.29104 http://www.ncbi.nlm.nih.gov/pubmed/25376427
35 Nolan K, Schall MW, Erb F, et al. Using a framework for spread: the case of patient access in the Veterans health administration. Jt Comm J Qual Patient Saf 2005; 31: 339–47. doi:10.1016/S1553-7250(05)31045-2 http://www.ncbi.nlm.nih.gov/pubmed/15999964
36 Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64. doi:10.3389/fpubh.2019.00064 http://www.ncbi.nlm.nih.gov/pubmed/30984733
37 Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health 2020; 8: 134. doi:10.3389/fpubh.2020.00134 http://www.ncbi.nlm.nih.gov/pubmed/32478025
38 The consolidated framework for implementation research (CFIR). Available: www.cfirguide.org
39 Healthmeasures.net. Available: www.healthmeasures.net
40 Presson AP, Zhang C, Abtahi AM, et al. Psychometric properties of the press Ganey® outpatient medical practice survey. Health Qual Life Outcomes 2017; 15: 1–7. doi:10.1186/s12955-017-0610-3
41 Medallia. Available: https://www.medallia.com/
42 Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–81. doi:10.1016/j.jbi.2008.08.010 http://www.ncbi.nlm.nih.gov/pubmed/18929686
43 NUCATS is funded in part by a clinical and translational science Award (CTSA) grant from the National Institutes of health (NIH). UL1TR001422 in.
44 Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implementation Science 2015; 10: 1–14. doi:10.1186/s13012-015-0209-1
45 Smith J, Norton W, DiMartino L. A longitudinal implementation strategies tracking system: development and initial acceptability. Paper presented at: 13 th Annual Conference on the Science of Dissemination and Implementation, 2020.
46 Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implementation Science 2013; 8: 1–11. doi:10.1186/1748-5908-8-139
47 Miller CJ, Barnett ML, Baumann AA, et al. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implementation Science 2021; 16: 1–12. doi:10.1186/s13012-021-01105-3
48 Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006; 6: 54. doi:10.1186/1471-2288-6-54 http://www.ncbi.nlm.nih.gov/pubmed/17092344
49 Krueger RA. Focus groups: a practical guide for applied research. Sage publications, 2014.
50 Brown CH, Wyman PA, Guo J, et al. Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide. Clin Trials 2006; 3: 259–71. doi:10.1191/1740774506cn152oa http://www.ncbi.nlm.nih.gov/pubmed/16895043
51 Clogg CC. Latent class models. In: Arminger G, Clogg CC, Sobel ME, eds. Handbook of statistical modeling for the social and behavioral sciences. New York: Springer, 1995: 311–59.
52 Muthén B, Shedden K. Finite mixture modeling with mixture outcomes using the em algorithm. Biometrics 1999; 55: 463–9. doi:10.1111/j.0006-341X.1999.00463.x http://www.ncbi.nlm.nih.gov/pubmed/11318201
53 Muthen B. Latent variable mixture modeling. In: Marcoulides GA, Schumacker RE, eds. New developments and techniques in structural equation modeling. New Jersey: Lawrence Erlbaum, 2001: 1–33.
54 Muthén B. Second-generation structural equation modeling with a combination of categorical and continuous latent variables: New opportunities for latent class–latent growth modeling. In: Collins LMS A, ed. New methods for the analysis of change. Washington, D.C: APA, 2001: 291–322.
55 Duncan TE, Duncan SC, Strycker LA, et al. Growth mixture modeling of adolescent alcohol use data. Methodology workshop paper, 2002. Available: http://www.ori.org/methodology
56 Browne MW, Cudeck R, Bollen KA, et al. Alternative ways of assessing model fit. In: Bollen KA, Long JS, eds. Newbury Park, CA: Testing Structural Equation Models Sage Publications, 1993: 136–62.
57 Nylund KL, Asparouhov T, Muthén BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: a Monte Carlo simulation study. Struct Equ Modeling 2007; 14: 535–69. doi:10.1080/10705510701575396
58 Yang C-C. Evaluating latent class analysis models in qualitative phenotype identification. Comput Stat Data Anal 2006; 50: 1090–104. doi:10.1016/j.csda.2004.11.004
59 Ramaswamy V, Desarbo WS, Reibstein DJ, et al. An empirical pooling approach for estimating marketing mix elasticities with PIMS data. Marketing Science 1993; 12: 103–24. doi:10.1287/mksc.12.1.103
60 Muthen & Muthen. Mplus [computer program]. Version 7.4. Los Angeles, CA, 2015.
61 Aarons GA, Fettes DL, Sommerfeld DH, et al. Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreat 2012; 17: 67–79. doi:10.1177/1077559511426908 http://www.ncbi.nlm.nih.gov/pubmed/22146861
62 Palinkas LA, Aarons GA, Horwitz S, et al. Mixed method designs in implementation research. Adm Policy Ment Health 2011; 38: 44–53. doi:10.1007/s10488-010-0314-z http://www.ncbi.nlm.nih.gov/pubmed/20967495
63 Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15: 1277–88. doi:10.1177/1049732305276687 http://www.ncbi.nlm.nih.gov/pubmed/16204405
64 Krippendorff K. Agreement and information in the reliability of coding. Commun Methods Meas 2011; 5: 93–112. doi:10.1080/19312458.2011.568376
65 Yanez B, Pearman T, Lis CG, et al. The FACT-G7: a rapid version of the functional assessment of cancer therapy-general (FACT-G) for monitoring symptoms and concerns in oncology practice and research. Ann Oncol 2013; 24: 1073–8. doi:10.1093/annonc/mds539 http://www.ncbi.nlm.nih.gov/pubmed/23136235
66 Quality AfHRa. CAHPS cancer care survey, 2020. Available: https://www.ahrq.gov/cahps/surveys-guidance/cancer/index.html
67 Evensen CT, Yost KJ, Keller S, et al. Development and testing of the CAHPS cancer care survey. J Oncol Pract 2019; 15: e969–78. doi:10.1200/JOP.19.00039 http://www.ncbi.nlm.nih.gov/pubmed/31425009
68 Basch E, Reeve BB, Mitchell SA, et al. Development of the National cancer Institute's patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Natl Cancer Inst 2014; 106: dju244. doi:10.1093/jnci/dju244 http://www.ncbi.nlm.nih.gov/pubmed/25265940
69 Hay JL, Atkinson TM, Reeve BB, et al. Cognitive interviewing of the US National cancer Institute's patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). Qual Life Res 2014; 23: 257–69. doi:10.1007/s11136-013-0470-1 http://www.ncbi.nlm.nih.gov/pubmed/23868457
70 Dueck AC, Mendoza TR, Mitchell SA, et al. Validity and reliability of the patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). Journal of Clinical Oncology 2012; 30: 9047. doi:10.1200/jco.2012.30.15_suppl.9047
71 de Souza JA, Yap BJ, Hlubocky FJ, et al. The development of a financial toxicity patient-reported outcome in cancer: the cost measure. Cancer 2014; 120: 3245–53. doi:10.1002/cncr.28814 http://www.ncbi.nlm.nih.gov/pubmed/24954526
72 Morris NS, MacLean CD, Chew LD, et al. The single item literacy screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006; 7: 1–7. doi:10.1186/1471-2296-7-21
73 Barr PJ, Thompson R, Walsh T, et al. The psychometric properties of collaborate: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res 2014; 16: e3085. doi:10.2196/jmir.3085 http://www.ncbi.nlm.nih.gov/pubmed/24389354
74 Armenakis AA, Bernerth JB, Pitts JP. Organizational change recipients' beliefs scale: development of an assessment instrument. J Appl Behav Sci 2007; 43: 481–505.
75 Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implementation Science 2014; 9: 1–10. doi:10.1186/1748-5908-9-45
76 Rapley T, Girling M, Mair FS, et al. Improving the normalization of complex interventions: part 1 - development of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol 2018; 18: 1–17. doi:10.1186/s12874-018-0590-y
77 Finch TL, Girling M, May CR, et al. Improving the normalization of complex interventions: part 2 - validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol 2018; 18: 1–13. doi:10.1186/s12874-018-0591-x
78 Smith JD, Fu E, Rado J, et al. Collaborative care for depression management in primary care: a randomized roll-out trial using a type 2 hybrid effectiveness-implementation design. Contemp Clin Trials Commun 2021; 23: 100823. doi:10.1016/j.conctc.2021.100823 http://www.ncbi.nlm.nih.gov/pubmed/34401595
79 et al Malone S, Mckay V, Prewitt K. Assessing clinical sustainability: a new, user-friendly tool for evaluating real-world practices. Paper presented at: APHA’s 2020 VIRTUAL Annual Meeting and Expo; Oct. 24-28, 2020.
80 Malone S, Prewitt K, Hackett R, et al. The clinical sustainability assessment tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun 2021; 2: 1–12. doi:10.1186/s43058-021-00181-2
© 2022 Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. http://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ . Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Kallen, Michael 1 ; Webster, Kimberly A 1
; Madison Lyleroehr 1 ; Kircher, Sheetal 3 ; Bass, Michael 1 ; Cella, David 1 ; Penedo, Frank J 4 1 Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
2 Department of Population Health Sciences, The University of Utah School of Medicine, Salt Lake City, Utah, USA; Division of Hematology and Oncology, Department of Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
3 Division of Hematology and Oncology, Department of Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
4 Sylvester Comprehensive Cancer Center, Miller School of Medicine, University of Miami Health System, Miami, Florida, USA