Testing implementation facilitation for uptake of an evidence-based psychosocial intervention in VA homeless programs: A hybrid type III trial

Abstract

Background

Healthcare systems face difficulty implementing evidence-based practices, particularly multicomponent interventions. Additional challenges occur in settings serving vulnerable populations such as homeless Veterans, given the population’s acuity, multiple service needs, and organizational barriers. Implementation Facilitation (IF) is a strategy to support the uptake of evidence-based practices. This study’s aim was to simultaneously examine IF on the uptake of Maintaining Independence and Sobriety Through Systems Integration, Outreach and Networking-Veterans Edition (MISSION-Vet), an evidence-based multicomponent treatment engagement intervention for homeless Veterans with co-occurring mental health and substance abuse, and clinical outcomes among Veterans receiving MISSION-Vet.

Methods

This multi-site hybrid III modified stepped-wedge trial involved seven programs at two Veterans Affairs Medical Centers comparing Implementation as Usual (IU; training and educational materials) to IF (IU + internal and external facilitation).

Results

A total of 110 facilitation events averaging 27 minutes were conducted, of which 85% were virtual. Staff (case managers and peer specialists; n = 108) were trained in MISSION-Vet and completed organizational readiness assessments (n = 77). Although both sites reported being willing to innovate and a desire to improve outcomes, implementation climate significantly differed. Following IU, no staff at either site conducted MISSION-Vet. Following IF, there was a significant MISSION-Vet implementation difference between sites (53% vs. 14%, p = .002). Among the 93 Veterans that received any MISSION-Vet services, they received an average of six sessions. Significant positive associations were found between number of MISSION-Vet sessions and outpatient treatment engagement measured by the number of outpatient visits attended.

Conclusions

While staff were interested in improving patient outcomes, MISSION-Vet was not implemented with IU. IF supported MISSION-Vet uptake and increased outpatient service utilization, but MISSION-Vet still proved difficult to implement particularly in the larger healthcare system. Future studies might tailor implementation strategies to organizational readiness.

Trial registration

ClinicalTrials.gov, NCT02942979.

About the Authors:

David A. Smelson

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Veterans Affairs Bridging the Care Continuum-Quality Enhancement Research Initiative, Bedford, Massachusetts, United States of America, Department of Psychiatry, University of Massachusetts Medical School, Worcester, Massachusetts, United States of America

Vera Yakovchenko

Roles Conceptualization, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliation: Veterans Affairs Bridging the Care Continuum-Quality Enhancement Research Initiative, Bedford, Massachusetts, United States of America

Thomas Byrne

Roles Conceptualization, Data curation, Formal analysis, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliations Veterans Affairs Bridging the Care Continuum-Quality Enhancement Research Initiative, Bedford, Massachusetts, United States of America, School of Social Work, Boston University, Boston, Massachusetts, United States of America

Megan B. McCullough

Roles Conceptualization, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing

Affiliations Veterans Affairs Bridging the Care Continuum-Quality Enhancement Research Initiative, Bedford, Massachusetts, United States of America, Department of Public Health, Zuckerberg College of Health Sciences, University of Massachusetts Lowell, Lowell, Massachusetts, United States of America

Jeffrey L. Smith

Roles Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – review & editing

Affiliation: Central Arkansas Veterans Healthcare System, Little Rock, Arkansas, United States of America

Kathryn E. Bruzios

Roles Investigation, Project administration, Resources, Writing – review & editing

Affiliations Veterans Affairs Bridging the Care Continuum-Quality Enhancement Research Initiative, Bedford, Massachusetts, United States of America, Department of Psychiatry, University of Massachusetts Medical School, Worcester, Massachusetts, United States of America

ORCID logo https://orcid.org/0000-0001-9674-348X

Sonya Gabrielian

Roles Conceptualization, Investigation, Project administration, Supervision, Writing – review & editing

Affiliations Veterans Affairs Greater Los Angeles Health Care System, Los Angeles, California, United States of America, David Geffen School of Medicine, University of California at Los Angeles, Los Angeles, California, United States of America

Introduction

Large healthcare systems, including the Department of Veterans Affairs (VA), are working towards eliminating care inefficiencies by integrating administrative, operations, and novel clinical interventions [1]. This is relevant to the VA’s strategic commitment to end Veteran homelessness, resulting in dramatic expansion in their programs [2]. This expansion included implementing evidence-based practices VA-wide to improve outcomes, which can often prove difficult given the need to broadly address individual (e.g., training, involvement in decision making) and organizational factors (organization size, climate, support for the practices among staff and administrators [3–5]. Evidence-based practices are often challenging to uniformly adopt across a system even when they are known to improve outcomes due to a variety of barriers including time-consuming training requirements or changing provider routines [3]. These barriers can be particularly challenging for programs serving vulnerable populations, such as Veterans experiencing homelessness, given that these programs often address multiple behavioral health, substance abuse, medical care and social client needs simultaneously, while also being mindful of such issues as care fragmentation and treatment engagement [6].

Maintaining Independence and Sobriety through Systems Integration, Outreach and Networking-Veterans Edition (MISSION-Vet) is an evidence-based multicomponent wraparound treatment engagement approach for homeless Veterans with a co-occurring mental health and substance use disorder [7]. While a detailed description of MISSION-Vet is included in the methods section below, in short, MISSION-Vet is a team-based hybrid psychosocial and linkage intervention with the primary objective of engaging homeless Veterans with co-occurring disorders in outpatient care. This intervention aims to address the challenges posed by low rates of treatment engagement for homeless Veterans with co-occurring disorders, as treatment is critical for housing sustainability and recovery [8, 9]. MISSION-Vet has improved treatment attendance and engagement, mental health and substance abuse outcomes, and reduced days homeless [5, 10–14]. MISSION-Vet was also certified by Substance Abuse and Mental Health Services Administration’s National Registry of Evidence Based Practices [5, 10–12]. While MISSION-Vet implementation within VA homeless programs fills a current care gap, as a team-based multicomponent intervention, it also presents implementation challenges [15, 16]. These challenges include MISSION-Vet requiring 1) case managers and peers to work together as a team despite having somewhat different training and philosophies, 2) staff to use as hybrid model, which includes both running psychoeducational groups and doing assertive community outreach, and 3) staff using a stepdown model with decreasing intensity so clients ultimately engage in community supports [17]. Thus, the model intensity and complexity make it more complicated to implement as compared to single discipline and single modality interventions.

A previous study with Getting to Outcomes (GTO), an implementation model focused on capacity building for uptake of MISSION-Vet in VA homeless programs nationally found that while MISSION-Vet was implemented at all the sites, the intervention intensity and complexity, organizational demands, and time required for GTO resulted in lower MISSION-Vet uptake [18]. There are other implementation approaches to help organizations adopt and sustain complex evidence-based approaches like MISSION-Vet [19]. Implementation facilitation (IF) is an evidence-based strategy conceptualized within the integrated ‘Promoting Action on Research Implementation in Health Services’ (i-PARIHS) framework, in which facilitators external and/or internal to a healthcare organization work to support successful implementation of clinical innovations by assisting stakeholders with planning, execution, and refinement that addresses factors related to: (a) characteristics of the innovation itself; (b) the outer and inner context of the healthcare setting; and (c) characteristics of the recipients of the innovation [20]. IF is multifaceted, involving interactive problem-solving and support that occurs in a context of a recognized need for improvement and supportive interpersonal relationships [19]. Given previous work showing IF to be an effective strategy for implementing complex clinical innovations, we posited that it may be effective for implementing a multicomponent intervention like MISSION-Vet, in a complex healthcare system like VA, and in programs serving a high acuity population of homeless Veterans with a co-occurring disorder [21]. Therefore, consistent with recommendations for Type III hybrid implementation-effectiveness designs [22], the primary aim was to study the impact of the IF strategy to support MISSION-Vet implementation and fidelity. The secondary implementation aim was to examine organizational readiness differences between sites. The effectiveness aim was to examine the association between receipt of MISSION-Vet and treatment engagement in clinical services, as measured by VA inpatient and outpatient service utilization.

Method

Study design

This was a multi-site randomized hybrid type III implementation-effectiveness modified stepped-wedge trial in seven homeless programs at two VA Medical Centers (VAMCs) serving homeless Veterans with co-occurring mental health and substance use problems [22]. Hybrid type III trials are intended for interventions like MISSION-Vet with robust effectiveness data; though effectiveness data may be further gathered, the core focus of these trials is implementation of the client level intervention (here, MISSION-Vet). We selected a modified stepped-wedge design to compare MISSION-Vet uptake under two staff level intervention conditions: Implementation as Usual (IU) versus Implementation Facilitation (IF) [22]. This is a modified stepped-wedge design in that, in contrast to a traditional stepped-wedge wherein all sites initially start in the control condition, with programs switching over to the intervention condition at fixed intervals or steps and then remaining in the intervention condition for the duration of the study, in our study, it was not feasible to implement IF at all programs simultaneously. As a result, in the present study, each program received six months of IU and crossed over to IF for another six months, with data collection continuing for an additional six months after the end of IF. Moreover, in terms of calendar time, the rollout of the control and intervention conditions in our study occurred in a sequential manner across programs, with each program being in the intervention and control conditions for a fixed period of time, and not contributing to the analysis in periods in which they are neither in the intervention or control condition (Fig 1). This type of design is akin to what the stepped-wedge design literature refers to as an incomplete stepped-wedge design with limited measurements prior to and after crossover [23, 24] and is appropriate to use when it is not feasible to implement an intervention or collect data in all clusters simultaneously, either due to resource constraints or for other reasons. In this case, we did not have the resources to do IF in every location at the same time. In addition, to further the existing literature supporting MISSION-Vet outcomes, this trial also enabled us to extract existing data from the VA Electronic Medical Record to examine treatment engagement among those receiving MISSION-Vet.

[Figure omitted. See PDF.]

Fig 1. Modified stepped-wedge trial design.

https://doi.org/10.1371/journal.pone.0265396.g001

The project was deemed quality improvement and received an exempt status by the Institutional Review Board at the Bedford, Massachusetts VAMC according to the VA Program Guide 1200.21 [25], thus waiving the need for written or verbal informed consent. The staff being trained were the study participants and they were informed about the project’s designation as Quality Improvement. Data from the medical record were extracted for the clients being served by the staff members offering MISSION-Vet. Staff were informed of this data extraction and records were not anonymized as it was necessary to identify the clients being served by the staff offering MISSION-Vet.

MISSION-Vet intervention

A detailed description of MISSION-Vet client level intervention is reported in a previous protocol paper [26]. In brief, MISSION-Vet is delivered by a master’s level social work case manager, and a peer specialist, the latter of which is someone with prior lived experience with homelessness, substance abuse and mental health issues. The MISSION-Vet team delivers the following five treatment components: critical time intervention, dual recovery therapy (DRT), peer support, vocational and educational support, and trauma-informed care, all guided by Housing First and harm reduction philosophies that emphasize low barrier services for clients [27–32]. Both the case manager and peer specialist offer psychoeducational sessions either individually or in groups (13 DRT co-occurring disorders groups and 11 peer support recovery groups both designed to empower Veterans to engage in treatment) along with unstructured community outreach sessions to engage clients in care and link Veterans to other needed community support. MISSION-Vet was offered for approximately 2-hours a week for 6-months, and service delivery was guided by a Treatment Manual [7]. Veterans could also receive a MISSION-Vet Workbook that includes assignments reinforcing recovery [33].

Participants and recruitment

The project was conducted at two VAMCs (hereafter Sites A and B) and offered to staff at seven homeless programs (four locations at Site A and three locations at Site B) with the unit of measurement being the two VAMC’s. The two VAMCs were selected because of the size and scope of the healthcare systems, geographic dispersion, and the rate of homelessness in the regions [34]. Site A was in a large VA urban setting, serving approximately 83,000 unique Veterans annually with the highest-level complexity, and two on-site residential buildings, and two off-site buildings. Site B was a smaller suburban medium complexity VA serving approximately 18,000 unique Veterans annually, with one on-site and off-site residential building. Both Sites A and B had community-based non-residential treatment which included housing placement, case management, linkages to mental health and substance use programing, but not MISSION-Vet.

There were two groups of participants in the study: staff and clients. The first group are the case managers and peer specialists delivering MISSION-Vet; staff participation was voluntary with no incentives provided. The second group were, Veterans (clients) being served by these staff in their respective VA homeless programs. Staff were encouraged to follow the recommended MISSION-Vet inclusion and exclusion criteria. This included: (1) enrolled in a VA homeless program at one of the implementation sites; (2) met Diagnostic and Statistical Manual of Mental Disorders, 5th Edition [35] diagnostic criteria or International Classification of Diseases, 10th Revision [36] for current substance use disorder (e.g., alcohol, marijuana, cocaine) and a co-occurring mental illness which includes anxiety, mood, or a psychotic spectrum disorder.

Implementation strategy

Implementation as usual (IU).

At the outset, leadership from both VAMCs were introduced to MISSION-Vet and invited to participate. IU was comprised of a 1.5-hour webinar training offered at least twice to case manager and peer specialist staff within the two VAMCs and seven programs to accommodate scheduling. Training provided an overview of the MISSION-Vet approach and the implementation materials (MISSION-Vet Treatment Manual, MISSION-Vet Consumer Workbook, MISSION-Vet Fidelity Measure), and staff roles [7, 33]. It also presented how to use the MISSION-Vet service delivery fidelity measure, embedded within the VA medical record. We used this fidelity measure to capture the total number and type of MISSION-Vet sessions delivered, which served as our measures of MISSION-Vet uptake. This passive implementation strategy has been used in previous studies [37].

Implementation facilitation (IF).

Following initial training, there was a 6-month waiting period prior to the 6-months of IF being offered to each of the seven programs at the two VAMCs. As noted in Fig 1, IF was turned on in a stepwise fashion at each of the seven sites over a 21-month period, with the sites randomly assigned to a particular step. External facilitation is the form of IF used in this study and delivered by outside IF experts with specialized knowledge of implementation and quality improvement approaches [38–40]. In MISSION-Vet, IF experts partner with facility staff to implement MISSION-Vet through implementation planning, goal-setting and problem-solving [41–43]. Another IF goal was to work with VAMCs to tailor the evidence based practice where appropriate to meet local contextual demands. The external facilitators (JLS, VY) held bi-weekly meetings with local program staff executing MISSION-Vet to address implementation barriers, troubleshoot, and provide implementation fidelity reports, which included feedback on number and type of MISSION-Vet services delivered. External facilitators also provided regular feedback on the staff’s use of the fidelity measure within the VA medical record since this fidelity measure was used to construct our measure of implementation uptake.

Measures

Project measures captured information about organizational readiness, implementation outcomes (both IF and the implementation of MISSION-Vet), and VA health services utilization. Depending on the outcome, data were measured at the site, staff, and/or veteran level. With regard organizational readiness, we used an abbreviated version of the Organizational Readiness to Change Assessment (ORCA) context subscale and Jacobs’ Implementation Climate survey, resulting in a 21-item 5-point Likert scale to get at site and staff level readiness [44, 45]. Higher scores indicate greater organizational readiness and implementation climate. Following MISSION-Vet training, staff were asked to complete the organizational readiness survey and demographic survey regarding their age, sex, role/position, and tenure in VA.

Consistent with recommendations for type III hybrid effectiveness-implementation designs, our primary outcome was MISSION-Vet uptake (as measured by number of MISSION-Vet sessions delivered) during the IU versus IF time periods [22] and the secondary aim was to assess clinical outcomes (health service utilization). For this comparison of IU versus IF timeframes, we used a standardized facilitation tracking sheet completed by the external facilitators at the site level that included date, length of time, parties involved, activity type [46]. In addition, MISSION-Vet implementation was collected with a fidelity measure that was embedded in the Veterans’ electronic medical record using a specially created service tracking note template to quantify the type and amount of MISSION-Vet delivered [18]. Information captured in this note template included: which DRT sessions, peer support sessions, and Consumer Workbook exercises were completed; whether the MISSION-Vet Consumer workbook was provided; whether community activities were done with a Veteran (e.g., taken to appointment, NA/AA meetings, meetings with landlords); and referrals made to other services. Finally, treatment engagement as an outcome was captured with Veterans’ medical records obtained from the VA Corporate Data Warehouse, which included number of MISSION-Vet contacts and other outpatient visits (mental health, substance use, medicine, primary care, emergency department, other, total). Each service utilization outcome was aggregated over the 1-year period following the date of Veterans’ initial MISSION-Vet session.

Data analysis

Our analytic strategy involved four components that align with the four study aims. Specifically, we examined: 1) pre-implementation organizational readiness; 2) IF process, including IF events; 3) MISSION-Vet implementation in the IU and IF time periods; and 4) association between MISSION-Vet and VA health services. Because the number of trained providers to deliver MISSION-Vet and the number of Veterans who received it at the seven homeless programs was too small for meaningful program comparison, our analysis focuses on a comparison between the two VAMCs (Sites A and B) rather than the seven individual programs when making comparisons for all measures of interest. In other words, practical considerations necessitated that we modify our intended analysis plan to make the site, rather than the program the cluster unit of interest.

First, we examined organizational readiness using descriptive statistics and conducted comparisons of organizational readiness between the Sites A and B and by staff type (case manager vs. peer specialist), staff age, staff sex, and duration of employment with the VA using non-parametric Wilcoxon and Kruskal-Wallis tests. Second, we used descriptive statistics to examine IF events, including number, duration, and type of IF activities. Third, and similarly, we used descriptive statistics to examine implementation of MISSION-Vet. We summarize information about the number and type of MISSION-Vet sessions provided overall, at the Veteran-level, and by VAMC. We also examined provision of MISSION-Vet separately by staff type (i.e., whether a case manager or peer specialist). Additionally, to assess the potential impact of IF on MISSION-Vet, we examined how the overall provision of MISSION-Vet changed over time both before and after the start of IF using descriptive measures of the number of MISSION-Vet sessions provided at each site by month. Our intent was to estimate the intervention effect using a statistical model in line with established practices for stepped-wedge designs. However, this analysis plan was not feasible for two reasons. First, as noted above, the number of trained providers deliver MISSION-Vet and the number of Veterans receiving MISSION-Vet at each of the seven original program sites was small, which caused us to shift from using the program to the site as our cluster of interest in our analysis. Second, because neither of the two sites provided any MISSION-Vet services in the IU period, there was no variation in the outcome of interest during this time thus rendering it impractical to estimate such a model. We therefore rely solely on descriptive statistics to examine the impact of IF on the provision of MISSION-Vet services.

Fourth, as an exploratory analysis, we examined the relationship between receipt of MISSION-Vet and Veteran-level measures of engagement in clinical service (i.e., VA inpatient and outpatient services). To do so, we estimated a series of bivariate linear regression models in which our service utilization measures (i.e., number of outpatient and inpatient visits, by type, in the year after a Veteran’s initial MISSION-Vet session) served as the outcomes of interest and the number of MISSION-Vet sessions in the year following a Veteran’s initial MISSION-Vet session served as the predictor of interest in all models. As these models were purely exploratory, they did not adjust for any additional covariates.

Results

Staff characteristics

This study commenced in February 2016 and recruitment stopped in July 2019. 108 staff were trained in MISSION-Vet as part of Implementation as Usual across two VAMCs (93 at Site A across 11 trainings,15 at Site B across four trainings). Following training, 77 staff (69% Site A, 87% Site B) completed an organizational readiness and demographic survey. Most respondents were case managers (77%) or peer specialists (16%) and few were unknown (8%). There was an even mix of males (47%) and females (45%), and unknown (8%); the average age was 49 ± 12 (range 26–72), length of time in VA was 5 ± 5 years (range <1 year-25 years). It is noteworthy that while MISSION-Vet was intended to be implemented by a case manager-peer specialist dyad, peer specialists were less available at Site A, which is also why fewer peer specialists were trained throughout the project.

Organizational readiness

As described in Table 1, staff reported moderate to high organizational culture and climate at both sites. Site B had higher scores than Site A on nearly all individual items, although only several differences were statistically significant, perhaps due to the small sample size. The most consistent differences between sites were on the ORCA context staff culture subscale, with 100% of Site B staff agreeing to cooperate to maintain and improve patient care effectiveness and being willing innovate to improve clinical procedures, compared to 85% (p = .055) and 77% (p = .03) at Site A, respectively. Sites also differed on implementation climate scores regarding support to use MISSION-Vet being higher at Site B than Site A (92% vs 76%, p = .005), despite both sites reporting low recognition and appreciation for using MISSION-Vet. There were no statistically significant differences in organizational measures by role, sex, age, or site. There was, however, a positive significant relationship (r = .32, p = .01) between duration of VA employment and the ORCA measurement scale regarding goals, guidelines, feedback, and accountability.

[Figure omitted. See PDF.]

Table 1. Organizational readiness scores by site, N = 77.

https://doi.org/10.1371/journal.pone.0265396.t001

Implementation facilitation

Following Implementation as Usual, one external facilitator per site supported MISSION-Vet implementation. While we recognize that on the surface, it might suggest an imbalance in workload given that Site A was much larger than Site B, the facilitator in Site A had more available time to devote to this project. Facilitation activities included stakeholder engagement, site assessment, preparation/planning, ongoing process monitoring, education, program adaptation, marketing, and problem identification and problem solving. A total of 110 facilitation events averaging 27-minutes were conducted. At Site A, there was a total of 70 virtual (phone or Skype) external facilitation events across the four participating programs at Site A that averaged 24 minutes over a 17-month period. There was a total of 40 facilitation events across the three participating programs at Site B averaging 34-minutes over an 11-month period, including 60% virtual and 40% in-person. Site B had a higher per person facilitation dose (i.e., how many staff participated in MISSION-Vet vs. how many facilitation events were provided) than Site A.

Implementation as usual versus implementation facilitation

To explore the primary study objective, which is the potential impact of IF on provision of MISSION-Vet services, we examined how the overall provision of MISSION-Vet across both sites changed over time following the start date of IF. As noted above, although the calendar date on which IF began at each site varied, we standardized our analysis of MISSION-Vet over time to examine the provision of MISSION-Vet services over a standard 21-month period indexed to the start date of IF by site.

Fig 2 shows the results of this analysis and aggregates the total number of MISSION-Vet sessions provided across both Sites in the 6-months prior to and 12-months following the start of IF. There were no MISSION-Vet services provided at either site prior to the start of IF. However, MISSION-Vet services commenced immediately after the start of IF, suggesting that IF was effective in increasing the implementation of MISSION-Vet.

[Figure omitted. See PDF.]

Fig 2. Total number of MISSION-Vet sessions delivered across 2 VAMCs by month relative to the start of facilitation.

https://doi.org/10.1371/journal.pone.0265396.g002

Table 2 summarizes results of our analysis of the provision of MISSION-Vet, including the between site comparisons of the provision of MISSION-Vet during the IF period. As the table shows, when examining the actual implementation of MISSION-Vet in the IF period, of the 108 staff trained, 53% at Site B tried MISSION-Vet, as evidenced by staff entering at least one MISSION-Vet note, as compared to 14% of staff at Site A completed a MISSION-Vet note. This difference was statistically significant (p = .002). A total of 574 MISSION-Vet notes were entered during the IF period. This included 424 DRT notes (273 at Site A vs 151 at Site B) and 89 peer notes (32 at Site A vs 57 at Site B) by 14 case managers (9 at Site A vs 5 at Site B) and seven peer specialists (four at Site A vs three at Site B), with 93 (70 at Site A vs 23 at Site B) Veterans. Relatively fewer peer support sessions were conducted at Site A (12% of all sessions), as compared to Site B (38% of all sessions), as evidenced by the notes. Nearly all the community events were conducted at Site B (39 vs 2). No service referrals were conducted at Site A, compared to 114 at Site B.

[Figure omitted. See PDF.]

Table 2. MISSION-Vet implementation outcomes.

https://doi.org/10.1371/journal.pone.0265396.t002

With regards to the Veterans served, 93 received MISSION-Vet services by the trained staff during the IF period. Again, Site B provided more per-person DRT sessions, on average, (4.5 sessions, vs 7.9 sessions, p = .017) and similar per-person peer sessions, on average, (6.4 vs 4.1, p = .426) than Site A. No Veterans received the complete DRT dose or peer dose: 62 (67%) received only DRT sessions, two (2%) received only peer sessions, 17 (18%) received both DRT and peer sessions, and 12 (13%) received unstructured MISSION-Vet. MISSION-Vet was implemented with great overall fidelity with regards to both DRT and peer support at Site B as compared to Site A (61% vs 4%, p < .001). Furthermore, across Sites A and B, Veterans overall tended to remain engaged in MISSION-Vet, averaging 4.5-months of MISSION-Vet services of the anticipated 6-months of services.

Health service utilization

The regression models we estimated to assess the relationship between the number of MISSION-Vet sessions and our health services utilization measures identified significant positive associations between the total number of MISSION-Vet sessions and the total number of outpatient medical-specialty visits, with each additional MISSION-Vet session associated with, on average, an additional 0.28 visits (B = .28, t(91) = 2.63, p = .001) in the one-year period after the start of MISSION-Vet services. Likewise, each additional MISSION-Vet session was associated with an increase of roughly 1 outpatient substance abuse treatment visit (B = .92, t(91) = 2.61, p = .01) and 2 total outpatient visits (B = 2.18, t(91) = 2.25, p = .03) in the one-year period following a Veteran’s initial MISSION-Vet session. The number of MISSION-Vet sessions was not significantly associated with any of our inpatient service utilization measures, or the total number of inpatient hospitalizations.

Discussion

In this hybrid III trial, we used a modified stepped-wedge design to compare IU versus IF with respect to the provision of MISSION-Vet. Our key findings were that neither site implemented MISSION-Vet with usual implementation and that IF significantly increased uptake of MISSION-Vet. Further, after IF, significantly more staff at Site B implemented MISSION-Vet compared to Site A. We also examined associations between pre-implementation organizational readiness and implementation. While we knew that Site B was a smaller and less complex site than Site A, we did not know the extent to which complexity and organizational readiness would drive outcomes. We found that the smaller site (Site B) had more flexibility to adopt MISSION-Vet than Site A. This suggests that it is not only the size of a site but also the organizational characteristics affected uptake. Higher MISSION-Vet uptake was associated with higher doses of IF overall, and this may be impacted by the in-person or virtual delivery. In exploratory analysis, this study also identified a positive relationship between the number of MISSION-Vet sessions and outpatient service utilization. These findings may suggest that MISSION-Vet assisted with outpatient treatment engagement, although some caution is warranted in making any firm conclusions about the extent to which this was the case, as this analysis did not adjust for any potential individual level or site-level factors that might confound the relationship between volume of receipt of MISSION-Vet and service utilization.

This project was done during a period of heightened attention to Veteran homelessness and following a commitment from the President of the United States and VA Secretary to end Veteran Homelessness [47]. It is not surprising that regarding organizational readiness, staff felt a personal responsibility for enhancing patient care and were interested in innovative practices to improve outcomes. However, these attributes alone were insufficient to stimulate uptake of MISSION-Vet. As mentioned above, several organizational characteristics were related to MISSION-Vet outcomes. We found that although Sites A and B had similar perceptions of leadership culture, staff culture, and implementation climate, Site B had a more supportive climate and staff were more willing to improve current practices. Powell et al. had similar findings that strategic (implementation climate) compared to general (organizational culture and climate, transformational leadership) organizational factors had more influence on knowledge and attitudes towards current mental health evidence-based practices [48]. Closer examination of how practitioners’ perspectives of context and climate shift between pre-implementation (i.e., knowledge and attitudes) to implementation may offer important clues into the tailoring of facilitation and selection of other implementation strategies [40].

Consistent with other recent work on implementation of behavioral health care interventions, we determined IF could be an appropriate strategy to support staff behavior change and address baseline needs by sites, and specifically, interpersonal support tailored to local needs, the mechanism underlying IF, which may be of benefit [38, 39, 49]. As noted above, sites implemented MISSION-Vet only after IF was initiated as training alone did no stimulate MISSION-Vet use. On-site facilitation (Site B) appeared to be more effective in generating use compared to virtual support (Site A). Although Site B had fewer facilitation events, Site B’s facilitation was delivered both in-person and virtually, and dose/personal attention was higher and more focused. These results also align with our prior MISSION-Vet study using GTO as the implementation strategy which also found that implementation support was needed to initiate MISSION-Vet use [18]. However, we do not attribute the differences only to virtual IF, as in reality, a constellation of contextual factors are responsible for the greater uptake and also include, medical center size and program scope and organizational readiness; these bear further study.

It is also interesting to note that no Veterans received a full dose of MISSION-Vet (24 sessions along with outreach and linkages as needed). Other studies of multicomponent behavioral health interventions have found implementation fidelity difficult to achieve compared to the implementation of singe component behavioral interventions and one provider [11, 12, 37, 50, 51]. Recent literature on facilitation indicates that it has a mixed legacy in implementation work; in some other studies facilitation is not as effective as the literature suggests in terms of format (active versus passive implementation strategies) and dose, however, for MISSION-Vet facilitation was a relatively productive approach [52, 53]. The contrast between Site A and B illustrates another aspect of this issue which is that larger medical centers may face additional barriers in implementing multimodal team-based behavioral interventions. However, among those Veterans offered MISSION-Vet, more MISSION-Vet services were associated with increased VA outpatient treatment engagement. Our prior studies have also found a relationship between MISSION-Vet, increased use of outpatient services and a reduction in homelessness [54]. Engagement in both MISSION-Vet and other outpatient treatments is critical for homeless Veterans with a co-occurring disorder as these Veterans often cycle in and out of healthcare services and are unstably engaged, which can result is housing instability and loss and exacerbation of medical problems [54].

Despite the promising findings, this study has several limitations. First, it was done in two VAMCs, of incomparable size, thus the data is not generalizable to the entire VA system. Importantly, practical considerations regarding the volume of MISSION-Vet provided across our original seven programs required that we adjust our clustering unit to the VAMC site. Second, given that no Veterans received a full dosed of MISSION-Vet, it was not possible to assess differences across the intended seven individual programs within VAMCs and thus we had to report comparison between the two VAMCs (Sites A and B). Third, this study used an IU comparison group and a slightly more active low intensity implementation intervention could have shown some effect. Nonetheless, this study did teach us that training alone (IU) was not effective in these VA settings and more proactive strategies (such as IF) may be needed to help guide future implementation. Fourth, IF was offered in person and virtually at Site B and virtually alone in site A, and this could have also been responsible for the site differences above and beyond the site size and organizational readiness differences, which were unable to be teased apart. Fifth, despite access to the VA medical record, no data was available regarding other clinical outcomes besides engagement as the VA data tracking client progress are not offered for every client at a standard time, thus making other clinical comparisons impossible. Sixth, it was not possible to control for secular changes over time between sites. Therefore, it is plausible that the increased national attention to the problem of homelessness during the implementation period or some other extraneous factor could account for the higher readiness and cooperation in Site B. However, while this is unlikely as no site did MISSION during IU, a qualitative component would have added additional nuance to the analyses.

Conclusion

Despite these limitations, this study offers the field lessons on how to get a team-based multimodal intervention like MISSION-Vet incorporated in both smaller and larger VA’s and in programs serving a population of homeless Veterans with a co-occurring disorder. This study suggests that standard MISSION-Vet web training is insufficient, and a more active and ongoing implementation support is needed. It is also possible that some tailoring of MISSION-Vet within certain settings might be warranted in future implementation studies involving MISSION-Vet given the time and intensity needed for service delivery. Moreover, MISSION-Vet was specifically developed to offer a comprehensive service delivery experience rather than Veterans receiving these services from separate programs and providers, the latter of which could fragment care. However, in the future, it might be feasible to cross train staff in programs already delivering MISSION-Vet type services such as assertive community outreach to deliver the psychoeducational components of MISSION-Vet or to train staff offering co-occurring disorders psychoeducational groups how to deliver community outreach and linkage support. Another important future direction might be to enhance the MISSION-Vet training, select a more intensive implementation strategy, as well as perhaps have a more robust external plus internal facilitation approach, the latter of which was previously found to be successful in implementing other complex clinical innovations in diverse healthcare settings [55]. Lastly, future studies might also examine virtual versus in-person facilitation in more rigorous ways.

Acknowledgments

The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government, and shall not be used for advertising or product endorsement purposes. We wish to thank the VA Central Office, Homeless Program Office for their support and guidance on this project and the seven homeless programs within the two VAMCs for the collaboration.

Citation: Smelson DA, Yakovchenko V, Byrne T, McCullough MB, Smith JL, Bruzios KE, et al. (2022) Testing implementation facilitation for uptake of an evidence-based psychosocial intervention in VA homeless programs: A hybrid type III trial. PLoS ONE 17(3): e0265396. https://doi.org/10.1371/journal.pone.0265396

References

1. Bentley TGK, Effros RM, Palar K, Keeler EB. Waste in the U.S. health care system: a conceptual framework. Milbank Q. 2008;86(4):629–59. pmid:19120983

2. U.S. Department of Veterans Affairs Homeless Veterans Program. Employment Toolkit. 2017, July.

3. Mechanic D, Tanner J. Vulnerable people, groups, and populations: societal view. Health Aff. 2007;26(5):1220–30.

4. Goetz MB, Hoang T, Knapp H, Burgess J, Fletcher MD, Gifford AL, et al. QUERI-HIV/Hepatitis Program. Central implementation strategies outperform local ones in improving HIV testing in Veterans Healthcare Administration facilities. J Gen Intern Med. 2013;28(10):1311–7. pmid:23605307

5. Smelson D, Losonczy M, Ziedonis D, Castles-Fonseca K, Kaune M. Six month outcomes from a booster case management program for individuals with a cooccurring substance abuse and a persistent psychiatric disorder. Eur J Psychiatry Clin Neurosci. 2007;21:143–52.

6. Szymkowiak D, Montgomery AE, Johnson EE, Manning T, O’Toole TP. Persistent super-utilization of acute care services among subgroups of veterans experiencing homelessness. Med Care. 2017;55(10):893–900. pmid:28863030

7. Smelson DA, Sawh L, Kane V, Kuhn J, Ziedonis DM. MISSION-VET Treatment Manual. Worcester: University of Massachusetts Medical School; 2011.

8. Durbin A, Kapustianyk G, Nisenbaum R, Wang R, Aratangy T, Khan B, et al. Recovery education for people experiencing housing instability: an evaluation protocol. Int J Soc Psychiatry. 2019;65(6):468–78. pmid:31250692

9. Ellison ML, Schutt RK, Yuan LH, Mitchell-Miland C, Glickman ME, McCarthy S, et al. Impact of peer specialist services on residential stability and behavioral health status among formerly homeless veterans with cooccurring mental health and substance use conditions. Med Care. 2020;58(4):307–13. pmid:31914105

10. Smelson D, Losonczy M, Castles-Fonseca K, Sussner B, Rodrigues S, Kaune M, et al. Preliminary outcomes from a community linkage intervention for individuals with co-occurring substance abuse and serious mental illness. J Dual Diagn. 2005;1(3):47–59.

11. Smelson D, Kalman D, Losonczy M, Kline A, St. Hill L, Castles-Fonseca K, et al. A brief treatment engagement intervention for individuals with co-occurring mental illness and substance use disorders: Results of a randomized clinical trial. Community Ment Health J. 2012;48(2):127–32. pmid:20859765

12. Smelson D, Kline A, Kuhn J, Rodrigues S, O’Connor K, Fisher W, et al. A wraparound treatment engagement intervention for homeless veterans with co-occurring disorders. Psychol Serv. 2013;10(2):161–7. pmid:23244030

13. Smelson DA, Pinals DA, Sawh L, Fulwiler C, Singer S, Guevremont N, et al. An alternative to incarceration: co-occurring disorders treatment intervention for justice-involved veterans. World Med Health Policy. 2015;7(4):329–48.

14. Smelson D, Farquhar I, Fisher W, Pressman K, Pinals DA, Samek B, et al. Integrating a co-occurring disorders intervention in drug courts: an open pilot trial. Community Ment Health J. 2018;55(2).

15. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350. pmid:25791983

16. Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5(1):1–9. pmid:20815872

17. Horton TJ, Illingworth JH, Warburton WH. Overcoming challenges in codifying and replicating complex health care interventions. Health Aff. 2018;37(2):191–7. pmid:29401020

18. Chinman M, McCarthy S, Hannah G, Byrne TH, Smelson DA. Using Getting To Outcomes to facilitate the use of an evidence-based practice in VA homeless programs: a cluster-randomized trial of an implementation support strategy. Implement Sci. 2017;12(1):34. pmid:28279207

19. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21. pmid:25889199

20. Harvey G, Kitson A. Implementing evidence-based practice in healthcare: Taylor & Francis; 2015.

21. Kirchner JE, Kearney LK, Ritchie MJ, Dollar KM, Swensen AB, Schohn M. Research & services partnerships: lessons learned through a national partnership between clinical leaders and researchers. Psychiatr Serv. 2014;65(5):577–9. pmid:24585229

22. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217. pmid:22310560

23. Unni RR, Lee SF, Thabane L, Connolly S, Van Spall HG. Variations in stepped-wedge cluster randomized trial design: Insights from the Patient-Centered Care Transitions in Heart Failure trial. American heart journal. 2020;220:116–26. pmid:31805422

24. Hemming K, Lilford R, Girling AJ. Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Statistics in medicine. 2015;34(2):181–96. pmid:25346484

25. Department of Veterans Affairs. Office of Research and Development Program Guide: 1200.21. VHA Operations Activities That May Constitute Research. Washington, DC: Department of Veterans Affairs; 2019.

26. Simmons MM, Gabrielian S, Byrne T, McCullough MB, Smith JL, Taylor TJ, et al. A hybrid III stepped wedge cluster randomized trial testing an implementation strategy to facilitate the use of an evidence-based practice in VA Homeless Primary Care Treatment Programs. Implement Sci. 2017;12(1):46. pmid:28376839

27. Susser E, Valencia E, Conover S, Felix A, Tsai WY, Wyatt RJ. Preventing recurrent homelessness among mentally ill men: a "critical time" intervention after discharge from a shelter. Am J Public Health. 1997;87(2):256–62. pmid:9103106

28. Ziedonis DM, Stern R. Dual recovery therapy for schizophrenia and substance abuse. Psychiatr Ann. 2001;31(4):255.

29. Chinman M, Shoai R, Cohen A. Using organizational change strategies to guide peer support technician implementation in the Veterans Administration. Psychiatr Rehabil J. 2010;33(4):269. pmid:20374985

30. Bond GR, McHugo GJ, Becker DR, Rapp CA, Whitley R. Fidelity of supported employment: lessons learned from the National Evidence-Based Practice Project. Psychiatr Rehabil J. 2008;31(4):300. pmid:18407879

31. Najavits LM. Expanding the boundaries of PTSD treatment. JAMA. 2012;308(7):714–6. pmid:22893170

32. Tsemberis S, Gulcur L, Nakae M. Housing first, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. Am J Public Health. 2004;94(4):651–6. pmid:15054020

33. Smelson DA, Sawh L, Rodrigues S, Munoz EC, Marzilli A, Tripp J, et al. The MISSION-VET Consumer Workbook. Worcester: University of Massachusetts Medical School; 2011.

34. Department of Veterans Affairs. VA 25 Cities Initiative 2014, March [https://www.va.gov/HOMELESS/25cities.asp.

35. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5). Arlington, VA: American Psychiatric Publishing; 2013.

36. World Health Organization. International statistical classification of diseases and health related problems (The) ICD-10: World Health Organization; 2004.

37. Smelson DA, Zaykowski H, Guevermont N, Siegfriedt J, Sawh L, Modzelewski D, et al. Integrating permanent supportive housing and co-occurring disorders treatment for individuals who are homeless. J Dual Diagn. 2016;12(2):193–201. pmid:27064939

38. Chinman M, Hunter SB, Ebener P, Paddock SM, Stillman L, Imm P, et al. The getting to outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008;41(3–4):206–24. pmid:18278551

39. Fritz J, Wallin L, Söderlund A, Almqvist L, Sandborgh M. Implementation of a behavioral medicine approach in physiotherapy: a process evaluation of facilitation methods. Implement Sci. 2019;14(1):94. pmid:31684977

40. Parchman M, Hsu C, Fagnan L, van Borkulo N, Tuzzio L. Building a learning health care organization: external facilitation tailors support to the learning capacity of primary care settings. J Patient Cent Res Rev. 2017;4(3):187.

41. Stetler CB, McQueen L, Demakis J, Mittman BS. An organizational framework and strategic implementation for system-level change to enhance research-based practice: QUERI Series. Implement Sci. 2008;3:30. pmid:18510750

42. Ajzen I, Fishbein M. Attitude-behavior relations: a theoretical analysis and review of empirical research. Psychol Bull. 1977;84(5):31.

43. Rosenheck RA. Organizational process: a missing link between research and practice. Psychiatr Serv. 2001;52(12):6. pmid:11726750

44. Helfrich CD, Li Y-F, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4(1):38. pmid:19594942

45. Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014;9(1):46. pmid:24742308

46. Ritchie MJ, Kirchner JE, Townsend JC, Pitcock JA, Dollar KM, Liu CF. Time and organizational cost for facilitating implementation of primary care mental health integration. J Gen Intern Med. 2019;35(4):1001–10. pmid:31792866

47. The White House. Ending Veteran Homelessness [https://obamawhitehouse.archives.gov/issues/Veterans/ending-homelessness.

48. Powell BJ, Mandell DS, Hadley TR, Rubin RM, Evans AC, Hurford MO, et al. Are general and strategic measures of organizational context and leadership associated with knowledge and attitudes toward evidence-based practices in public behavioral health settings? A cross-sectional observational study. Implement Sci. 2017;12(1):64. pmid:28499401

49. Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2015;11(1):33.

50. Brunette MF, Asher D, Whitley R, Lutz WJ, Wieder BL, Jones AM, et al. Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatr Serv. 2008;59(9):989–95. pmid:18757591

51. Drake RE, Bond GR. Implementing integrated mental health and substance abuse services. J Dual Diagn. 2010;6(3–4):251–62.

52. Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, et al. Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implement Sci. 2010;5:75. pmid:20942951

53. Garner B, Gotham H, Chaple M, Martino S, Ford J, Roosa M, et al. The implementation & sustainment facilitation strategy improved implementation effectiveness and intervention effectiveness: results from a cluster-randomized type 2 hybrid trial. 2020.

54. Smelson DA, Perez CK, Farquhar I, Byrne T, Colegrove A. Permanent supportive housing and specialized co-occurring disorders wraparound services for homeless individuals. J Dual Diagn. 2019:1–10.

55. Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29 Suppl 4:904–12. pmid:25355087