Telepsychology also called telemental health, or telebehavioral health is broadly defined as the delivery of behavioral health services via technological means, typically across a distance. Telepsychology can be performed in real time (synchronous) or as a delayed correspondence (asynchronous). Within large organizations such as the Department of Defense (DoD), Indian Health Service, and Department of Veterans Affairs (VA), video teleconferencing (VTC) is the most common means of telepsychology. Globally, telepsychology is used to deliver care to remote areas or postdisaster settings. For example, in 2010, telepsychology was leveraged to provide services to Haitians following a devastating earthquake (Augusterfer, Mollica, & Lavelle, 2015).
An important aspect of telepsychology is its utilization in the provision of effective treatments to those at facilities where access to care is limited due to provider shortages and the uneven distribution of low density specialized providers (e.g., pediatric psychiatrists). Research on telepsychology has demonstrated favorable findings related to both process variables (i.e., satisfaction and rapport; Gros, Lancaster, López, & Acierno, 2018; Morland et al., 2015) and outcome variables (Bashshur, Shannon, Bashshur, & Yellowlees, 2016). High treatment acceptability and credibility has also been found with an older adult population (Egede et al., 2016). Randomized trials (Nelson, Barnard, & Cain 2003; O’Reilly et al., 2007; Ruskin et al., 2004) as well as less rigorous studies (Frueh et al., 2007; Tuerk, Yoder, Ruggiero, Gros, & Acierno, 2010) suggest telepsychology is generally as effective as face-to-face (FTF) psychotherapy (Shore et al., 2018) and psychiatry (Hubley, Lynch, Schneck, Thomas, & Shore, 2016). Similarly, evidence-based psychotherapies delivered via telepsychology have generally, though not universally, been found to produce comparable clinical outcomes (Gros et al., 2013). For example, a number of well-designed randomized noninferiority clinical trials have recently been completed in the VA system demonstrating comparable outcomes in telepsychology delivered via VTC and FTF therapy; this includes cognitive processing therapy (CPT) for posttraumatic stress disorder (PTSD; Morland et al., 2014), prolonged exposure for PTSD (Acierno et al., 2017), and behavioral activation and therapeutic exposure for PTSD (Acierno et al., 2016).
However, telepsychology is not without limitations. While telepsychology is employed, in large part, to overcome conventional barriers to care, novel challenges inherent in technology-enhanced treatment present a new set of obstacles. These obstacles require unique ethical and legal considerations beyond those required to perform FTF clinical care; for example, the legal ability to provide services in both the location of the provider and the patient. For good reason, researchers and available resources have largely focused on the legal, regulatory, and ethical considerations pertinent to telepsychology (Glueckauf et al., 2018); for example, a telepsychology provider typically needs to be licensed to provide services both in the state where the provider is and the state where the client is at the time of services (in the military medical system providers need to have clinical privileges in both locations). These constraints are well addressed in other reviews and are beyond the scope of the current paper (Brooks, Turvey, & Augusterfer, 2013; Landry-Poole, Pujol, & Moore, 2017); however, information about the actual clinical practice of telepsychology is lacking.
Regarding the logistical and practical delivery of telepsychology services, a survey of telepsychology experts found that adaptations to treatment protocols were recommended for three domains: use of self-report questionnaires, treatment handouts, and in-session examples (Gros et al., 2013). Use of objective standardized outcome tracking measures and self-monitoring logs (e.g., sleep logs) is a part of evidence-based assessment, which is a critical component of evidence-based practice (Beidas et al., 2015; Brewin, 2013). Historically, cognitive-behavioral therapists have been more open to telepsychology than their psychodynamic counterparts (Mora, Nevid, & Chaplin, 2008; Perle et al., 2013); they are also more likely to use treatment manuals in their practice (Becker, Smith, & Jensen-Doss, 2013). Further, a sizable portion of these manualized treatments have worksheets that are core components of the treatment (Shafran et al., 2009). Clinicians attempting to provide these therapies via telepsychology may find themselves confused about how to practically go about providing those services, as distributing and completing worksheets through telepsychology is logistically different than FTF care. The term “in-session examples” refers to the use of visual aids such as a whiteboard or diagram to illustrate a point, facilitate attentional practice, and aid in the evaluation of a treatment target (Beck & Haigh, 2014; Waltman, 2015).
Suitable adaptations in the three recommended domains are demonstrated in the following case series in which telepsychology was utilized to deliver evidence-based treatments to an active duty military population. This case series will comprise three manualized therapies that include worksheets as a key component of treatment. A pharmacological treatment case will also be presented. The focus of these cases will be illustrating the adaptations to treatment protocols for the three domains that Gros and colleagues (2013) suggested (i.e., use of self-report questionnaires, treatment handouts, and in-session examples). Some details have been changed or omitted to maintain anonymity without impacting significant aspects of the case presentations. Informed consent was obtained from all individual participants included in the study.
The treatment context for these cases is that these were all active duty service members who were treated in military hospitals via telepsychology by clinicians who were located at a different military hospital. Treatment referrals were made from the location where the service members (SMs) were located and cases were managed through the military health system. This model of telepsychology is different from the VA home-based telepsychology model where the client uses telepsychology from their home. In the military model, telepsychology is used to increase access to care and decrease problems related to provider shortages (Landry-Poole et al., 2017). In these settings, the client sits in a telepsychology room in a hospital and connects with a provider in a different hospital using VTC. There is some support staff available to help provide technical support, assist in appointment booking, and coordinate crisis response as needed. This makes the emergency procedures in these settings different than VA or non-VA settings, which will be reviewed below.
CPT for PTSD via Telepsychology
Sergeant Smith (SGT S; pseudoname) was a noncommissioned officer with almost two decades of service, including five tours of combat. His position became emotionally more difficult when advances in technology allowed SGT S to see high-definition video footage of precision artillery strikes. He previously had never seen the carnage resulting from the attacks. He found himself haunted by these images and he would regularly have nightmares of what he saw.
The military mental health system uses a web-hosted outcome measure service called Behavioral Health Data Portal (BHDP; Brewin, 2013). Screening and outcome tracking measures were administered via BHDP immediately prior to the telepsychology sessions. A clinical interview was conducted and SGT S met full criteria for a diagnosis of PTSD (American Psychiatric Association, 2013). The PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013) was administered and he scored in the clinical range with a score of 66. Though SGT S had been exposed to multiple traumas, he was most distressed by the intrusive memories of those he had killed. A chief component of his PTSD symptoms was a prevailing sense of guilt and shame related to his combat experiences; this persistent sense of iniquity is commonly referred to as moral injury in the literature (Drescher et al., 2011). SGT S was presented with his treatment options and selected CPT (Resick, Monson, & Chard, 2007).
CPT is a manualized cognitive-behavioral treatment for PTSD that has been recommended by the updated VA/DoD Clinical Practice Guideline for the Management of PTSD (VA & DoD, 2017). A noninferiority trial demonstrated the CPT delivered via telepsychology can produce comparable outcomes to CPT delivered FTF (Morland et al., 2014). The goal of CPT is to foster cognitive modification. This involves coming to terms with what happened and integrating this new learning into the person’s belief set. This is typically accomplished through the use of a series of written thought records.
The military version of the CPT manual is available electronically, allowing for worksheets and handouts to be digitally shared with the patient in session via screen-sharing technology. Sharing hard copies of handouts and worksheets with patients via telepsychology can be difficult as it relies on the patient having access to a working printer or the technician printing the handout for the patient (Gros et al., 2013). Worksheets and handouts can be directly sent to the client via encrypted e-mail when that is an option. Some providers also choose to mail a paper copy of worksheets to the client. Typically, before a first session, it is unclear which handouts will be needed with a new patient, and the patient receiving a handout from the technician prior to the session can interrupt the flow of therapy. Also, staffing changes, off-site factors, and responsiveness of the technician can affect the timeliness and availability of technicians to provide worksheets to patients on behalf of the therapist. To account for this, the therapist relied on screen-sharing technology and SGT S’s smart camera phone. The therapist had him take pictures of the computer monitor to capture handouts and the impact statement instructions to facilitate his homework. It is also hoped that taking a picture of the computer monitor will make it harder to lose the information, as people can misplace worksheets, but tend to not lose their smartphones.
To aid with psychoeducation and to test his openness toward incorporating smartphone application technology into treatment, SGT S was provided with the VA application PTSD Coach. SGT S used the smartphone app to access high-quality psychoeducation materials, which he used to talk with his family about his PTSD. He responded well to the application, as there was some difficulty getting the paper CPT worksheets to the patient at the remote site. Consequently, he was introduced to an alternate resource in session. CPT Coach (Pospos et al., 2018; Shore et al., 2014) is a smartphone application that can be used as a supplement to psychotherapy with a therapist who is providing CPT. The CPT Coach application contains the worksheets and handouts from the CPT manual in a free smart phone app, which SGT S preferred as an alternative to hand-written worksheets.
Within the application, SGT S was able to log his stuck points, view electronic copies of the handouts discussed, and complete thought logs to target his PTSD-related beliefs. In session, Socratic questioning was used to evaluate his stuck points; this involved a series of introspective questions to encourage help him break down his thoughts and come to new conclusions on his own. In lieu of a white board, screen-sharing technology allowed the therapist to share a blank word processor document that could be used to weigh evidence and break down concepts. For example, to address his belief that he was a monster we typed out on the shared screen our jointly derived definition of a monster and then used that to evaluate where he fit the definition. We also created a shared definition of the opposite construct: his being a good person. This definition was saved and whenever new evidence of his being a good person was encountered it was added to the word document in session. This accrual of evidence allowed for an aggregating effect on his stuck points and reduction in self-blame. This, paired with continued use of CPT worksheets housed in the CPT Coach app, allowed for his main stuck point to be addressed and restructured. SGT S’s PTSD symptoms gradually decreased to the subclinical range across eight sessions of CPT, with his initial PCL-5 score of 66 decreasing to a 23. He reported decreases in marital problems and increases in life satisfaction. Based on his improvements, he decided to not retire from military service and he extended his contract with the army. At follow up, his PTSD symptoms were still in the subclinical range.
Collaborative Assessment and Management of Suicidality (CAMS) via Telepsychology
Gros, Veronee, Strachan, Ruggiero, and Acierno (2011) and Gros et al. (2013) provide excellent descriptions on how to manage risk in home-based telepsychology. Captain C (CPT C; pseudoname) was a 28-year-old soldier with 7 years of service who presented as a medical referral for individual psychotherapy due to depressive symptoms. CPT C reported his persistent suicidal thoughts had recently increased in frequency and intensity in context of a breakup as well as mounting financial concerns and occupational stressors associated with a new position. He had recently begun drafting a suicide note and had purchased a pistol. BHDP (Brewin, 2013) was unavailable at the remote site at the time of the encounter so paper-and-pencil versions of the measures were administered and then scanned and sent via encrypted e-mail to the provider by a psychiatric technician. At intake, his score on the Patient Health Questionnaire (Löwe, Kroenke, Herzog, & Gräfe, 2004) was a 23 (severe depression), baseline Columbia-Suicide Severity Rating Scale (C-SSRS; Posner et al., 2011) was a 5 (lifetime thoughts of suicide with plan or actual behavior endorsed), and current C-SSRS was a 4 (past month thoughts of suicide with some intent endorsed). CPT C met criteria for major depressive disorder but the initial focus of clinical attention was suicidal ideation.
The majority of the initial session was used to assess risk of harm to self; this was completed through use of the CAMS (Jobes, Moore, & O’Connor, 2007). CAMS is an evidence-based clinical approach for assessing, treating, and tracking suicidal risk. The framework is designed to strengthen the therapeutic alliance and to enhance patient motivation through a highly interactive assessment process that is guided by a tool called the Suicide Status Form (SSF; Jobes, 2006). The SSF uses Likert and qualitative items to assess psychological pain, stress, agitation, hopelessness, self-hate, and overall suicide risk. Traditionally CAMS is performed with the clinician sitting side by side with the patient, and the SSF is handed back and forth between the dyad during the assessment.
Prior to beginning the intake session with the patient, the initial SSF was sent from the provider to the technician at the remote site based on the positive suicide screen. Following introductions and consents, CAMS was introduced to the patient. The traditional protocol was discussed as well as the necessary adaptations to conduct the treatment through telepsychology. Specifically, the patient was told a single SSF was meant to be shared and coauthored by the patient and clinician, but would be completed together on two separate copies by each participant during session. As CPT C discussed the constructs and wrote responses on his copy of the form, the clinician wrote the same responses on her copy. In a traditional face-to-face session, this process usually takes 10-15 min. An additional 5-10 min was needed in the telepsychology session to ensure the dyad was accurately recording identical responses.
The interim SSF tracking sheets were completed in the same fashion at each encounter. The provider would e-mail the form to the technician at the remote site, and the technician would print the form and provide it to the patient before he entered the treatment room. Information gleaned from the qualitative responses on the SSF was used to inform treatment interventions in subsequent sessions. The treatment plan and crisis response plan were similarly updated as necessary. At the conclusion of each session, CPT C and the provider kept their own copy of the form, and the therapist’s worksheet was uploaded into the soldier’s electronic medical record for documentation purposes.
The first several sessions were marked by significant declines in severity ratings on the tracking sheets. After five sessions, CPT C was no longer reporting suicidal thoughts. The subsequent session focused on understanding warning signs for possible future episodes of suicidal ideation and additional coping skills to address inevitable life stressors. During the final session, the final SSF was compared side by side with the initial SSF to serve as a visual aid of the patient’s treatment progress.
Cognitive-Behavioral Therapy for Insomnia via Telepsychology
Specialist Martinez (SPC M; pseudoname) was a 29-year-old enlisted service member with 3 years of service who presented with symptoms of anxiety and sleep disturbance after sustaining injuries as a result of a physical assault by her husband. Following the assault, she began psychotherapy with a telepsychology psychotherapy provider and telepsychology prescribing psychologist. Trials of two medications for sleep were unsuccessful due to next-day sedation, so she was referred to a cognitive-behavioral treatment of insomnia (CBT-I) group delivered through telepsychology.
CBT-I is an evidence-based treatment that has been shown to be effective for individuals with insomnia. Reviews of the literature show that CBT-I is a standard of care treatment for chronic and severe insomnia (e.g., Morin et al., 1999), with clinical gains that last over the long term. The original protocol described by Perlis, Jungquist, Smith, and Posner (2005) is delivered over eight sessions and contains presentation of the rationale for the treatment, sleep hygiene guidelines (e.g., maintain a regular sleep schedule, avoid alcohol in the evening), diaphragmatic breathing, relaxation strategies, and worry control in later sessions. Sleep diaries are used to calculate sleep efficiency, or the amount of time one is sleeping while in bed, which is ideally in the low 90 percentiles. Importantly, individuals undergoing CBT-I are given a sleep schedule to promote consolidated sleep that initially begins with “prescribing” time in bed to be no more than the average time asleep over 2 weeks according to the sleep diary. Few studies have reported using telehealth for delivery of CBT-I, but one study reported positive results with group CBT-I delivered to veterans (Gehrman, Shah, Miles, Kuna, & Godleski, 2016).
Group CBT-I
SPC M initially engaged in group CBT-I through telepsychology. She was one of six members of a group that met in a conference room with a large-screen TV. The psychologist facilitator of the 5-week, 1.5-hr/week group originated a video-conferencing call to the conference room where participants sat around a large table. During the first session, SPC M told group members that she slept between 4 and 6 hr, on average, per night and her goal was to sleep 8 hr/night. Screen sharing technology was used to show a brief PowerPoint slides presentation relevant to each group session topic. Group success relied on a behavioral health technician at the receiving site to set up the technology, deliver packets of information used for the group, and be available in case of technology glitch. In subsequent groups, the technician collected sleep diaries from participants, scanned and sent them to the group facilitator via encrypted e-mail at the beginning of each group. An electronic sleep diary embedded in a smart phone application called CBT-I Coach developed by the Veteran’s Administration and the Telemedicine and Advanced Technology Research Center (TATRC) can be used to record daily diaries and transmit the compiled information directly to the provider, but this application could not be used in this instance because of installation security concerns of nonencrypted transmission.
The second through fifth groups followed a brief CBT-I protocol. Participants were encouraged to use Breathe to Relax, a smart phone application, also developed by TATRC, to practice diaphragmatic breathing daily. SPC M reported early gains in sleep efficiency during the first 2 weeks in the group that were not maintained during the last two groups. She ended with a sleep efficiency of 80.5% and a total sleep time of 6 hr and requested an individual therapist to continue CBT-I.
Individual CBT-I
At intake, SPC M’s anxiety had heightened and she endorsed a suicidal wish to die within the last month, which she indicated was “not a real wish to die,” but a “wish my problems would go away.” She and her primary therapist were working on these problems in weekly psychotherapy, but both agreed she should continue CBT-I. She was also taking hydroxyzine (25 mg) as needed for acute anxiety, but denied using it prior to bedtime. During the intake, screen sharing was used to reinforce how to fill out a sleep diary and to show her a colorful list of sleep hygiene guidelines. She indicated difficulties with nocturnal awakenings when a roommate came in during the night. She described two nightmares/month related to the physical assault and noted functional difficulties during the day. She estimated receiving 6.5 hr of sleep per night with a latency of greater than 60 min and described functional difficulties at work (e.g., falling asleep at work, yawning often, and reduced attention and concentration).
Over the course of three follow-up sessions conducted over 6 weeks, SPC M kept her sleep diary consistently and gained skills calculating her own sleep efficiency. She learned to use problem solving to address nocturnal awakenings, some of which she discovered were due to hip pain, which she addressed with her primary care practitioner. With concomitant psychotherapy, her nightmares ceased. Her sleep efficiency grew from 82% to 97% at the end of treatment and she was able to reach her goal of an average of 7.5 hr of sleep/night. Her scores on the Insomnia Severity Index (ISI; Bastien, Vallières, & Morin, 2001; administered via BHDP; Brewin, 2013) decreased from the clinical insomnia range (ISI = 17; moderate severity) at the beginning of the group to a subthreshold range (ISI = 11) over the course of treatment.
This example of CBT-I in the group and individual setting illustrates the use of screen sharing technology and of treatment-related phone applications. Because of security and privacy concerns with unencrypted transmission in a military population, the sleep diary in the CBT-I Coach was not used. Transmission of the sleep diary in this setting required a behavioral health technician to send the document through encrypted e-mail. Especially in the group setting when patients may arrive at different times prior to the group (i.e., some close to the start time of the group), there can be interruption from this transmission in group start times while the provider accesses the e-mail and prints the diary at one end and the hard copy is returned to the patient in the group at the other. Another issue with using telepsychology in the group setting, in general, is that real-time interactions are not mimicked as well for the leader. That is, when a person speaks, the static camera does not move to track who is speaking and focus on that person as experienced in a FTF group. Our organization subsequently tested a more sophisticated system with voice and movement tracking that was able to simulate a group more fully. However, this system is not without its downsides in that it took some time for participants to adjust to the camera moving to focus on them when they spoke. SPC M adapted to the telepsychology setting well. This example highlights the successful delivery of CBT-I through telepsychology in both group and individual settings.
Pharmacotherapy for PTSD via Telepsychology
First Lieutenant Reynolds (1LT R; pseudoname) was a 27-year-old Army infantry officer with less than 2 years of military service who presented for a medication evaluation. He reported symptoms of depression, irritability, and sleep difficulties connected to his combat deployment to Afghanistan. 1LT R’s deployment occurred less than 9 months following his arrival to his first duty station. As an infantry officer, 1LT R led an infantry platoon on over a hundred combat missions in hostile villages across Afghanistan. Exposed to multiple traumatic events during his deployment, within months of returning home, 1LT R manifested many symptoms associated with PTSD to include intrusive thoughts, dysphoric mood, hyperarousal, hypervigilance, and nightmares. His score on the PCL-5 was 63 at intake and he was formally diagnosed with PTSD.
Prior to presenting for a medication evaluation, 1LT R had completed a partial course (5 sessions) of prolonged exposure therapy. Although he acknowledged that the therapy was helpful, he found it subjectively too difficult to “constantly talk about what happened” and became frustrated with having to take time off from work every week for treatment. He felt that pharmacotherapy would be a better treatment option for him.
1LT R was prescribed sertraline 50 mg daily for his global symptoms of PTSD and prazosin 1 mg each night for his nightmares. Sertraline is a first-line pharmacological intervention for PTSD and is approved by the Food and Drug Administration (FDA) for the disorder. Prazosin is an alpha-adrenergic antagonist that was originally developed to treat hypertension. However, a number of clinical trials have shown it to be effective at reducing nightmares (Aurora et al., 2010; Nappi, Drummond, & Hall, 2012), although recent concerns have been raised regarding its efficacy (Raskind et al., 2018). Prazosin is not FDA approved for nightmares or PTSD. During the maintenance phase of treatment, 1LT R reached a dose of 150 mg of sertraline and 5 mg of prazosin. He also took over-the-countermelatonin on an as-needed basis.
In addition to dealing with PTSD, 1LT R was also suffering from back and neck pain and migraines related to an injury he sustained during his deployment. Consequently, he was prescribed a nonsteroidal anti-inflammatory (back and neck) and a triptan (migraines) by his primary care manager. Prior to prescribing medication for his PTSD, a discussion about the potential risks associated with drug-drug interactions occurred. 1LT R and the prescribing psychologist discussed the increased risk of internal bleeding and serotonin toxicity related to his medication regimen (common risks associated with serotonergicantidepressants and medications regularly used for pain management). During the discussion, the drug interaction program Lexicomp was used and the results were shared with the patient virtually. Lexicomp is an online tool that provides important information about potential interactions between medications and offers suggestions to prevent interactions from occurring. Similar programs are available at no cost to the user. As noted in the first case study, telepsychology providers have the capacity to share their computer screen images with their patients. Sharing the details of potential drug-drug interactions visually with the patient resulted in a more thorough and thoughtful discussion regarding potential interactions compared to a typical one-sided psychoeducational exchange. 1LT R stated, “I appreciate you reviewing these potential interactions in detail. If I had not seen them I probably would have forgotten what you said.” In this provider’s experience (author BAM), similar feedback from other patients is common. Discussions about potential drug-drug interactions is important in psychopharmacological practice, however, the level of retention and understanding of this material by patients is oftentimes uncertain. Seeing this key information “on screen” during the discussion appears to increase understanding and retention.
In addition to psychoeducation regarding potential drug-drug interactions, through the same screen-sharing process used for the drug information session, data were shared virtually with 1LT R regarding his progress. The BHDP tracks serial assessments of a variety of instruments to include the PCL-5. Each session 1LT R was able to see his current PCL-5 score and compare it to past scores. Over the course of 9 months he had 12 unique PCL-5 scores, which trended downward with the highest score being at the time of evaluation (63) and his lowest score being at Session 12 (28). Session 12 was the last session of the maintenance phase of treatment. 1LT Reynolds found the ability to track the trajectory of his scores to be highly reinforcing. At one point he stated:
When I can see my scores on the screen and watch how the scores go down over time, it becomes like a game for me. It makes me want to see how far I can go . . . how low I can get my scores . . . how quickly I can get better.
Research has shown that routine use of outcome measures, and the sharing of the results of those outcome measures, increases patient treatment adherence and improves outcomes (Lambert, 2013). Handouts were not used with this patient.
Discussion
An advantage of telepsychology is the increased ability to provide services to patients in low resourced environments (Landry-Poole et al., 2017). This can also decrease the time costs of attending treatment as the individuals treated in this study did not have to venture away from work to receive care. Further, telepsychology allowed for people in need of research-supported treatment interventions to receive quality treatments from providers who were several hours away. Consistent with previous research findings, delivering manualized treatments via telepsychology can be just as effective as FTF care (Shore et al., 2018); however, as noted in the introduction, adaptations to VTC are needed. The above cases illustrate the commonly used adaptations well (Table 1).
Self-Report Questionnaires
The use of self-report questionnaires (e.g., objective standardized outcome tracking measures and self-monitoring logs) is a part of evidence-based assessment, which is a critical component of evidence-based practice (Beidas et al., 2015; Brewin, 2013). Administering self-report questionnaires from a distance via telepsychology is different than being FTF, but there are several workable solutions.
As illustrated in the CBT-I case example, measures like sleep logs can be filled out on paper forms and then scanned and transmitted through encrypted messenger or emails. This process requires planning ahead and is not feasible for every provider or setting. Some of the smartphone applications discussed above have embedded self-report questionnaires and tracking measures, but there are questions about transmitting the data from the applications to the therapist in a secure manner. As an alternative option, a patient could use the measure in the application and read the clinician a summary. For example, the sleep log in the CBT-I coach will calculate an overall sleep efficiency quotient. A patient could describe his week and then verbally provide the summary figures from the applications. That sleep efficiency figure can then be used to inform that week’s sleep schedule recommendations.
Alternatively, patients may take the standardized measures online through secure web-based assessment surveys, such as BHDP described above. A number of web-based practice management programs allow for the secure transmission and collection of standardized measures and questionnaires but require foreplanning and coordination. Of course, HIPAA concerns would necessitate a business associate agreement and reputable companies will easily be able to provide these to consumers. The above cases benefitted from having a paraprofessional psychiatric technician on site with the patients to help facilitate the BHDP process. If this process was conducted without a technician’s help, clientele would need to be trained to complete the web-based surveys, which might be more difficult for some patients than others. There are alternative procedures that can be used.
When having the patient complete the web-based survey prior to the session is not feasible, the measure may be verbally administered during the session. This will take up some of the session time and may force a clinician to be more judicious about which outcome-tracking measures are utilized. Additionally, there are considerations about whether verbal administration of a measure affects item response, or if this method of administration is in keeping with measure standardization procedures. Future research may look at this topic to inform clinical practice. A list of free and psychometrically sound outcome measures can be found in a previous review (Beidas et al., 2015). As illustrated in the final case example, incorporating outcome tracking into the active components of treatment can enhance clinical outcomes, this being consistent with previous findings (Lambert, 2013).
Treatment Handouts
Treatment handouts and worksheets are a common component of manualized treatments (Shafran et al., 2009). Treatment handouts can be transmitted electronically, printed, filled out by hand, scanned, and sent back electronically. An additional option is that a treatment packet or patient manual can be mailed to the client or facility where the client is being seen to facilitate treatment-this strategy does require extra effort but it is a reliable method. A key take away is that using handouts in telepsychology requires forethought and planning-it is harder to be spontaneous with forms being used in telepsychology. In some cases, digital copies can be used; however, they are not always available and copyrights may limit a clinician’s ability to create a digital copy of an extant form from a treatment manual.
Conceptually, there is a difference between handouts and worksheets. Handouts are informational and easier to use with screen-sharing technology or when using secure e-mail. In the above cases, the providers used screen sharing to share a number of different figures, but this is different from having a handout someone can physically take away with them. A low cost solution is to have the patient take a picture of the screen to capture the information. Software-based solutions such as screen shots might also be workable solutions, depending on whether or not it is a shared device. Additionally, a photograph of a shared screen or a screen shot could be printed for later use.
Many of the newer smart phone applications (e.g., CPT Coach and CBT-I Coach) described above have the treatment handouts embedded in the application. Notably, these mobile health apps were developed by the VA or Defense Health Agency, and so they are secure and free to use. The user’s data is stored on their own device and many of these apps allow for personal identification number locking of the apps data. In these cases, one can reference the application and go over it together with the information already contained in their phones. The CPT Coach app also has an embedded ABC worksheet function that can be used as a thought record if verbally reviewed together. The patient described in the first case preferred this over the paper ABC worksheets as he was less worried about losing it or leaving it out where someone else might read it.
Some manualized treatments specify how the handouts are to be used. This requires careful accommodation when utilizing telepsychology. For example, in CAMS the dyad is supposed to collaboratively fill out the forms together. This can be difficult from a distance but the above case illustrates how planning can allow for the therapist and patient to each have their own copy of requisite forms that they complete together.
In-Session Examples
Use of in-session examples can help increase the attentional focus on the information being discussed and the intervention being used (Beck & Haigh, 2014). This is most often completed through use of screen-sharing technology that is a core function in most telepsychology VTC software. When digital copies of forms are illustrations or figures are available, they can be shared via screen sharing. When the provider needs to create something novel, a blank word processor document can be used as a white board to track or list out the items that might normally be written out in session. In the CPT example, the therapist used this white board function to list out a shared definition of what it meant to be a monster and what it meant to be a good person. Other common software applications can be used as a white board with screen sharing. For example, the common paint applications (e.g., Microsoft Paint, Corel Painter, Krita, MyPaint; SketchBook) can be used to sketch out a concept like a habituation curve. Additionally, screen sharing an Internet search engine can allow for some new in-session example options as the clinician and patient directly research topics together. For example, if a patient is exceedingly hopeless, thinking they cannot go on after a major setback, a provider could look up stories about people who had similar setbacks and went on to have meaningful and happy lives. Providers might choose to address web browser settings so that previous search terms do not autopopulate the search field.
Other Considerations
There may need to be additional modifications and considerations for group therapies. In the CBT-I case example, the group facilitator noted a limitation with the technology. A static camera did not focus on the person who was speaking and a camera that focused in on motion had a lag time. Alterations in the pacing and practice of delivering group evidence-based treatments via telepsychology are likely required. Further research and innovation on the topic are needed.
Additional considerations include patient access to a working printer and scanner. This might not be a static variable and having a backup plan for when technology fails is a requirement in telepsychology care (Landry-Poole et al., 2017). The key is to be flexible and think ahead wherever possible.
The setting where the telepsychology occurs is another consideration; some of the strategies discussed above may need to be modified to suit the needs and resources of different treatment systems of care. Active duty military hospitals have different procedures, than VA home-based telepsychology. VA homebased providers have safety procedures that start before the first session where there is an initial thorough safety screen regarding the suitability and safety of the homebased setting and the risk and protective factors of the clients being served to ensure that telepsychology is an appropriate option. Risk management continues in on ongoing manner, where at the start of the session they verify the physical location of the client so they know where to dispatch emergency services if needed, they verify the callback phone number in case they get disconnected, and they verify the phone number of the emergency contact. In military hospitals, the client is already in a hospital and on-site support staff can physically assist in an emergency.
Both of these procedures can be different from what a psychologist in independent practice might do, and other health systems might have other resources and limitations. Psychologists in independent practice will need to attend to practical and legal matters that we have not elaborated on: these include licensing requirements, HIPAA considerations, and practical considerations related to elements like screen sharing. Brooks and colleagues (2013) have a good review of how to overcome some of the common barriers that exist. Also, the scientific principle or using a pilot study or test run to find and work out potential barriers can be useful. We have attempted to draw out and demonstrate a variety of ideas and principles such that some, but not all, of this might be applicable depending on the setting and available resources. Further innovation is likely to be needed, as creativity and pragmatism is a hallmark of telepsychology. Clinicians in various settings are encouraged to follow the recommendations of Gros et al. (2013) and to make careful adaptations to the following domains use of self-report questionnaires, treatment handouts, and in-session examples, to ensure high quality care is being delivered.
Summary and Future Directions
Evidence-based psychotherapies and pharmacological therapies can be delivered effectively through telepsychology; this is well established in the literature. A recent survey of telepsychology experts demonstrated the need for guidance on how to adapt these practices for telepsychology with a specific focus on three domains: use of self-report questionnaires, treatment handouts, and in-session examples (Gros et al., 2013). The case series above demonstrated feasible workable solutions to address these domains of treatment. Further research and innovation on the topic is needed. Treatment developers may wish to provide guidance on the topic; additionally, they may choose to make available digital materials that can be used in telepsychology to facilitate the intended treatment effects. Research funders may choose to offer grant funding to study and develop adaptations to existing treatment protocols for telepsychology. These cases demonstrate the value of high quality smart phone applications (e.g., PTSD Coach, CPT Coach, PE Coach, CBT-I Coach; Pospos et al., 2018; Shore et al., 2014) that can be used in conjunction with telepsychology to help address the three different domains. Further research on how to best use these applications is needed and development of additional applications will also likely be needed. Journal editors may seek to solicit and publish more practice-focused literature on the topic of adapting evidence-based care to telepsychology settings to further clarify this process for front-line telepsychology clinicians. Graduate-level and postgraduate-level training programs may choose to include supervised training in these adaptations as telepsychology is a growing field of practice.
Statement of Human Rights
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (include name of committee + reference number) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Acknowledgements
Corresponding Author
Scott H. Waltman received his PsyD in Clinical Psychology from Pacific University in Oregon. He currently works as a clinical psychologist at the United States Army’s Warrior Resiliency Program, Regional Health Command-Central. His areas of interest relate to the dissemination and implementation of evidence-based care to underserved populations and practice-based research.
Julie M. Landry received her PsyD from Argosy University in Atlanta, GA. She currently works as a clinical psychologist at the United States Army’s Warrior Resiliency Program, Regional Health Command-Central and the United States Army Reserve Psychology Consultant to the Surgeon General. Her areas of professional interest include aeromedical psychology, evidence-based assessment and treatment of suicidality, and telepsychology
Lynette A. Pujol received her PhD in Counseling Psychology from the University of Iowa and her MS in Clinical Psychopharmacology from Fairleigh Dickinson University. She currently serves as a clinical psychologist with prescriptive authority at Brooke Army Medical Center and the United States Army’s Warrior Resiliency Program, Regional Health Command - Central. Her areas of interest include psychopharmacology, telepsychology, and the application of technology to self-management of behavioral health conditions.
Bret A. Moore received his MS in Clinical Psychopharmacology from Fairleigh Dickinson University and his PsyD from Adler University. He currently serves as the Deputy Director of the United States Army’s Warrior Resiliency Program, Regional Health Command - Central. He is also a prescribing psychologist and provides psychopharmacological services to active duty military personnel. His areas of personal interest include pharmacotherapy, complementary and alternative interventions for posttraumatic stress disorder, and the facilitation of posttraumatic growth in those who experience trauma.
The views expressed herein are those of the authors and do not reflect the official policy or position of Army Regional Health Command-Central, the United States Army Medical Department, the United States Army Office of the Surgeon General, the Department of the Army, the Department of the Air Force, and Department of Defense or the United States Government.Correspondence concerning this article should be addressed to Scott H. Waltman, PsyD, Warrior Resiliency Program, Regional Health Command-Central, 7800 IH-10 West, Suite 300, San Antonio, TX 78230
Email: [email protected]
Publication History
Received July 31, 2019
Revision received September 11, 2019
Accepted September 13, 2019
First published online October 3, 2019
Acierno, R., Gros, D. F., Ruggiero, K. J., Hernandez-Tejada, B. M., Knapp, R. G., Lejuez, C. W., . . . Tuerk, P. W. (2016). Behavioral activation and therapeutic exposure for posttraumatic stress disorder: A noninferiority trial of treatment delivered in person versus home-based telehealth. Depression and Anxiety, 33, 415–423. https://doi.org/10.1002/da.22476
Acierno, R., Knapp, R., Tuerk, P., Gilmore, A. K., Lejuez, C., Ruggiero, K., . . . Foa, E. B. (2017). A non-inferiority trial of prolonged exposure for posttraumatic stress disorder: In person versus home-based telehealth. Behaviour Research and Therapy, 89, 57–65. https://doi.org/10.1016/j.brat.2016.11.009
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.
Augusterfer, E. F., Mollica, R. F., & Lavelle, J. (2015). A review of telemental health in international and post-disaster settings. International Review of Psychiatry, 27, 540–546. https://doi.org/10.3109/09540261.2015.1082985
Aurora, R. N., Zak, R. S., Auerbach, S. H., Casey, K. R., Chowdhuri, S., Karippot, A., . . . the Standards of Practice Committee, & the American Academy of Sleep Medicine. (2010). Best practice guide for the treatment of nightmare disorder in adults. Journal of Clinical Sleep Medicine, 6, 389–401.
Bashshur, R. L., Shannon, G. W., Bashshur, N., & Yellowlees, P. M. (2016). The empirical evidence for telemedicine interventions in mental disorders. Telemedicine and e-Health, 22, 87–113. https://doi.org/10.1089/tmj.2015.0206
Bastien, C. H., Vallières, A., & Morin, C. M. (2001). Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Medicine, 2, 297–307. https://doi.org/10.1016/S1389-9457(00)00065-4
Beck, A. T., & Haigh, E. A. (2014). Advances in cognitive theory and therapy: The generic cognitive model. Annual Review of Clinical Psychology, 10, 1–24. https://doi.org/10.1146/annurev-clinpsy-032813-153734
Becker, E. M., Smith, A. M., & Jensen-Doss, A. (2013). Who’s using treatment manuals? A national survey of practicing therapists. Behaviour Research and Therapy, 51, 706–710. https://doi.org/10.1016/j.brat.2013.07.008
Beidas, R. S., Stewart, R. E., Walsh, L., Lucas, S., Downey, M. M., Jackson, K., . . . Mandell, D. S. (2015). Free, brief, and validated: Standardized instruments for low-resource mental health settings. Cognitive and Behavioral Practice, 22, 5–19. https://doi.org/10.1016/j.cbpra.2014.02.002
Brewin, B. (2013). Pentagon directs defensewide use of Army’s mental health data portal. Nextgov Newsletter. Retrieved fromhttp://www.nextgov.com/cloud-computing/2013/10/pentagon-directs-defensewide-use-armys-mental-health-data-portal/72953/
Brooks, E., Turvey, C., & Augusterfer, E. F. (2013). Provider barriers to telemental health: Obstacles overcome, obstacles remaining. Telemedicine and e-Health, 19, 433–437. https://doi.org/10.1089/tmj.2013.0068
Department of Veterans Affairs and Department of Defense. (2017). VA/DoD Clinical practice guidelines for the management of posttraumatic stress disorder and acute stress disorder. Retrieved fromhttps://www.healthquality.va.gov/guidelines/MH/ptsd/VADoDPTSDCPGFinal.pdf
Drescher, K. D., Foy, D. W., Kelly, C., Leshner, A., Schutz, K., & Litz, B. (2011). An exploration of the viability and usefulness of the construct of moral injury in war veterans. Traumatology, 17, 8–13. https://doi.org/10.1177/1534765610395615
Egede, L. E., Acierno, R., Knapp, R. G., Walker, R. J., Payne, E. H., & Frueh, B. C. (2016). Psychotherapy for depression in older veterans via telemedicine: Effect on quality of life, satisfaction, treatment credibility, and service delivery perception. The Journal of Clinical Psychiatry, 77, 1704–1711. https://doi.org/10.4088/JCP.16m10951
Frueh, B. C., Monnier, J., Grubaugh, A. L., Elhai, J. D., Yim, E., & Knapp, R. (2007). Therapist adherence and competence with manualized cognitive-behavioral therapy for PTSD delivered via videoconferencing technology. Behavior Modification, 31, 856–866. https://doi.org/10.1177/0145445507302125
Gehrman, P., Shah, M. T., Miles, A., Kuna, S., & Godleski, L. (2016). Feasibility of group cognitive-behavioral treatment of insomnia delivered by clinical video telehealth. Telemedicine and e-Health, 22, 1041–1046. https://doi.org/10.1089/tmj.2016.0032
Glueckauf, R. L., Maheu, M. M., Drude, K. P., Wells, B. A., Wang, Y., Gustafson, D. J., & Nelson, E. L. (2018). Survey of psychologists’ telebehavioral health practices: Technology use, ethical issues, and training needs. Professional Psychology: Research and Practice, 49, 205–219. https://doi.org/10.1037/pro0000188
Gros, D. F., Lancaster, C. L., López, C. M., & Acierno, R. (2018). Treatment satisfaction of home-based telehealth versus in-person delivery of prolonged exposure for combat-related PTSD in veterans. Journal of Telemedicine and Telecare, 24, 51–55. https://doi.org/10.1177/1357633X16671096
Gros, D. F., Morland, L. A., Greene, C. J., Acierno, R., Strachan, M., Egede, L. E., . . . Frueh, B. C. (2013). Delivery of evidence-based psychotherapy via video telehealth. Journal of Psychopathology and Behavioral Assessment, 35, 506–521. https://doi.org/10.1007/s10862-013-9363-4
Gros, D. F., Veronee, K., Strachan, M., Ruggiero, K. J., & Acierno, R. (2011). Managing suicidality in home-based telehealth. Journal of Telemedicine and Telecare, 17, 332–335. https://doi.org/10.1258/jtt.2011.101207
Hubley, S., Lynch, S. B., Schneck, C., Thomas, M., & Shore, J. (2016). Review of key telepsychiatry outcomes. World Journal of Psychiatry, 6, 269–282. https://doi.org/10.5498/wjp.v6.i2.269
Jobes, D. A. (2006). Managing suicide risk: A collaborative approach. New York, NY: Guilford Press.
Jobes, D. A., Moore, M. M., & O’Connor, S. S. (2007). Working with suicidal clients using the Collaborative Assessment and Management of Suicidality (CAMS). Journal of Mental Health Counseling, 29, 283–300. https://doi.org/10.17744/mehc.29.4.k881k101v0u79rqp
Lambert, M. J. (2013). Outcome in psychotherapy: The past and important advances. Psychotherapy, 50, 42–51. https://doi.org/10.1037/a0030682
Landry-Poole, J. M., Pujol, L., & Moore, B. A. (2017). Tele-behavioral health. In J. Beder (Ed.), Caring for the military: A guide for helping professionals (pp. 217–225). New York, NY: Routledge/Taylor & Francis Group.
Löwe, B., Kroenke, K., Herzog, W., & Gräfe, K. (2004). Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). Journal of Affective Disorders, 81(1), 61–66.
Mora, L., Nevid, J., & Chaplin, W. (2008). Psychologist treatment recommendations for Internet-based therapeutic interventions. Computers in Human Behavior, 24, 3052–3062. https://doi.org/10.1016/j.chb.2008.05.011
Morin, C. M., Hauri, P. J., Espie, C. A., Spielman, A. J., Buysse, D. J., & Bootzin, R. R. (1999). Nonpharmacologic treatment of chronic insomnia. An American Academy of Sleep Medicine review. Sleep, 22, 1134–1156. https://doi.org/10.1093/sleep/22.8.1134
Morland, L. A., Mackintosh, M. A., Greene, C. J., Rosen, C. S., Chard, K. M., Resick, P., & Frueh, B. C. (2014). Cognitive processing therapy for posttraumatic stress disorder delivered to rural veterans via telemental health: A randomized noninferiority clinical trial. The Journal of Clinical Psychiatry, 75, 470–476. https://doi.org/10.4088/JCP.13m08842
Morland, L. A., Mackintosh, M. A., Rosen, C. S., Willis, E., Resick, P., Chard, K., & Frueh, B. C. (2015). Telemedicine versus in-person delivery of cognitive processing therapy for women with posttraumatic stress disorder: A randomized noninferiority trial. Depression and Anxiety, 32, 811–820. https://doi.org/10.1002/da.22397
Nappi, C. M., Drummond, S. P., & Hall, J. M. (2012). Treating nightmares and insomnia in posttraumatic stress disorder: A review of current evidence. Neuropharmacology, 62, 576–585. https://doi.org/10.1016/j.neuropharm.2011.02.029
Nelson, E. L., Barnard, M., & Cain, S. (2003). Treating childhood depression over videoconferencing. Telemedicine Journal and E-health, 9(1), 49–55.
O’Reilly, R., Bishop, J., Maddox, K., Hutchinson, L., Fisman, M., & Takhar, J. (2007). Is telepsychiatry equivalent to face-to-face psychiatry? Results from a randomized controlled equivalence trial. Psychiatric Services, 58, 836–843. https://doi.org/10.1176/ps.2007.58.6.836
Perle, J. G., Langsam, L. C., Randel, A., Lutchman, S., Levine, A. B., Odland, A. P., . . . Marker, C. D. (2013). Attitudes toward psychological telehealth: Current and future clinical psychologists’ opinions of internet-based interventions. Journal of Clinical Psychology, 69, 100–113. https://doi.org/10.1002/jclp.21912
Perlis, M. L., Jungquist, C., Smith, M. T., & Posner, D. (2005).Cognitive behavioral treatment of insomnia: A session-by-session guide. New York, NY: Springer Science + Business Media.
Posner, K., Brown, G. K., Stanley, B., Brent, D. A., Yershova, K. V., Oquendo, M. A., . . . Mann, J. J. (2011). The Columbia–Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. American Journal of Psychiatry, 168(12), 1266–1277.
Pospos, S., Young, I. T., Downs, N., Iglewicz, A., Depp, C., Chen, J. Y., . . . Zisook, S. (2018). Web-based tools and mobile applications to mitigate burnout, depression, and suicidality among healthcare students and professionals: A systematic review. Academic Psychiatry, 42, 109–120. https://doi.org/10.1007/s40596-017-0868-0
Raskind, M. A., Peskind, E. R., Chow, B., Harris, C., Davis-Karim, A., Holmes, H. A., . . . Huang, G. D. (2018). Trial of prazosin for post-traumatic stress disorder in military veterans. The New England Journal of Medicine, 378, 507–517. https://doi.org/10.1056/NEJMoa1507598
Resick, P. A., Monson, C. M., & Chard, K. M. (2007). Cognitive processing therapy: Veteran/military version. Washington, DC: Department of Veterans’ Affairs.
Ruskin, P. E., Silver-Aylaian, M., Kling, M. A., Reed, S. A., Bradham, D. D., Hebel, J. R., . . . Hauser, P. (2004). Treatment outcomes in depression: Comparison of remote treatment through telepsychiatry to in-person treatment. The American Journal of Psychiatry, 161, 1471–1476. https://doi.org/10.1176/appi.ajp.161.8.1471
Shafran, R., Clark, D. M., Fairburn, C. G., Arntz, A., Barlow, D. H., Ehlers, A., . . . Wilson, G. T. (2009). Mind the gap: Improving the dissemination of CBT. Behaviour Research and Therapy, 47, 902–909. https://doi.org/10.1016/j.brat.2009.07.003
Shore, J. H., Aldag, M., McVeigh, F. L., Hoover, R. L., Ciulla, R., & Fisher, A. (2014). Review of mobile health technology for military mental health. Military Medicine, 179, 865–878. https://doi.org/10.7205/MILMED-D-13-00429
Shore, J. H., Yellowlees, P., Caudill, R., Johnston, B., Turvey, C., Mishkind, M., . . . Hilty, D. (2018). Best practices in videoconferencing-based telemental health April 2018. Telemedicine Journal and e-Health, 24, 827–832. https://doi.org/10.1089/tmj.2018.0237
Tuerk, P. W., Yoder, M., Ruggiero, K. J., Gros, D. F., & Acierno, R. (2010). A pilot study of prolonged exposure therapy for posttraumatic stress disorder delivered via telehealth technology. Journal of Traumatic Stress, 23, 116–123. https://doi.org/10.1002/jts.20494
Waltman, S. H. (2015). Functional analysis in differential diagnosis: Using cognitive processing therapy to treat PTSD. Clinical Case Studies, 14, 422–433. https://doi.org/10.1177/1534650115571003
Weathers, F. W., Litz, B. T., Keane, T. M., Palmieri, P. A., Marx, B. P., & Schnurr, P. P. (2013). The PTSD Checklist for DSM–5 (PCL-5). Scale available from the National Center for PTSD atwww.ptsd.va.gov
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
© 2020. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the associated terms available at https://publishing.aip.org/publications/journals/covid-19/.
Abstract
Telepsychology is a relatively well-established method for delivering care to underresourced environments. Much research has been done demonstrating the acceptability and effectiveness of telepsychology. Also, it has been shown that evidence-based psychotherapies and pharmacological therapies can be delivered effectively through telepsychology; however, there is a lack of information on how to best adapt the research-supported protocols to be delivered via telepsychology. A recent survey of telepsychology experts demonstrated the need for guidance on how to adapt these practices for telepsychology with a specific focus on three domains: use of self-report questionnaires, treatment handouts, and in-session examples (Gros et al., 2013). The presented case series demonstrates feasible solutions to address these domains of treatment such as using screen-sharing technology, incorporating other technology-based resources such as smart-phone applications, and using web-based methods to securely administer outcome tracking measures. Further research and innovation on the topic is needed.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer