Digital NHS Wales: a coding reliability analysis based on the voices of 22 978 patients and clinicians on the benefits, challenges and sustainability of video consulting

Abstract

Introduction

The use of video consulting (VC) in Wales UK has expanded rapidly. Previous VC evidence has been the subject of small-scale projects and evaluations. Technology Enabled Care Cymru is an all-Wales digital service and rolls out digital interventions and evaluates on large scales, thus capturing representative data sets across Wales, and therefore a wide range of National Health Service (NHS) specialties.

Objective

To extract and analyse narrative feedback from patients and clinicians using the NHS Wales VC Service for 6 months (September 2020 to March 2021).

Design

A coding reliability approach of a cross-sectional study was conducted.

Setting

From all health boards across Wales.

Participants

NHS patients and clinicians across primary, secondary and community care settings in Wales.

Results

Data were captured on benefits, challenges and sustainability of VC. A coding reliability analysis was used with six domain summaries materialising to include: ‘The Ease of VC’; ‘The Personal Touches’; ‘The Benefits of VC’; ‘The Challenges of VC’; ‘Technical Quality’; and ‘Recommendations & Future Use’. An additional 17 subdomains are included. Direct quotations from patients and clinicians are provided for context.

Conclusions

A total of 22 978 participants were included. These data help demonstrate that NHS remote service delivery, via the method of VC, can be highly satisfactory, well accepted and clinically suitable yielding many benefits. Despite this, the data are not without its challenges surrounding engagement and suitability for VC. The NHS Wales VC Service rolled out and evaluated at scale and demonstrates that VC has potential for long-term sustainability. For the future, use a ‘blended approach’ for NHS appointments that are clinically judged and centred on patient choice.

Correspondence to Gemma Johns; [email protected]

Strengths and limitations of this study

• This paper presents patient and clinician free text narrative boxes on a large scale considering experience of a relatively new digital service in National Health Service (NHS) Wales.

• The study is representative of Wales, in that it is an all-Wales study, across all health boards.

• The study is a mix of patient and clinician voices across all types of NHS specialties.

• Due to the voluntary and anonymised nature of the feedback data, it is unclear as to how often recurring participants completed the feedback, thus potentially skewing the sample.

• Due to the size of the study sample it was not possible to present both the narrative and quantitative findings together; however, access to these data is readily available on our website.

Background

Since 2020, the National Health Service (NHS) has seen a paradigm shift in the provision of healthcare services due to mandatory social distancing laws introduced because of the COVID-19 pandemic.^1–3 As a result, the UK along with the NHS observed a significant decrease in access to face-to-face appointments, and therefore an increase in remote services.⁴

Video consulting (VC) has accelerated through health and social care as one of the most common remote methods for conducting appointments with patients throughout the NHS, especially in Wales.^{5 6} VC within health services has been internationally used for decades, yet the unprecedented circumstances of the pandemic brought to light its widespread ability, use, value, benefits and challenges.¹

There is growing evidence that VC can deliver safe and timely care in many settings and offer significant benefits to the users.^{7 8} The use of VC permits services to continue across a wide range of healthcare conditions, appointment types, sociodemographic groups and health condition status.⁹ Furthermore, it is sometimes considered more suitable for reaching underserved and isolated populations.⁴ VC is reported to provide quality-ensured yet cost-effective care¹⁰ and treatment, while reducing patient waiting times and the likelihood of Did Not Attends and number of hospital admissions—ultimately relieving pressure on NHS staff and services.¹¹

However, the majority of evidence is based on pilot studies, with small and often highly selected samples, with limited questionnaire validity, ultimately casting speculation on its use, benefits and challenges across varied care sectors, specialties and circumstances.^{12 13}

There are often some concerns regarding the use of VC services within certain professions that rely on face-to-face physical examinations to make diagnoses, and the increased level of risk associated.¹⁴ These valid uncertainties highlight situations where sometimes it can be challenging to obtain the same level of accuracy when taking clinical measures via VC compared with obtaining them face to face. Not only this, but personal preferences in clinicians and patients can dictate whether or not a VC is used.

Therefore, the current evidence base suggests there is a need to continuously evaluate on a national level to allow for sustainable VC platforms to be embedded for the long term into health and social care systems where appropriate.^{15 16} Due to the need for a continual evaluation, Technology Enabled Care (TEC) Cymru as an all-Wales digital service rolled out the NHS Wales VC Service as a national emergency response to the COVID-19 pandemic.¹⁷ The evaluation spans a large and representative scale basis across a wide range of NHS healthcare sectors across all health boards in Wales. To contribute to the current evidence base, TEC Cymru works in partnership with the Welsh Government, academics, third sector and local health boards and trusts in Wales to adopt a clinically driven and data-informed approach to their digital service roll-out, spread and evaluation.

Aims and methods

The aim of this study was to explore the benefits, challenges and sustainability of VC from the perspective of Welsh NHS clinicians and patients by conducting a coding reliability analysis, and presenting the narrative feedback received from 22 978 participants during a 6-month period (September 2020 and March 2021). This period was chosen as a ‘mid-point’ from a larger ongoing VC evaluation during the COVID-19 pandemic to gain a better understanding of VC without the influence of initial issues during the earlier months of VC being rolled out, and current changes such as VC being scaled up with a focus on blended consultation approaches.

Methods

Design, setting, participants

This paper presents the all-Wales data captured across all seven health boards and one trust (see online supplemental appendix 1) across a range of NHS healthcare settings within primary, secondary and community care (see online supplemental appendices 2 and 3). This is a coding reliability analysis of VC experience feedback captured in a larger cross-sectional study¹⁸ held by the NHS Wales VC Service, TEC Cymru.^{17 19} Participant eligibility included NHS clinicians and patients using VC in NHS Wales (see online supplemental appendices 4–9).

Measures

This paper presents national (all-Wales) data from free text narrative boxes from a cross-sectional feedback study. The feedback appeared as internet browser pop-ups at the end of each VC appointment—one per clinician and patient and completed immediately as live data to reflect the use, benefits, challenges and sustainability of VC (see online supplemental files 1 and 2).

Sampling

Opportunity sampling was used due to accessibility of the VC intervention and ability to capture data at the end of each consultation via an online feedback link. There is acknowledgement of the risks surrounding sampling in this way, when considering the feedback being completed by those more willing, thus sharing potentially more extreme ‘positive’ or ‘negative’ data towards VC, potentially missing out ‘neutral’ responses of those individuals in the middle. To limit this, TEC Cymru conducts multiple phases of re-evaluation using a phased approach to their research and evaluation work (see online supplemental file 3), which provides ample opportunity across their digital interventions to explore a wider range of methodologies and study types.

Patient and public involvement

No patient or public involvement as survey work and during the emerging roll-out did not have patient and public involvement (PPI) team., as time has gone on we now have a specific PPI team and young representatives.

Analysis

For the data discussed in this paper, there are a total of 22 978 clinician and patient feedback narrative submissions. Using steps for a coding reliability analysis,²⁰ the data were familiarised by three researchers of the TEC Cymru team (GJ, BW, MW), codes were manually identified and generated and placed into an Excel sheet for manageable order (due to large numbers). Domain summaries were then generated from the data, reviewed and defined and the report was produced following a recursive process of movement between the phases, ensuring quality and rigour, with an additional 20% validation check on all data by the national clinical lead for Wales (AA).

Results

A coding reliability analysis of the free text narrative data collected at the end of VC feedback was conducted. From the 22 978 patient and clinician responses captured during September 2020 and March 2021, six domain summaries materialised with an additional 17 subdomains. These include: the ease of VC; the personal touches; the benefits of VC; the challenges of VC; technical quality; and recommendations and future use. Direct quotations from patients and clinicians are provided. Each quotation is referenced to describe either the patient by their age range, gender, health board/trust, healthcare specialty and type of appointment (eg, first appointment, follow-up), or the clinician by their professional occupation and the health board/trust in Wales in which they are based. The domains are analysed in order of the most common comment/feedback due to the voluntary responses.

Domain summary 1: ease of use

Patients and clinicians generally rate their VC as high in quality¹⁶ and their free text narratives reflect this level in terms of high satisfaction and acceptability in relation to both technical and overall experience. For example, when patients and clinicians rate their VC as ‘excellent’, ‘very good’ or ‘good’ this is often paired with positive comments in relation to either the VC’s technical performance as a VC platform or the overall experience of using VC as a healthcare delivery service.

Ease of technical use

One of the most common subdomains associated to the platform’s technical performance was that of ‘ease of use’. It was often stated that the VC platform used in NHS Wales (Attend Anywhere) was ‘easy to use’ for both patients and clinicians.

Easy to use, lots of good information. (Parent of patient under 12 years, ABUHB, Physiotherapist, paediatrics and child health, Advice)

It was easy to use, and appropriate to use during the pandemic. (Patient, Female, HDUHB, 25–44, Midwife, Obstetrics and gynaecology, First appointment)

In addition, this ease of use was expressed as a ‘surprise’ to some, in that both patients and clinicians found the VC platform much easier to use than they initially anticipated, and in some instances, this exceeded expectation.

More effective than I expected a non-face to face appointment to be. (Patient, Male, 45–64, HDUHB)

This is my first experience of a video call, so I was pleasantly surprised. (Patient, Female, 64–80, BCUHB, Doctor, Follow-up)

First time to use video call I was very impressed, better than expected. (Patient, ABUHB, Podiatrist, Follow-up)

Ease of experience

For some clinicians, it was felt that having access to a VC platform was ‘easier’ for some of their patients than a face-to-face appointment would be. This was especially apparent in terms of patient experience and their personal circumstances, and those with access difficulties, anxiety issues or complex home situations that were made more convenient with VC.

Easier to access with social anxiety. (Doctor, CVUHB)

Very helpful for autistic patient. (Dentist/dental nurse, SBUHB)

This was also expressed in more depth by the patients themselves, whom in addition felt VC was better than attending a face-to-face appointment, such as making the patient feel safer, less stressed and more empowered, as opposed to their prior experience of face-to-face appointments.

Easier and safer than going to the hospital. I didn’t have to take much time off work. (Patient, SBUHB, Female, 25–44, Dietician, First appointment)

Just as good as a face-to-face meeting and to be honest I felt like I was being listened to far more than when I have been in face-to-face meetings on the same subject. (Parent of patient, ABUHB, Female, under 12 years, Nurse, Mental health, Advice)

Ease and unique for collaborations

Clinicians comment that the ‘ease’ of the VC platform and its positive associations to patient experience provides an additional unique opportunity. This opportunity is the ability to link up others to the video call, thus enabling multidisciplinary appointments to take place. This is felt to be unique in the sense that this collaborative approach would not have been possible if conducted face to face, thus in turn produces additional advantages and improved outcomes for patients, families and clinicians.

It was easy to join both my patients and other colleagues in. (Doctor, HDUHB)

Also, his Wife was able to join session—significant information shared by Wife today. (SLT, BCUHB)

It also means that in some instances, there is an increase in patient or parental onus which is perceived as an additional advantage to patient care.

Parents have to take a more proactive role than they might in clinical session. (SLT, ABUHB)

Definitely helped with family involvement today. (Nurse, HDUHB)

Mum appears happy to support and possible not very involved until now. (SLT, ABUHB)

Domain summary 2: the personal touches

The narrative data highlighted several incidences of where VC has been able to increase clinician to patient relations.

Communication, personalisation and rapport

Patients commonly expressed how VC helps them to communicate effectively, to receive a more personalised and patient-centred approach and build rapport with their clinicians.

Having a video call made it more personal for me the support given to me was excellent. (Parent of patient, CAVUHB, under 12 years, Paediatrics and child health, Advice)

We have built up a relationship with our clinician via VC. (Patient, Male, 45–64, CAVUHB, Counsellor, Mental health)

Patient positivity and appreciation

A strong consensus of patient ‘positivity’ and ‘appreciation’ towards their clinicians is expressed widely in the narrative. This positive clinical presence led to many patients feeling safe, comfortable and supported during and after their VC. This was particularly evident across specialties such as mental health and therapies.

Had a really tough week, but [name removed] was amazing and she listened to me. She gave me great support and was really kind to me. (Patient, CAVUHB, 25–44, Psychologist, Mental health, Therapy/treatment)

She was engaging, courteous and professional in explaining what she felt the issues were with my hand. I found the whole experience thoroughly satisfactory. (Patient, CAVUHB, 45–64, Male, Occupational therapist, Trauma and orthopaedics, First appointment)

It was great to feel that someone was there to chat to, who could see a difference in [patient name removed], while at the same time supporting us as a family. (Guardian/carer of patient, SBUHB, 25–44, Male, Mental health, Follow-up)

Domain summary 3: the benefits of VC

The benefits associated to using VC were one of the most common domains that materialised in the data.

Convenience, safety and home comforts

Many of the patients demonstrate the benefit of convenience when using VC as opposed to a traditional face-to-face appointment, with additional advantages such as improved safety and home comforts.

As we live quite far away, the virtual meetings are a lot more convenient and it’s nice to feel more comfortable at home. Thank you. (Patient, CAVUHB, Female, 25–44, Obstetrics and gynaecology, Follow-up)

I felt really comfortable talking to [name removed]. I was able to get things off my chest, and talk about the assault more deeply than I have ever done. (Patient, ABUHB, 45–64, Female, Counsellor, Mental health, Therapy/treatment)

Client is pregnant and so is vulnerable to the COVID virus. AA means she can continue with therapy without the additional risks. (Mental health, HDUHB)

Flexibility of VC

For many patients, a benefit of using VC was the flexibility it allowed. For example, patients reported to be able to continue ‘getting on with other things’ while waiting in the ‘virtual waiting room’, which would not have been possible in a physical location.

I felt it was good as I could start the call and then get on with things around the house while I waited. (Parent of patient, BCUHB, Female, under 12 years, Physiotherapist, Paediatrics and child health, Therapy/treatment)

I think that it is excellent to have a consultation this way. It was easy to log on and saves so much time for both of us. (Patient, ABUHB, 45–64, Female, Physiotherapist)

In addition, having three modes of appointment (VC, telephone or face to face) provided the patient with a stronger sense of patient choice and flexibility. However, it was felt that VC, as opposed to a telephone consultation, allowed comparable aims and goals to be achieved similar to a face to face.

VC let us achieve patient’s 1st choice, which could not have been achieved over the phone. (Occupational therapist, SBUHB)

AA is a way of bridging direct face-to-face and a visual interaction can be helpful as part of the clinical assessment. (Nurse, HDUHB)

Having a video consultation is so much better than just a telephone call—it allows you to chat as if it was in person. (Patient, PTHB, 64–80, Female, Nurse, Respiratory medicine, Advice)

The ability to be able to visually ‘see’ the patient is considered imperative to clinicians, as for many healthcare conditions VC is needed to enable visual cues.

Better than just telephone call as could get non-verbal clues about emotions. (Doctor, ABUHB)

Really useful being able to see patient via system—really added to consultation, infinitely superior to telephone consultation. (Doctor, SBUHB)

Time savings

When using VC as opposed to face-to-face appointments many clinicians and patients expressed that they had saved time in several ways, and this was a considerable benefit to patients, families, clinicians and the NHS service as a whole.

For example, clinicians felt that the ‘time’ used to conduct a VC was reduced in comparison to the usual components of a face to face, for example, logistics. The time saved from travelling to and from appointments was able to be combined into the overall virtual consultation in some cases, ultimately benefiting clinicians’ availability to attend to other patient needs and clinical tasks.

Video consultation reduces time required the next day. (Midwife, CVUHB)

Video consultation prior, ensures that less time on home visits. (Health visitor, SBUHB)

In addition, the use of VC lowered the ‘wait times’ in some instances for patients, in comparison to waiting for a face-to-face appointment.

Fantastic way to be able to have an appointment without having to wait months. (Parent of patient, CTMUHB, under 12 years, Male, Doctor, Otolaryngology (ENT), First appointment)

Reduces time required for next appointment. (Audiovestibular medicine, BCUHB)

From the data, ‘travel time’ or ‘time off’ work or school was perceived as the biggest saving of time in comparison to attending a face-to-face appointment for patients.

Less travel and disruption of [patient] school day. (Dietician, ABUHB)

I just had just finished a night shift, and live a fair distance from work, so doing a video call made my life a lot easier. (Patient, SBUHB, 45–64)

Not having to travel to the hospital and waiting in the waiting room was much better, and there was no stress trying to get around everything all of the time. (Patient, CVUHB, 45–64)

Clinical value

Many clinicians suggested that VC has the ability to enhance a clinical session or determine clinical need. For example, some clinicians demonstrated opportunities to share visual resources immediately within the appointment. Others reported that VC allowed for an effective triaging tool to determine the ‘need’ for a face-to-face appointment as opposed to a remote consultation alternative.

It enhanced the clinical session because it added visual opportunity. (Speech and language therapy, CVUHB)

I could open investigations on screen easily, I shared internet resources links to patient, and she got them on screen straight away. (Doctor, BCUHB)

Domain summary 4: the challenges of VC

While there are a number of evident benefits when considering VC, it is important to highlight the challenges faced to gain an overall picture of both patient and clinician experiences which are subject to subtle nuances.

VC is not for everyone or everything

There are some clinical situations and personal circumstances which continue to necessitate the need for face-to-face consultations, where VC does not achieve the outcomes necessary, or suit the clinical condition or patient type.

Still needs face-to-face as cannot test hearing over VC. (Audiovestibular medicine, BCUHB)

Only thing missing was ability to weigh and get height. (Dietician, ABUHB)

Patient and clinician digital ability

Some issues with patient and clinician user abilities were also highlighted in the data, clinicians made note that on occasion patients struggled to undertake VC due to their lack of technological ability. This affected the potential quality of the VC and therefore impacted the patients’ opportunity to receive care via VC.

Client unable to get full screen. Client not familiar with using equipment at home. (Counsellor, Psychiatry and Mental Health, Mental health, Secondary, SBUHB)

Patient didn’t receive link so unable to do. I think it may be due to me being unfamiliar with new system in the end and it worked well. (Physiotherapist, Paediatrics and Child Health, Therapies (AHP), CVUHB)

Engagement over VC

Engagement was a further challenge that clinicians experienced when using VC. Within therapies for younger patients where parents were present, clinicians found it challenging to engage with children via video in the same way as face to face. Children were reported to be more distracted during these appointments as the concept of video was relatively new, therefore parents and clinicians had to attempt to engage with the child more than via face to face.

Poor picture quality does not encourage children who already have attention/listening difficulties to take part. (Speech and language therapist, ABUHB)

Child had short attention span and parents had to work hard to keep him in front of camera. (Speech and language therapist, ABUHB)

Sound quality not adequate at times to determine success of target so reliant on adult feedback. Child however responding better than expected although once attention levels drop it is quite difficult to return to tasks. (Speech and language therapist, Paediatrics and child health, ABUHB)

Waiting room issues

A problem that was reported by a small number of patients was associated to the virtual waiting time being exceedingly long, or that their appointment was missed by a clinician, or in some instances, where no clinician attended at all.

Waited for over 40 minutes in the waiting room. (Parent of patient, SBUHB, and parent of child under 12 years, Paediatrics and child health, Follow-up)

We waited in the virtual waiting room for an hour and 30 minutes without anyone answering. We couldn’t wait any longer and due to our poor internet connection in our area, I had to use all my monthly data. (Parent of patient, ABUHB, Parent of child under 12 years, Male, Paediatrics and child health, Follow-up)

Domain summary 5: technical quality

When considering the technical aspects of VC, clinicians and patients tended to rate their VC highly when the audio and visual picture were of good quality.

Good versus bad quality

For high-quality ratings, these were paired with praise for VC in the free text narrative box, suggesting that the audio and visual elements of the VC were of high quality.

Great connection. No glitches very smooth call. (Parent of patient, ABUHB, under 12 years, Female, Physiotherapist, First appointment)

The video and picture was perfect, was able to hear the doctor clearly. (Patient, CAVUHB, 45–64, Female, Doctor, Follow-up)

However, there were technological challenges reported within the narrative such as poor connectivity, thus impacting on visuals and audio.

Lag in audio/video sometimes causes miscommunication or difficulty with younger patients. (SLT, CVUHB)

The video was very choppy and when my therapist was talking it was delayed video with speech. (Patient, CAVUHB, Female, 25–44, Counsellor, Mental health, Therapy/treatment)

In some instances, technical issues were associated with specific device types and their perceived incompatibilities with the VC platform.

The video call app did not give me the option of using my inner camera so I had to turn my phone around so the doctor could see me. (Patient, BCUHB, 25–44, Ophthalmology, Advice)

Problems with Safari on iPad. (Health visitor, SBUHB)

Session being done on Father’s phone so screen obviously small for child to watch. (SLT, ABUHB)

Clinical innovation and troubleshooting

Yet, despite these technological challenges, with the right amount of technical support and appropriate equipment available, clinicians report to be able to troubleshoot many issues and continue to use VC in most situations.

Tried to do call with mobile phone and there were issues for the patient not being able to grant access for use on mobile phone, but the consultation worked perfectly on their computer. (Dentist/dental nurse, BCUHB)

Issues at the start of the call with the audio but we disconnected and reconnected and it was then fine. (Nurse, SBUHB)

Some clinicians were able to troubleshoot the problems easily to make the consultation work best for them and their patients.

I was unable to connect through the desktop in clinic due to computer being extremely slow…. I was luckily able to connect through my Netbook which supports the platform. (SLT, CVUHB)

School initially struggling with internet connection but then able to move to a room with better signal and VC quality. (SLT, CVUHB)

Domain summary 6: recommendations and future use

When considering the experiences of both clinicians and patients using VC, it is important to consider how the narrative can be built on to consider suggestions and recommendations to ensure that VC is suitable for future use and in conjunction, blended with face-to-face and telephone consultations.

Clinical recommendations

One of these suggestions was improved infrastructure and resources for clinical and administrative staff to have access to. It was felt that by having better equipment, they would deliver better patient care via VC. Not only this, but in some areas the number of devices and access to workspace was limited and needed significant improvement in the future.

It would be useful to have 2 microphones so I can share videos with my clients about EMDR therapy and PTSD. (Nurse, BCUHB)

Need appropriate screens and two monitors to view downloads and see patients, desk and chairs at right height. (Dietician, BCUHB)

Clinicians also suggested that there needed to be an improvement with the technical support that was on offer across health boards regarding VC.

Being taught how to split screen so we can write notes at the same time, rather than making paper notes and writing up after. (Occupational therapist, BCUHB)

This suggestion of technology support would ensure clinicians could use VC to the best of their ability, using all aspects of the platform. Some clinicians suggested new features that they felt would be useful in ensuring clinician/patient confidentiality and safety.

To be able to blur/add a background when working from home. (Dietician, SBUHB)

I would like to be able to leave the call screen but still be able to see patient in a little pop-out screen. (SLT, BCUHB)

Additional administrative support was also suggested for VC so that they could mirror the way standard face-to-face consultations were run.

For this to work administrative clinic support needed to mirror that provided for face-to-face appointments. (Nurse, CVUHB)

Patient wants and needs

Patients’ narrative also suggested that technical and digital skills support would be useful in the future use of VC. Some patients were slightly unsure of how to use the technology needed for VC and ran into some issues. By having support for this, it may lead to an increase in digital skills for future digital implementations and the move towards a new NHS digital strategy.

I couldn’t work out how to use the camera on the front of my and wasn't sure how to connect via my computer to the appointment. (Patient, ABUHB, 25–44, Female, Mental health, First appointment)

I was unable to switch my camera to front facing, so not able to see who I was talking to. (Patient, CAVUHB, Female, 45–64, Nurse, Otolaryngology (ENT), First appointment)

Patients provided narrative to suggest a blended approach of digital healthcare services was needed going forward. This was due to a large number of patients highlighting that VC provided numerous benefits, and help to supplement the quality of care received from clinicians, and believed a blended approach of VC and face to face was the way forward for the future of the NHS Wales support by clinicians.

Definitely the way forward for consultations, I live 100 miles away so for the purpose of consultation rather than treatment this is brilliant! (Patient, SBUHB, 45–64, Female, Doctor, Plastic surgery)

I think this will be the future. I felt more relaxed being able to do it from my home. (Patient, BCUHB, Female, 64–80, Doctor, Obstetrics and gynaecology, First appointment)

Video consultations act as a useful complement to face-to-face sessions and home visits. (Audiologist, BCUHB)

Discussion

The coding reliability analysis of the free text narrative boxes captured at the end of VC provided feedback from a large data set of 22 978 clinician and patient submissions expressing a vast and overall view of VC experiences in Wales. Six dominant domain summaries and 17 additional subdomains materialised. Due to the high response rate in free text narrative box responses, the analysis of the feedback data was able to be conducted using a coding reliability approach, thus providing context for each domain and its perspective, supported by patient and clinician quotation. The domains that materialised in the analysis provide a strong sense that the NHS Wales VC Service on a whole is highly satisfactory, well accepted and clinically suitable for a wide range of patient and clinical teams using the service. Despite this, it is important to draw attention to the challenges that have also occurred for both clinicians and patients, such as VC not always being suitable for every individual or appointment.

The data provide a strong consensus that the VC platform currently being used in NHS Wales is ‘easy to use’ in both technical and experience terms, with the additional value of its ability of enhanced collaboration, thus providing a multidisciplinary approach to patient care. In addition, the data highlight the real life and personal aspects of VC experience, which suggests that patients who are using the VC service are satisfied with using it and provide narrative around its ease of use and personalisation felt in their patient care.

In addition, there is a heartfelt sense of patient appreciation and gratitude to their clinicians for their hard work and dedication to delivering patient care. Furthermore, the data demonstrate the benefits that are associated to using VC. These benefits are felt by patients, families and clinicians, and the NHS service. Challenges are also apparent within the data with VC not always being appropriate for all patients or appointments. This is in combination with difficulties surrounding engagement, particularly with children via VC and issues with digital ability across clinician and patient populations. Although there were disparities of digital ability that sometimes hindered a VC appointment, and suggestive of a digital divide, in this data set and the wider evaluation²¹ we did not find this to be the case. The quantitative findings that run alongside these data provide additional support, specifically regarding patient representation, concluding that regardless of patient age, gender, ethnicity, household income, health condition, disability or place (urban vs rural), VC can provide a high standard of healthcare delivery across Wales.^{18 21 22} Though apparent, the challenges were heavily outweighed by the number of benefits experienced from using VC.

The data also present a comparison between good and bad technical quality on the platform regarding audio and visuals for both patients and clinicians. Improvements for future use should encapsulate recommendations such as more resources to be made available to clinical teams, and that VC platform features are considered as priority for improvement. It is also noted that increased technical support and education is provided to ensure that VC can appropriately be used in the future, and possibly offered directly to patients, so that VC is used as a long-term blended approach to suit patient choice and preferences moving forward. While challenges have been identified, the data captured in this study are comparable to previous literature that suggests the benefits of VC outweigh these challenges^{18 22} and can support the use and sustainability of VC in NHS healthcare services. As discussed within the Results section, VC is not seen to be used for everything within healthcare, despite the benefits highlighted within this study. The need to ensure that VC is offered within every healthcare environment is pertinent to its sustainable future use along with shared decisions between clinicians and patients.²³

Limitations and conclusions

There are many strengths to this study, including its narrative approach among a very large and representative sample for Wales. However, the study did have some limitations. Due to the voluntary and anonymised nature of the feedback data collection, it is unclear as to how often recurring participants completed the feedback, thus potentially skewing the sample. Furthermore, due to the size of the study, it was not possible to present both the qualitative and quantitative findings together; however, access to these data is readily available.²²

Originally, the data were broken down into specialties, but were regrouped for the purpose of this narrative analysis as most of the data showed little difference between specialties. While the quotations used within these data are true of the narrative at the time of collection, it is important to note that these are fitting of a time during the pandemic and so reflect this period. Data are being captured in an ongoing evaluation of VC within Wales, and suggest that those using VC are still rating it positively.

While this paper does not cover specific clinical implications, VC is being used across various specialties and evaluation is ongoing to explore this further.

Data availability statement

Data are available upon reasonable request. All analysed data are published on the TEC Cymru website in the format of a full report of all data for the public to view. To access these reports please see https://digitalhealth.wales/tec-cymru. Other data can be requested as a reasonable request to the corresponding author.

Ethics statements

Patient consent for publication

Obtained.

Ethics approval

This study involves human participants and TEC Cymru obtained full ethical approvals and risk assessments from their host Aneurin Bevan University Health Board Research and Development Department (reference number: SA/1114/20), and then national approval was obtained from all other health boards in Wales. Full consent was obtained from all participants. At the end of each feedback link, a statement of consent and a compulsory tickbox was required prior to feedback submission. Participants gave informed consent to participate in the study before taking part.

Footnote

Twitter @teccymru

Contributors GJ contributed to the main design of the study and development of the research questions, the main structure and write-up of the paper, and final amendments to the manuscript. GJ, BW, BT and MW analysed the data with the supervision of AA, SK and MO. All authors discussed and interpreted the data once analysed and helped structure the manuscript. AA, SK and MO contributed to the clinical understanding of the findings and shaped the discussion, conclusions and recommendations. AA was responsible for overseeing the full development of the study design and data collection, the analysis and development and final sign-off of manuscript from a clinical and programme perspective. All authors contributed to proofreading and amendments of the final manuscript. GJ is the guarantor for this article.

Funding Technology Enabled Care (TEC) Cymru and its NHS Wales Video Consulting Service is funded by the Welsh Government.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Connor MJ, Winkler M, Miah S. COVID-19 pandemic - is virtual urology clinic the answer to keeping the cancer pathway moving? BJU Int 2020; 125: E3–4. doi:10.1111/bju.15061 http://www.ncbi.nlm.nih.gov/pubmed/32232915

² Ramalho R, Adiukwu F, Gashi Bytyçi D, et al. Telepsychiatry during the COVID-19 pandemic: development of a protocol for telemental health care. Front Psychiatry 2020; 11: 552450. doi:10.3389/fpsyt.2020.552450 http://www.ncbi.nlm.nih.gov/pubmed/33173507

³ Leng S, MacDougall M, McKinstry B. The acceptability to patients of video-consulting in general practice: semi-structured interviews in three diverse general practices. J Innov Health Inform 2016; 23: 493–500. doi:10.14236/jhi.v23i2.141 http://www.ncbi.nlm.nih.gov/pubmed/27869579

⁴ Whaibeh E, Mahmoud H, Naal H. Telemental health in the context of a pandemic: the COVID-19 experience. Curr Treat Options Psychiatry 2020; 7: 198–202. doi:10.1007/s40501-020-00210-2 http://www.ncbi.nlm.nih.gov/pubmed/32292687

⁵ Greenhalgh T, Wherton J, Shaw S, et al. Video consultations for covid-19. BMJ 2020; 368: m998. doi:10.1136/bmj.m998 http://www.ncbi.nlm.nih.gov/pubmed/32165352

⁶ John G, Khalil S, Ogonovsky M. Phase 1 report. Chapter 1 live data - patients & clinicians. The NHS Wales video consulting service, TEC Cymru, 2020. Available: https://digitalhealth.wales/tec-cymru/howwe-can-help/evidence/eval-reports

⁷ Donaghy E, Atherton H, Hammersley V, et al. Acceptability, benefits, and challenges of video consulting: a qualitative study in primary care. Br J Gen Pract 2019; 69: e586–94. doi:10.3399/bjgp19X704141 http://www.ncbi.nlm.nih.gov/pubmed/31160368

⁸ Wade VA, Karnon J, Elshaug AG, et al. A systematic review of economic analyses of telehealth services using real time video communication. BMC Health Serv Res 2010; 10: 1–3. doi:10.1186/1472-6963-10-233

⁹ Smith AC, Thomas E, Snoswell CL, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–13. doi:10.1177/1357633X20916567 http://www.ncbi.nlm.nih.gov/pubmed/32196391

¹⁰ Hollander JE, Carr BG. Virtually perfect? Telemedicine for Covid-19. N Engl J Med Overseas Ed 2020; 382: 1679–81. doi:10.1056/NEJMp2003539

¹¹ Reinhardt I, Gouzoulis-Mayfrank E, Zielasek J. Use of telepsychiatry in emergency and crisis intervention: current evidence. Curr Psychiatry Rep 2019; 21: 1–8. doi:10.1007/s11920-019-1054-8

¹² CWTCH Cymru toolkit: step by step guide to using video consulting in telepsychiatry, 2020. Available: https://www.rcpsych.ac.uk/docs/default-source/members/divisions/wales/cwtch-ready-set-gotoolkit.pdf

¹³ Barsom EZ, van Hees E, Bemelman WA, et al. Measuring patient satisfaction with video consultation: a systematic review of assessment tools and their measurement properties. Int J Technol Assess Health Care 2020; 36: 356–62. doi:10.1017/S0266462320000367 http://www.ncbi.nlm.nih.gov/pubmed/32624044

¹⁴ Gallo G, Grossi U, Sturiale A, et al. E-consensus on telemedicine in proctology: a RAND/UCLA-modified study. Surgery 2021; 170: 405–11. doi:10.1016/j.surg.2021.01.049 http://www.ncbi.nlm.nih.gov/pubmed/33766426

¹⁵ Wherton J, Greenhalgh T. Evaluation of the attend anywhere/near me video consulting service in Scotland, 2019-20. Scottish Government, 2020.

¹⁶ Willman AS. A service user evaluation of eConsult use by defence primary healthcare primary care clinicians using a mixed-method approach. medRxiv 2020.

¹⁷ The NHS Wales video consulting service, 2020. Available: https://digitalhealth.wales/tec-cymru/vcservice

¹⁸ Technology enabled care (TEC) Cymru. Available: https://digitalhealth.wales/tec-cymru

¹⁹ Phase 2A qualitative data. The NHS Wales video consulting service, technology enabled care (Tec) Cymru, 2021. Available: https://digitalhealth.wales/

²⁰ O’Connor C, Joffe H. Intercoder reliability in qualitative research: debates and practical guidelines. Int J Qualit Method 2020; 19: 1609406919899220.

²¹ Johns G, Khalil S, Ogonovsky M, et al. Access to the digital NHS is not much of a problem in Wales. BMJ 2021; 374: n2212. doi:10.1136/bmj.n2212 http://www.ncbi.nlm.nih.gov/pubmed/34518195

²² Phase 2A quantitative data. The NHS Wales video consulting service, technology enabled care (Tec) Cymru, 2021. Available: https://digitalhealth.wales/

²³ Barsom EZ, Jansen M, Tanis PJ, et al. Video consultation during follow up care: effect on quality of care and patient- and provider attitude in patients with colorectal cancer. Surg Endosc 2021; 35: 1278–87. doi:10.1007/s00464-020-07499-3 http://www.ncbi.nlm.nih.gov/pubmed/32198552