Content area
Background
Online NHS111 was introduced in 2018 in response to increasing and unsustainable demand for telephone NHS111. Despite high levels of use, there is little evidence of channel shift from the telephone to the online service. We explored user and staff perspectives of online NHS111 to understand how and why online NHS111 is used and whether there may be potential for shift from the telephone to online service.
Methods
As part of a wider mixed-methods study, we used qualitative semistructured interviews to explore perspectives of recent users of online 111 who had responded to a user survey (n=32) and NHS 111 staff (n=16) between November 2019 and June 2020. Interviews were recorded and transcribed verbatim. The data sets were analysed separately using framework analysis (user interviews) and thematic analysis (staff interviews).
Results
Telephone NHS111 health adviser skills in probing and obtaining ‘soft information’ were perceived as key to obtaining advice that was considered more appropriate and trusted than advice from online interactions, which relied on oversimplified or irrelevant questions.
Online NHS111 was perceived to provide a useful and convenient adjunct to the telephone service and widened access to NHS111 services for some subgroups of users who would not otherwise access the telephone service (eg, communication barriers, social anxiety) or were concerned about ‘bothering’ a health professional. The nature of the online consultation meant that online NHS111 was perceived as more disposable and used more speculatively.
Conclusion
Online 111 was perceived as a useful adjunct but not a replacement for telephone NHS 111 with potential for channel shift hindered by reduced confidence in the online service due to the lack of human interaction. Further development of OL111 algorithms will be required if it is to meet the needs of people with more complex health needs.
Correspondence to Fiona C Sampson, ScHARR, The University of Sheffield, Sheffield, UK; [email protected]
WHAT IS ALREADY KNOWN ON THIS TOPIC
Online NHS111 was introduced in the UK in 2018 to reduce unsustainable demand on the NHS111 telephone service.
Quantitative routine data analysis showed that the introduction of online NHS111 had limited impact on the demand for the NHS111 telephone service but does not explain how or why there has been minimal ‘channel shift’ of demand from the NHS111 telephone service.
WHAT THIS STUDY ADDS
In this qualitative interview study of both patients and staff, we found that users trusted and followed advice from telephone NHS111 more than online NHS111 due to the human interactions involved in answering questions appropriately.
Online NHS111 was used more speculatively and advice potentially seen as more ‘disposable’ due to the lack of contact with health professionals.
However, the introduction of online NHS111 improved overall access to NHS111 services for a subset of users.
HOW THIS STUDY MIGHT AFFECT RESEARCH, POLICY OR PRACTICE
Online triage has limited potential for shifting demand due to ambiguity in algorithm question wording.
Refinement of questioning will be required for online NHS 111 to increase in value and use for people with multimorbidity and long-term conditions.
A hybrid option whereby online users can clarify question meaning using live chat options could be considered in order to improve the usefulness of online NHS111.
Introduction
In the UK, emergency and urgent care (UEC) is provided by a range of services including emergency services (999 ambulance service, EDs), urgent care services (out-of-hours primary care, minor injury units, walk-in centres, urgent care treatment centres, NHS 111) and in-hours primary care (requests for same-day appointments and telephone advice). There is widespread concern about rising and unsustainable demand for UEC services, which is not wholly explained by population increases or an ageing population.1–3
Over the past two decades, numerous organisational interventions including NHS Direct and its successor, NHS 111, were implemented in England to try and address the rise in demand and ensure that people with low urgency health problems can be directed to a service that best meets their clinical need. The NHS 111 telephone service (Tel111) was set up to provide triage and assessment and to direct people with low acuity non-emergency health problems to the most appropriate UEC service, or provide self-care advice. Health advisers (usually non-clinical) ask the caller a series of questions, following different clinical algorithms and then provide a clinical ‘disposition’ or recommendation about what service to contact, and the time period for doing so. For some callers, a transfer to, or call back from, an NHS clinician will be offered. However, demand for the Tel111 service has risen significantly, with nearly 15 million calls in 2017/2018,4 resulting in the service struggling to meet demand and an increase in waiting times for urgent calls. However, there was little impact on demand for emergency services.5 6
The NHS 5-year forward view policy identified the scope for online services to relieve some of the pressures on telephone services.7 In Australia, a 33% reduction in call volume was observed after 3 years of an online service implementation. In 2018, in an attempt to stem increasing demand for Tel111, NHS England piloted and then rolled out nationally a digital alternative to the NHS111 to incorporate online symptom checkers and health advice.7–9 OL111 uses the same clinical algorithms as Tel111 and provides similar dispositions but requires users to direct themselves through the initial questions using a computer, tablet or smartphone.
At the time of rollout, little was known about how acceptable the online service would be to users, and whether there was potential for OL111 to offset or ‘channel shift’ demand for Tel111 (ie, patients moving from use of Tel111 to OL111). There is some evidence that the user profile of digital and online symptom checkers and health assessment services differs from other platforms (online users being more likely to be young, employed and female),10–12 suggesting that OL111 may reach a different audience and induce demand but not channel shift.13 The process and outcomes of the OL111 service may lead to different user experience, with evidence suggesting other algorithm-based triage systems to be more risk-averse than those involving healthcare professionals.13 14
The work presented in this paper is part of a wider mixed-methods evaluation of the impact of the introduction of online NHS 111 on the NHS 111 telephone service.15 The wider study identified no significant impact of the introduction of OL111 on the number of triaged calls to Tel111. The study reported a survey of OL111 and Tel111 users which found that Tel111 users had higher levels of satisfaction with the service than OL111 users across all comparable measures, yet over half of users said they would use the online service again. We present here more in-depth findings from the qualitative components of the study in order to explore stakeholder perspectives of how and why OL111 is used and understand whether there may be potential for channel shift from Tel111 to OL111.
Methods
Study design
The wider study used a mixed-methods approach to understand the acceptability and usability of OL NHS 111, explore potential for ‘channel shift’ and other impact on telephone service. Due to short overall project deadlines, we used a convergent parallel design with quantitative and qualitative elements undertaken concurrently, different components analysed independently and then results interpreted together.16 Within this paper, we present integrated findings from the qualitative components which were undertaken concurrently and analysed separately initially: staff interviews and OL111 user interviews. We present the converged results from the separate analyses described below.
Participants
As this was part of a wider evaluation of online NHS 111, we recruited users of the new online service as well as NHS 111 staff who were working at the telephone service at the time the online service was set up.
Staff interviews with NHS111 staff and commissioners
Within the wider study, we selected nine case study sites to cover a range of early implementer sites, geographical areas and service provision models. Five of these sites were invited to take part in interviews and four sites responded positively (two ambulance services, one private provider and one social enterprise). Recruitment took place between November 2019 and June 2020. Information about the study was circulated to relevant staff via key contacts at each site, and participants were asked to contact the research team directly if they wished to take part. We aimed to purposively sample staff with a variety of roles and length of experience. After a low initial response rate, we amended the process to offer participants a shopping voucher and asked sites to recirculate the invitation. Recruitment continued to be low due to significant pressures on the service and was then halted due to the COVID-19 pandemic.
Interview topic guides were developed initially by the wider project management group along with patient and public involvement (PPI) representatives. These were then revised following discussions with site leads as part of a data gathering exercise that was undertaken to understand the models of service delivery. Topic guides are included within the online supplemental material.
Interviews were conducted by telephone by one member of staff (FCS) and digitally recorded. Interviews lasted an average of 33 min (range 15–62) and were transcribed verbatim. Data were loaded into NVivo software and analysed using thematic analysis according to the principles of Braun and Clarke.17 FCS read and checked the transcripts for familiarisation. The coding framework was developed by FCS in consultation with JL and ELK, who also read a sample of transcripts. Initial themes were developed around a priori themes (ie, those within the topic guide) including resource implications, perceptions of use of NHS 111 online and role of the service within UEC system. Further interpretative themes were developed around trustworthiness and belief, knowledge and awareness, and integration.
Online user interviews
We recruited OL111 users from a previous national survey of OL111 users.15 After completing a follow-up NHS Digital survey, participants could view a link inviting them to take part in an interview and provide their contact details. A researcher then contacted individuals by telephone and arranged a time and date for the interview as soon after contact as possible to ensure good recall of events. We planned to undertake up to 40 interviews. We initially took a non-selective approach to sampling, which resulted in a largely female, middle aged, ethnically white sample, who had received a call-back from NHS 111 after their online assessment. After completing 20 interviews, we purposively recruited only younger service users and those who had not received a call-back.
Semistructured interviews took place by telephone between December 2019 and April 2020. Informed consent was recorded at the start of each interview. The topic guide covered general use of online services and health services, previous use of NHS 111 (telephone and online), experience of using OL111, any preferences between the online service and telephone service, and the likelihood of using the online service in the future. Topic guides were developed following discussion within the project management group and based on research objectives and designed to elaborate on existing survey responses. One researcher (JL) undertook all of the interviews, which lasted between 24 and 55 min. Participants were sent a shopping voucher on completion of the interview.
Interviews were audio recorded and transcribed verbatim. Transcripts were analysed in NVivo following the first steps of framework analysis, according to Ritchie and Spencer.18 All transcripts were read and checked by one author (JL) during the familiarisation phase, with a subset of transcripts also read by two other authors (EK, FS). The framework was developed from our research questions, the topic guide and an initial reading of a sample of transcripts. Themes included positive and negative aspects of the experience of OL111, compliance with the advice given, propensity to use OL111 again, and recommendations for future development of the service. Once the initial framework was agreed, the researcher who had undertaken the interviews (JL) coded all transcripts to the framework, adding further emergent themes and subthemes and discussing any areas of uncertainty with a second researcher. JL and EK then reviewed extracts within each theme to refine the final thematic framework.
Patient and public involvement
We used an established PPI user group to support the study, including one member who was a co-applicant on the research grant and attended project management group meetings throughout. The group advised on documents related to ethical approval, and inputted into discussion of emerging themes within the analysis and discussions of early drafts of the results.
Results
Characteristics of respondents
Interviews were conducted with a total of 32 OL111 users and 16 NHS 111 staff. OL111 users were predominantly female (n=22), white British (n=28) and from younger age brackets (<35: 7, 35–65: 9, >65: 6). Most participants (n=26) reported this to be the first time they had used OL111 and the majority (n=28) reported having used Tel111 in the past.
For NHS 111 staff interviews, we interviewed commissioners (n=2), head of integrated and urgent care (n=1), clinical team manager (n=1), non-clinical team leaders (n=2), clinical advisers (n=6) and health advisers (n=4).
Findings
We identified four key themes from data analysis of OL111 users and staff that helped to identify how the different 111 services were used, and the potential for channel shift.
Higher satisfaction for telephone 111 users may be due to improved confidence and trust in the dispositions reached due to interaction with a health adviser
Both staff and users highlighted problems with ambiguous and oversimplified question wording within OL111. The lack of flexibility in how questions could be answered in the online system appeared to contribute to the lower confidence in the resulting advice. This was contrasted with advice given during the human interaction of the telephone call, which was perceived as key to extracting the most appropriate and relevant information from the user. Almost half of the OL111 users interviewed felt that the questions were poorly worded, overgeneralised or not relevant, which made them question the professionalism of the service and reduced their confidence in the advice given.
I found it was very basic and for me that meant that I would not have been completely reassured with any conclusive advice it would have given because it seemed very generic, you know? And it’s like those horoscopes that come through the email. It just didn’t seem professional enough for me. (OL111 user 20)
Well, I always like to talk. Well, it’s always better cos you can, you get a rapport with someone and usually it’s relevant questions … normally you can tell almost immediately whether they’re trained well enough to do the job or not… I have confidence in people… I suppose it’s because I prefer dealing with people than machines. (OL111 user 22)
Yes. They’ve put their, what they perceive to be their symptoms, their issues, into the online service, and then whatever the online service is then telling them to do, they’re phoning in to the service well it says I need to this, or it says I need to do that. So then they want, in my perception is, they then want the human contact. They want somebody to say yes that’s right, that’s what you need to be doing. Or well I’ll assess it and I’ll do this for you. (Tel111 staff 07)
Concerns about inappropriate and difficult to understand terminology were reflected by staff who reported spending more time with OL111 referrals due to the need to re-triage users who had misunderstood questions. The ability of health advisers to probe and obtain ‘softer’ information that would guide users down the most appropriate algorithm was seen as key to obtaining trustworthy outcomes by both users and staff.
When we speak to someone I can, as a nurse, I can question that. How bad is your chest pain? ‘Oh it’s really bad’. But I say ‘I’m talking to you and I cannot feel that you’re struggling for a pain that is very, very bad. Your voice is, you know, fluent, you’re not struggling, you’re not gasping for air. So pain that’s really, really bad will not make you sound so well. So are you sure that you have understood this question?’ (Tel111 staff O6)
It [online assessment] drove you down one straight route, you couldn’t necessarily pose questions, whereas with something like chat over the internet you can actually ask questions. (OL111 user 24)
It [the questions] was very black and white. There was nothing in the middle, it was very general-, it was almost like can you breathe or can’t you breathe. There was nothing in the middle. (…) specific enough, there weren’t enough options. (OL111 user 2)
In particular, Tel111 was perceived to be preferable for more complex or urgent problems when the human interaction could help direct the user to more appropriate advice, or intervene where necessary. The importance of human interaction in understanding the complexity of healthcare problems was again highlighted as key, particularly for users with chronic conditions or multimorbidity, as the online questioning did not enable consideration of underlying problems that may impact on the problem being consulted about.
They [the online question] said ‘is this a new cough’ you know, or is it the old cough come back, you know just a bland question ‘is this a new cough?’ just doesn’t hack it, you didn't know how to answer it. (OL111 user 25)
It was a bit limited with regards to COPD. The questions didn’t go anywhere and I suspect probably there was not enough depth for it to have been terribly useful. I was feeling I knew more than the 111. (OL111 user 24)
I had a sore throat, but I had no option to say that I’m asthmatic as well, which was the reason why I am worried. (OL111 user 20)
NHS 111 online may provide a useful and convenient adjunct to the telephone service, rather than a replacement
Participants described OL111 as their preference for information or advice relating to what they perceived to be simpler, low acuity health problems such as stomach bugs and colds, or identifying the location of an emergency dentist, particularly when their primary care services were unavailable. Although the participants reported the advice was less trusted, the speed and convenience of the service made it a preferred option for people with time pressures (eg, those with young children). Both the lack of waiting time and the speed of the actual process of doing the assessment, particularly at a time when waiting times for Tel111 precluded quick access to a telephone adviser, were seen as advantages. Some users reported using both services simultaneously or doing an initial screening with OL111 before then contacting Tel111.
Let’s say I’m looking after my son or something else is going on in my life I get a phone call or something, I can stop [NHS 111 Online] at any time I want, I can restart it at any time I want so I like that kind of convenience. (OL111 user 31)
I looked up online while calling 111 *laughs* and then I hung up, I said I’d do it online. I think it’s easier and faster. (OL111 user 13)
…it’s it in some ways it’s better than the phone service, because you don’t have that waiting in a queue and if you’ve got a child and you’re worried about like I would be worried like my daughter with the swelling and her eye was closing up, I didn’t want to be sitting there for maybe half an hour or one in a queue looking an thinking, well is; should I be really be ringing for an ambulance you know is this gonna get worse. Erm so in that instance it was really good that I could just go on immediately, I was not in a queue, and I could just get advice, answer the questions and get advice, tailored to my answers really. (OL111 user 10)
NHS 111 online widens access to NHS 111 services for some subgroups of users who would not access the telephone service
While the lack of personal contact was felt to negatively influence the quality of advice, there was a subset of individuals who would not normally use the telephone service who reported they were happy to use OL111. This included people who did not like or were unable to use the telephone due to verbal communication difficulties or confidence (eg, hearing or language), or lacked a private space in which to talk. The impersonal nature of the interaction also suited people who wanted to avoid human interaction due to social anxiety or due to the sensitive nature of the problem being consulted about (eg, genito-urinary, sexual or mental health problems).
I usually choose the online just cause it’s slightly easier because like I do have a bit of like anxiety issues, and like while I can talk on the phone and stuff, like a lot of the time it’s easier just to go online. (OL111 user 18)
If I don’t understand something then you know you never know how people explain things. Sometimes accents can be… like English accent … the pace someone is speaking [can be a problem] … factors like that are something that I would consider. So for me it’s only when I can have a think about or check the translation of some questions … it’s the time that I can take to do, to answer the questions. I feel, you know, on the phone, I would be stressed a bit just to answer as quickly as possible because there might be someone else waiting. So for me, the time I can do it and then having the option to go back to the questions. (OL111 user 19)
We get, we get erm, I suppose it’s surprising, for me alarming amount of people with mental health problems. Because that, in a way, it’s frightening because you don’t want them to just fill forms in, you want to be able to talk to them. But we do understand that, it might be a really good first step for seeking help. Because you don’t have to talk to somebody, you can just fill, fill those in, and then sort of cast it out and hope that someone can help you. Er, as opposed to ringing and having to, you know, say out loud how you feel. (Staff 01)
Perceptions of OL111 as a ‘computer’ rather than human-based service encouraged its use by people who were conscious of not ‘over-burdening’ the healthcare system and ‘wasting’ people’s time by contacting a health professional when they were unsure whether this was necessary. Participants explained how they accessed the online service rather than Tel111 due to concerns about using health professional’s time inappropriately.
Well yes, because I'm going to be wasting their time, well let me rephrase that. I am going to be wasting other people’s time less […]. Unfortunately I’ve got mixed issues medically so I do need information sometimes where you know I don’t know what to do so for people like me, I think it’s actually a lifeline and saves all that hassle of wasting time going to the hospital or calling somebody in the middle of the night. (OL111 user 8)
… I feel that you don’t seem to be wasting anybody’s time as much as if you were phoning up. (OL111 user 04)
I think a first port of call to save taking up people’s time unnecessarily, erm that it’s a good place to go. If only to reinforce what you’re already thinking. (OL111 user 24)
I also think it’s good for you know it puts less strain on the system obviously, than having to have staff man a phone, and so and that is, that is important like not just to the wider NHS but also to me personally Because you know when you feel like you’ve got something not that serious and you maybe don’t want to bother someone, erm that is a factor for me. (OL111 user 31)
OL111 was perceived as more ‘disposable’ and used more speculatively by users but staff felt responsible for clinical interactions with online users
The lack of human interaction and perceived lack of responsibility associated with OL111 interaction enabled people to use it more speculatively than the telephone service. Users described an implicit understanding of a hierarchy of services that they would use, with online 111 lower down the hierarchy than telephone 111. For many users, OL111 was seen as a limited version of Tel111 that offered ‘the next thing to Googling’ (OL111 user 20), providing some reassurance about decisions (due to the NHS trusted brand) about which service to use rather than definitive, trusted advice. Participants characterised the service as ‘just a computer’ and therefore limited in ability and usefulness.
This is more just generally concerned, but not too worried if I can put it that way, I might use the online service but if I had any genuine concerns, I would probably call 111 and if I had any real concern, I’d phone 999. (OL111 user 25)
I: So, what, would you use the online again do you think?
P: Erm I, I would but I wouldn’t rely on it, I’d probably look at it and then probably ring 111 telephone anyway. If I really needed proper advice. I, I, I would look at it out of interest if it’s something that’s not too bad and I might be thinking, ‘Oh what might this be?’But I’m just as likely to look on the NHS website, cos that gives me stuff as well. (OL111 user 03)
It’s not so good so much for like possibly like I suppose, telling you what might be wrong with you. But then like I was saying, well it can’t really because it’s only a computer service. (OL111 user 18)
However, while users perceived OL111 as a low-resource and low-risk service, staff reported feeling a sense of responsibility towards patients who were referred into the telephone service by OL111. They characterised online interactions as being seen as ‘disposable’ and free from personal responsibility by patients, but described their own concern and anxiety when they were unable to contact patients for a call-back. This concern was exacerbated by the lower levels of trust in the data the user had entered as part of their online assessments.
So, if people fill them in late on, in the evening, we tend to be unable to make contact with them, erm, because they, they fill them in and then go to sleep [laughs]. And I think that although it tells them they’re gonna get a call back, I think like a lot of things that are online, there’s a sort of disposable element to it. They’ll fill it in at the time, because they’re interested in it, and then forget about it. (Staff 01)
So if somebody says I’ve taken three more types of pain relief, and they’re not answering, then I’m worried because I’m thinking how much pain relief have they been taken? And then if they’re not answering again, I’m thinking am I safe to close this call or not? (Staff 06)
Discussion
Summary of findings
Our findings suggest that human interactions involved in the telephone consultation appeared key to participants demonstrating higher levels of trust in telephone than online recommendations. Previous findings that telephone users were more likely to rate the advice given as ‘very helpful’ and to be very satisfied with the NHS111 service are potentially explained by telephone health adviser’s skills in probing and obtaining ‘soft information’ leading to more appropriate and trusted advice than that resulting from online interactions. Users valued the speed and convenience of OL111 and perceived it as a useful and convenient adjunct to the telephone service. The lack of personal interaction appears to widen access to NHS 111 services for some subgroups of users who would not otherwise access the telephone service for a variety of reasons. The nature of the online consultation meant that OL111 was perceived as more ‘disposable’ and used more speculatively, which reflects findings from the larger survey of OL11 users in the wider study that suggests a higher proportion of OL111 users would not have contacted another service if they had not used 111 and reported lower compliance with advice for online users compared with telephone users.15
Our findings suggested that perceptions of OL111 and Tel111 were shaped by perceptions of one as a human and the other as a computer-based service, with higher trust in the human. Staff and users described how the human interaction involved in the telephone call enabled them to provide more appropriate and accurate responses to the questions which in turn increased their confidence in the dispositions provided. Conversely, the perception of OL as an impersonal computer service, void of human interaction, opened up access to the 111 service for a subset of users who were drawn to the lack of human interaction, either because they found this difficult, or because they did not want to burden services involving a healthcare professional.
Previous literature reviews have highlighted the lack of evaluation of user experience of digital and online symptom checkers.13 19 Existing studies evaluating online and digital systems focus principally on areas of diagnostic accuracy and patient safety,13 20 with a lack of evidence comparing them to telephone triage services. 19 21 22 Compliance with advice was reported to be around 57% in a small web-based triage system in the Netherlands,23 although slightly higher at 67% within our survey. Both are lower than the reported 88% compliance with advice reported for telephone users again reflecting lower confidence in advice or more speculative use of the online services. Lee et al 24 reported user reluctance to trust health apps, compared with health professionals.
There is some evidence that algorithm-based triage may be more risk-averse than triage involving human interaction, with the potential to recommend higher levels of care than necessary.25 Other studies have reported diagnostic accuracy of online symptom checkers to be lower than triage systems involving humans.22 26 This supports the findings within our study that suggest that patients believe that human interactions are required to obtain more pertinent information than can be obtained via digital or online symptom checkers.
Strengths and limitations of this study
This study uses qualitative research methods to explore and explain findings from a national online NHS 111 user survey, to obtain a more in-depth understanding of the complexities of service use that may explain potential for channel shift.
The majority of the data collection undertaken for this project was undertaken prior to the COVID-19 pandemic. During the pandemic, in particular the first lockdown, demand for both online and telephone 111 services soared, in part due to public health messaging asking people to contact 111 as a first point of call. During the pandemic, there were long waits for Tel111, with recorded messages asking people to use the online service, which may have affected subsequent use of the online service. Health-seeking behaviours may have changed as a consequence of the pandemic and it is likely that public perceptions of both online and telephone 111 will also have changed.
The study was undertaken during the first months of the online service being rolled out, which may have an impact on findings as users had less experience of and familiarity with the service. Most of the user interview participants were interviewed after using the online service just once. However, although they could not reflect on multiple uses of the service, they were able to provide information on first impressions, which will be key to influencing future behaviour.
Use of OL111 and actions undertaken in response to the disposition provided was shaped by social expectations and protection of the NHS, which may not be transferable to health settings in other countries.
Interview participants were broadly representative of the online user population in terms of ethnicity and sex, although slightly under-represented the younger age bracket who are high users of the online platform.15 Adjekum et al 27 identified that sociodemographic factors influence individual’s trust in digital health and it is plausible that our findings may be less transferable to younger populations, who have higher levels of digital literacy.27
Implications
It is unclear whether there will be significant channel shift to the online service, particularly given how the two services are used differently. OL111 is seen as a useful adjunct to but not a replacement for Tel111. It is most valued for simple conditions, as a more trusted alternative to Google, but will potentially lead to duplication of services rather than significant channel shift if people use the online service speculatively prior to using the telephone service. This may have an impact on overall demand for wider services, but any increase in service use may be limited if users do not follow the advice provided. The telephone service provided higher levels of trust and satisfaction in both the interaction itself and the recommendation provided, which suggests that this may always be a preferred first option for the majority of users when call times allowed. If the online service does not provide users with what they needed, duplicated demand may diminish over time.
Whether people use the online or the telephone service will be a function of the busyness of the telephone service, the reason for the call and the specific needs and preferences of the individual. For some, OL111 will deliver all that they require, but we do not know what they would definitively have done if they had not accessed this service. Tel111 was perceived to be preferable for more complex or urgent problems, and future development of OL111 will require refinement of algorithms if it is to meet the needs of people with multimorbidity and long-term conditions. This might include an accompanying live chat option to check that questions have been understood correctly and to explore whether people have been able to say everything they felt was important with respect to their health problem. This would increase the value of the interaction and the human interaction involved in the chat may increase confidence in recommendations.
Communication about NHS 111 telephone and online services should include advice about which service to use, and help to guide users towards the most appropriate service for their need at the first point of contact.
The authors would like to thank all of the staff and 111 online users who gave their time to help with the research, Dan Fall and members of the Sheffield Emergency Care Forum (PPI group) for their useful feedback, NIHR for funding the research, members of the steering group and Marc Chattle for providing administrative support throughout.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Not applicable.
Ethics approval
Ethics approval for the interview-based work with service users and stakeholders was granted by North West Haydock Research Ethics Committee (reference 19.NW/0361). The University of Sheffield Research Ethics Committee granted ethics approval for the telephone and online NHS 111 user surveys (reference 030991). Participants gave informed consent to participate in the study before taking part.
Handling editor Alex Novak
Twitter @fcsampson
Contributors JT conceived the study, obtained research funding, oversaw the study and contributed to the analysis. ELK undertook user interviews and led the analysis of interview data. JL undertook user interviews and contributed to the analysis of interviews. FCS undertook staff interviews and analysis, led the converged analysis and drafted the article. All authors contributed to the research design. JC contributed to the analysis and drafting the paper. FCS drafted the article and ELK, JL, JT and JC critiqued the paper for important intellectual content. FCS takes responsibility for the paper as a whole and acts as guarantor.
Funding This paper presents independent research funded by the National Institute for Health and Care Research (NIHR HSDR 127655). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
1 NHS England. The NHS five year forward view; 2014.
2 NHS England Website 2022, Available: https://www.england.nhs.uk/statistics/statistical-work-areas/ [Accessed 24 Nov 2022 ].
3 Turner J, Coster J, Chambers D, et al. What evidence is there on the effectiveness of different models of delivering urgent care? A rapid review. Health Serv Deliv Res 2015; 3: 1–134. doi:10.3310/hsdr03430
4 England N. NHS 111 minimum data set 2018-19; 2018.
5 Munro J, Nicholl J, O’Cathain A, et al. Impact of NHS direct on demand for immediate care: observational study. BMJ 2000; 321: 150–3. doi:10.1136/bmj.321.7254.150
6 Turner J, O’Cathain A, Knowles E, et al. Impact of the urgent care telephone service NHS 111 pilot sites: a controlled before and after study. BMJ Open 2013; 3: e003451. doi:10.1136/bmjopen-2013-003451
7 NHS England. Next steps on the NHS Five Year Forward View. NHS England: Leeds, 2017.
8 NHS England. NHS111 online evaluation. NHS England: Leeds, 2017.
9 Armstrong S. The Apps attempting to transfer NHS 111 online. BMJ 2018; 360: k156. doi:10.1136/bmj.k156
10 Carter M, Fletcher E, Sansom A, et al. Feasibility, acceptability and effectiveness of an online alternative to face-to-face consultation in general practice: a mixed-methods study of webGP in six Devon practices. BMJ Open 2018; 8: e018688. doi:10.1136/bmjopen-2017-018688
11 Cowie J, Calveley E, Bowers G, et al. Evaluation of a Digital consultation and self-care advice tool in primary care: A multi-methods study. Int J Environ Res Public Health 2018; 15: 896. doi:10.3390/ijerph15050896
12 Madan A. Webgp: the virtual general practice. London, 2014.
13 Chambers D, Cantrell A, Johnson M, et al. Digital and online symptom Checkers and assessment services for urgent care to inform a new Digital platform: a systematic review. Health Serv Deliv Res 2019; 7: 1–88. doi:10.3310/hsdr07290
14 Nijland N, Cranen K, Boer H, et al. Patient use and compliance with medical advice delivered by a web-based triage system in primary care. J Telemed Telecare 2010; 16: 8–11. doi:10.1258/jtt.2009.001004
15 Turner J, Knowles E, Simpson R, et al. Impact of NHS 111 online on the NHS 111 telephone service and urgent care system: a mixed methods study. Health Serv Deliv Res 2021; 9: 1–148. doi:10.3310/hsdr09210
16 Creswell JW, Plano Clark VL. Designing and conducting mixed methods Research2011;
17 Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006; 3: 77–101. doi:10.1191/1478088706qp063oa
18 Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: 573. 2002: 305–29. doi:10.4135/9781412986274
19 Gottliebsen K, Petersson G. Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review. BMJ Health Care Inform 2020; 27: e100114. doi:10.1136/bmjhci-2019-100114
20 Wallace W, Chan C, Chidambaram S, et al. The diagnostic and triage accuracy of Digital and online symptom checker tools: a systematic review. NPJ Digit Med 2022; 5: 118. doi:10.1038/s41746-022-00667-w
21 Donovan E, Wilcox CR, Patel S, et al. Digital interventions for parents of acutely ill children and their treatment-seeking behaviour: A systematic review. Br J Gen Pract 2020; 70: e172–8. doi:10.3399/bjgp20X708209
22 Yu SWY, Ma A, Tsang VHM, et al. Triage accuracy of online symptom Checkers for accident and emergency Department patients. Hong Kong Journal of Emergency Medicine 2020; 27: 217–22. doi:10.1177/1024907919842486
23 Semigran HL, Levine DM, Nundy S, et al. Et al comparison of physician and computer diagnostic accuracy. JAMA Intern Med 2016; 176: 1860–1. doi:10.1001/jamainternmed.2016.6001
24 Lee VV, Vijayakumar S, Lau NY, et al. Understanding the user: patients’ perception, needs, and concerns of health Apps for chronic constipation. DIGITAL HEALTH 2022; 8: 205520762211046. doi:10.1177/20552076221104673
25 Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom Checkers and Apps in Australia. Med J Aust 2020; 212: 514–9. doi:10.5694/mja2.50600
26 Berry AC, Berry NA, Wang B, et al. Symptom Checkers versus doctors: a prospective, head-to-head comparison for cough. Clin Respir J 2020; 14: 413–5. doi:10.1111/crj.13135
27 Adjekum A, Blasimme A, Vayena E. Elements of trust in Digital health systems: Scoping review. J Med Internet Res 2018; 20: e11254. doi:10.2196/11254
© 2023 Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ.