An evaluation of the national testing response during the COVID-19 pandemic in England: a multistage mixed-methods study protocol

Abstract

Introduction

In 2020, the UK government established a large-scale testing programme to rapidly identify individuals in England who were infected with SARS-CoV-2 and had COVID-19. This comprised part of the UK government’s COVID-19 response strategy, to protect those at risk of severe COVID-19 disease and death and to reduce the burden on the health system. To assess the success of this approach, the UK Health Security Agency (UKHSA) commissioned an independent evaluation of the activities delivered by the National Health System testing programme in England. The primary purpose of this evaluation will be to capture key learnings from the roll-out of testing to different target populations via various testing services between October 2020 and March 2022 and to use these insights to formulate recommendations for future pandemic preparedness strategy. In this protocol, we detail the rationale, approach and study design.

Methods and analysis

The proposed study involves a stepwise mixed-methods approach, aligned with established methods for the evaluation of complex interventions in health, to retrospectively assess the combined impact of key asymptomatic and symptomatic testing services nationally. The research team will first develop a theory of change, formulated in collaboration with testing service stakeholders, to understand the causal pathways and intended and unintended outcomes of each testing service and explore contextual impacts on each testing service’s intended outcomes. Insights gained will help identify indicators to evaluate how the combined aims of the testing programme were achieved, using a mixed-methods approach.

Ethics and dissemination

The study protocol was granted ethics approval by the UKHSA Research Ethics and Governance Group (reference NR0347). All relevant ethics guidelines will be followed throughout. Findings arising from this evaluation will be used to inform lessons learnt and recommendations for UKHSA on appropriate pandemic preparedness testing programme designs; findings will also be disseminated in peer-reviewed journals, a publicly available report to be published online and at academic conferences. The final report of findings from the evaluation will be used as part of a portfolio of evidence produced for the independent COVID-19 government inquiry in the UK.

Transparency declaration

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate and transparent account of the study being reported; no important aspects of the study have been omitted, and any discrepancies from the study as planned have been explained.

Correspondence to Dr Reshania Naidoo; [email protected]

STRENGTHS AND LIMITATIONS OF THIS STUDY

• This is the first national-scale evaluation of the COVID-19 testing programme in England, a critical part of the UK’s pandemic response strategy, to incorporate a broad scope of testing services across different settings.

• Whereas most complex interventions are ideally accompanied by a prospective evaluation design initiated at the time of the intervention or earlier, this study will comprise a retrospective evaluation, and therefore, the methods used are limited by the availability of data from research already conducted.

• The rapid turnaround time required for the evaluation by the funder (UK Health Security Agency) also means that the scope is necessarily narrower and trade-offs must be made between feasibility and precision when selecting appropriate research methods.

• However, the proposed mixed-methods approach, comprising economic, statistical and qualitative strands, could feasibly be applied to improve the rigour and value of evaluations of complex public health interventions or policy in other health emergency contexts in the future, especially where there are time constraints but data are available retrospectively.

Introduction

On 11 March 2020, the WHO declared COVID-19 a pandemic and exhorted member states to ‘test, test, test’.¹ In response to the COVID-19 pandemic, the UK government committed to mass testing, with initial testing commencing in March 2020.² National Health System (NHS) Test and Trace (NHSTT) was formally established in May 2020, as a part of the Department of Health and Social Care (DHSC), to lead an ‘at scale’ national testing and tracing service.³ The UK Health Security Agency (UKHSA) was established, also as an Executive Agency of DHSC, on 1 April 2021 and was operational on 1 October 2021.⁴ UKHSA combines the health protection, clinical and scientific functions formerly carried out by Public Health England with the functions of NHSTT and the Joint Biosecurity Centre.⁵

During the initial phases of the pandemic, in the absence of pharmaceutical interventions or a vaccine, testing was seen as a key means to reopen society. The UK government’s COVID-19 response strategy was, therefore, deployed in the context of mitigating the impact of the pandemic and the measures employed to control it on key areas of society, such as education, protection of livelihoods and preventative and mental health.⁶ NHSTT was tasked with providing mass-scale testing and tracing systems to rapidly identify individuals with COVID-19 and their close contacts, thereby minimising the spread of the disease.⁷ The NHSTT programme had four main stated objectives: (1) to increase the speed and availability of testing, (2) to identify close contacts of positive cases and require them to isolate, (3) to contain local outbreaks via a coordinated response and (4) to enable the government to learn more about the virus and explore ways to ease infection control measures as the science developed.³ The programme, at its scale, was the first of its kind in the UK and was created and delivered at pace during a period of unprecedented uncertainty.

The testing programme component of NHSTT played an integral role in the government’s COVID-19 response through the programme’s various testing services. The testing programme sought to work in partnership with national and local public health bodies, local authorities, the NHS, and commercial and academic providers. The delivery of testing for each of the target populations was multimodal, through combinations of in-person testing, for example, public regional testing sites and mobile testing units, and via pharmacies and home direct self-test kit deliveries; it was also driven in part by the technology available at the time, for example, accredited self-sample collection was originally not an option, so physical testing sites were required. The delivery of testing was initially focused on regional and local PCR testing sites, followed by service-specific testing sites, and then PCR and lateral flow device (LFD) home testing was rolled out as evidence accrued that this was a viable approach. These approaches to testing were subject to ongoing revision by policy-makers throughout the pandemic, dependent on factors such as changing epidemiological prevalence, the emergence of new variants of concern, for example, the Delta and Omicron variants, updated scientific evidence and vaccination roll-out.⁸

Tests for COVID-19 include those that detect the presence of the SARS-CoV-2 virus and those that detect the presence of antibodies to the virus.⁹ Tests for the virus that detect viral nucleic acids, such as PCR and loop-mediated isothermal amplification, are usually performed in a laboratory. These tests can appear positive beyond the period of infectiousness.¹⁰ LFDs are test devices that can detect SARS-CoV-2 viral protein (antigen) and represent a more rapid approach to testing. They can also be used for self-testing, although they are less sensitive than nucleic acid-based tests¹¹ and tend only to appear positive during the period of maximal viral shedding.¹² At the start of the pandemic, only PCR tests were available for testing swabs from suspected cases of COVID-19; however, LFDs for COVID-19 were rapidly developed and being evaluated for use by mid-2020.¹³ LFDs were initially rolled out for asymptomatic testing, followed by a confirmatory PCR test in the case of a positive LFD result.

The overall success and effectiveness of any national COVID-19 testing service are dependent on multiple contextual factors and the combined impacts of the various testing services. From a public health perspective, increased testing has been shown to result in reduced transmission of SARS-CoV-2 and associated hospitalisations and mortality.^14–18 Behavioural responses to testing strategies are dependent on public awareness and trust, with adherence to testing policy shown to be driven by perceptions of disease risk and socioeconomic factors.¹⁹ From an equity perspective, COVID-19 testing has been found to exacerbate existing health inequities, with disparities in access to testing for individuals from ethnic minority backgrounds and those living in socially deprived areas.²⁰ The economic impact of testing can be assessed via cost-effectiveness evaluations of testing strategies, including testing unit costs, operational deployment costs and, on a macroeconomic level, quantifying the economic productivity gained from shortened isolation periods and savings to the taxpayer.^{21 22}

UKHSA has appointed a consortium from Ernst & Young in partnership with academics from the University of Oxford, contracted through Oxford University Innovation Limited, to evaluate the testing programme; the evaluation will take 8 months. This consortium (hereafter referred to as ‘the evaluation consortium’) will undertake an independent evaluation of the testing capabilities delivered by the national COVID-19 testing programme in England from October 2020 to March 2022 (hereafter referred to as ‘the evaluation’), with a focus on public health outcomes. This evaluation will seek to capture key learnings from the roll-out of testing to the various target populations via the different testing services during this period. The insights gained will inform the formulation of key considerations and recommendations for the future pandemic preparedness. Owing to the complexity and rapidly evolving nature of the testing response, no evaluation of the national testing service in England, using a mixed-methods approach, has been conducted to date.

Testing services in scope

The evaluation consortium and UKHSA have decided to conduct an overarching evaluation of the national COVID-19 testing programme in England by assessing the combined impact of the asymptomatic and symptomatic testing services. A major component of this will be an evaluation of the universal testing service. This will be complemented by ‘deep-dive’ evaluations of three ‘priority’ testing services (see table 1 for further details):

• Schools (secondary school pupils aged 11–18 years).

• Adult social care (staff and residents in care homes).

• Healthcare workers.

Table 1

Overview of the testing services of the COVID-19 testing programme identified as within scope for this evaluation

Testing service	Purpose of testing
Schools	Testing of staff and students to prevent transmission within schools and, as they are a high-contact group, within the community and to allow schools to continue functioning as normally as possible
Adult social care	Testing to prevent outbreaks in vulnerable populations (predominantly those in care homes and those receiving domiciliary care)
Healthcare	Testing of NHS staff and patients to reduce the risk of transmission within hospital and other healthcare settings, which could then impact transmission in the community
Universal testing	Rapid, asymptomatic, at-home testing with LFDs that were made available to the general public to enable faster detection of cases, thereby allowing more rapid self-isolation

LFDs, lateral flow devices; NHS, National Health System.

These four services have been included to ensure a broad spectrum of testing populations in England will be evaluated within the given time frame, that is, high volumes (universal testing), high-contact groups (schools) and high-risk groups (care homes and healthcare workers), to best reflect the challenges faced during the pandemic and to balance the findings and recommendations for future pandemics.

The evaluation will follow a hypothesis-led approach, with three key hypotheses developed; based on these hypotheses, the evaluation will seek to answer several research questions (box 1).

Box 1

Hypotheses developed and research questions to be explored in the evaluation

Hypothesis 1: Testing services aimed at protecting high-risk groups, for example, people in care homes and healthcare workers, led to a reduction in hospitalisations and deaths in these risk groups.

Hypothesis 2: Testing services aimed at high-contact groups, for example, children in schools, led to a reduction in hospitalisations and deaths in the general population.

Hypothesis 3: Testing services aimed at increasing access to and eligibility for testing and targeting disproportionately impacted groups, for example, the universal testing service, led to increased testing uptake in these populations

Research questions:

• How was the national COVID-19 testing programme delivered and what factors affected this?

• What were the barriers and facilitators to access, use and deliver the programme?

• What were the costs and the cost-effectiveness of the programme?

• For the universal testing service:

  • Did the diversity of those reporting test results increase?

  • Did the barriers and facilitators for testing, reporting and acting on a result change?

• For each testing service:

  • Did the service achieve UKHSA’s intended aims and purposes of the service?

  • Was the service cost-effective?

  • If testing was to be implemented again, what are the barriers and facilitators to increase access, use and delivery of tests?

Methodology

Principles guiding the evaluation

The testing programme can be considered to be a complex public health intervention due to the complexity of the intervention design, which evolved over time according to policy changes and the evolving disease context within which it was implemented.²³ Therefore, this retrospective evaluation will employ a mixed-methods approach, using existing frameworks that have previously been applied to the evaluation of complex interventions, being broadly divided into process, outcome and impact evaluation components.^24–27

The evaluation will consist of a phased process:

• A scoping phase, to develop a theory of change (ToC), evaluation aims and research questions.

• A design phase, to agree the mixed-methods approach to the evaluation and confirm process, outcome and impact indicators.

• A conduct phase, to collect, review and synthesise data.

• A recommend phase, during which findings and data will be triangulated to form conclusions and recommendations.

The Medical Research Council framework for process evaluation details three key interrelated components required as part of any evaluation of a complex public health process: implementation, mechanisms of impact and context.²⁷ The ToC approach^{28 29}—a theory of how and why an initiative works—will, therefore, be used to map the causal pathways for each of the testing services, as this approach lends itself to understanding complex interventions with multiple causal pathways.³⁰ The ToC framework will be used to understand the causal pathways and intended and unintended outcomes of each service, as well as exploring the effect of context on each individual testing service’s intended outcomes.

The ToCs will be formulated using existing evidence provided by UKHSA and will be subject to further refinement following interviews with UKHSA policy and operational stakeholders. Insights from existing UK government-commissioned evidence in the form of previous COVID-19 service-specific testing evaluations and pilot testing programme data will be used where possible to avoid duplication of work. Owing to the unpredictability of evaluating complex interventions such as this, sufficient flexibility will be allowed within the research design to allow emerging research questions to be addressed.²⁷

A comprehensive ToC for each of the testing services in scope will subsequently inform process and outcome indicators (table 2) that will be used to assess each service, address the research questions and determine how well each service met its objectives and intended purpose.³¹ Based on data availability and granularity, these indicators will be further refined based on what is achievable within the evaluation time frame.

Table 2

Example of potential evaluation indicators for the overall national testing programme

Indicator type	Indicator description	Proposed method type to evaluate the indicator
Process/Output	Average percentage of LFDs distributed that were reported over the 18-month time period	Statistical
	Average percentage of PCR tests registered that were reported over the 18-month time period	Statistical
	Number of LFDs distributed over the 18-month time period	Statistical
	Number of PCR tests registered over the 18-month time period (separated by pillars 1 and 2* and total)	Statistical
	Cost per LFD conducted over the 18-month time period	Economic
	Cost per PCR test conducted over the 18-month time period	Economic
	Total financial cost of the testing service over the 18-month time period	Economic
	Total number of tests reported per week	Statistical
	Reported test positivity of LFD and PCR by indices of multiple deprivation decile before and after roll-out of universal testing	Statistical
	Ratio of mean number of reported tests in the upper-income decile: the lowest-income decile	Statistical
Outcome	Number of hospitalisations averted with different assumptions of reductions in new cases	Economic
	Number of deaths averted with different assumptions of reductions in new cases	Economic
	Cost savings from hospitalisations averted with different assumptions of reductions in new cases	Economic
	What were the barriers and facilitators to taking a test, reporting a test and acting on a positive test result?	Rapid scoping review
Impact	Cost per quality-adjusted life-year gained with different assumptions of reductions in new cases	Economic

*Pillar 1: swab testing for SARS-CoV-2 in UKHSA laboratories and NHS hospitals for those with a clinical need and for health and care workers; pillar 2: swab testing for SARS-CoV-2 in the wider population, through commercial partnerships, either processed in a laboratory or more rapidly via LFDs.

LFDs, lateral flow devices; NHS, National Health System; UKHSA, UK Health Security Agency.

The indicators will be categorised as follows:

• Process and output indicators: how did the delivery and uptake of the service compare with what was planned over time, and what factors affected this?

• Outcome indicators: what was the effectiveness of each service in terms of its intended outcomes?

• Impact indicators: what were the broader economic and societal impacts? What were the overall impacts on minimising transmission while limiting harm?

A gap analysis will also be performed during the ‘conduct’ stage to determine whether any crucial data, research or information is unavailable within the existing literature or data sources provided by UKHSA and whether subsequent adjustment of the research questions or descoping is required. This will be decided through a participatory consultation approach with key UKHSA stakeholders.

Mixed-methods approach to be used in the evaluation

A convergent parallel mixed-methods approach will be taken to provide a more holistic understanding of the testing programme, allowing the researchers to capture any unanticipated aspects of testing that may be relevant to the research questions and to explain the quantitative findings.³²

Findings from the rapid scoping review will be triangulated with the results of the statistical and economic analyses and fed back into the developing ToCs, to refine and explain the indicators, address the hypotheses and help form the recommendations for each testing service during the various evaluation phases. In addition, findings across each of the four testing services will be compared, to identify universal as well as service-specific barriers and facilitators to testing, reporting and self-isolation following a positive test result.

The ToCs will be populated in a collaborative manner, enabling the quantitative and qualitative research teams between them to define and align the process and outcome indicators for each testing service. Each team will conduct research concurrently for the various testing services and will communicate their findings at weekly meetings, allowing for emerging findings to be discussed and explored further if deemed necessary. Findings will also be synthesised across the testing services to inform programme-level insights to meet the programme-level research aims and inform the overarching ToC and indicators.

Qualitative data analysis: rapid scoping review of testing behaviours

A rapid scoping review will be conducted to evaluate the barriers and facilitators to engaging with COVID-19 testing, reporting results and self-isolating following a positive result in the UK. This review will aim to (1) provide a summary of the research undertaken on this topic available both in the UKHSA grey literature and in the wider academic literature, (2) identify gaps in research efforts and (3) provide an overview of key barriers and facilitators for engagement with each testing service, as well for the overall testing programme. A scoping study approach has been selected as the method to synthesise the broad knowledge base on this topic, owing to the large volume of heterogeneous literature available and to accommodate the time constraints of the commissioned evaluation.³³ The scoping review will be conducted adhering to the 2005 Arksey and O’Malley framework,³⁴ incorporating the adaptations proposed in 2010 by Levac et al on iterative study selection and stakeholder consultation³⁵ and also using the 2015 Joanna Briggs Institute guidance on conducting scoping reviews.³⁶ The findings will be reported according to the standardised Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews checklist.³⁷

Search strategy and the selection of evidence

A wide search strategy has been developed, using key phrases from relevant articles³⁴ (see table 3 for categories and example terms). This will be used to identify literature that describes behaviour around COVID-19 testing, reporting and self-isolation in the UK during the COVID-19 pandemic. The search strategy will be adapted for each database and information source that is searched and will be refined based on keywords in sources identified during the search.

Table 3

Search categories and examples of search terms

Category	Example search terms
COVID-19	COVID* OR corona OR coronavirus OR SARS-CoV-2 OR ‘SARS CoV 2’ OR ‘SARS CoV-2’ OR SARS-CoV2 OR SARSCoV2 OR ‘Severe Acute Respiratory Syndrome Coronavirus 2’ OR ‘Severe Acute Respiratory Syndrome Corona Virus 2’ OR 2019-nCoV
AND
Key activities	test* OR screen* OR RT-PCR OR PCR OR polymerase chain reaction’ OR ‘lateral flow’ OR ‘lateral flow device*’ OR ‘lateral flow assay*’ OR LFD OR self-test* OR ‘test and trace’ OR ‘contact trac*‘ OR surveillance OR POCT OR report* OR self-report* OR selfreport* OR ‘test positive’ OR ‘testing positive’ OR result* OR ‘self-isolation’ OR ‘self isolation’ OR isolat* OR containment OR reopening OR re-opening OR mitigat* OR flatten*
‘AND
Behaviour, barriers and facilitators	knowledge OR understand* OR attitude* OR perception* OR perceive OR belief* OR believ* OR expectation* OR trust OR willing* OR intention* OR behaviour* OR behavior* OR practice* OR enact* OR engag* OR adher* OR complian* OR comply OR experience* OR view* OR motivation* OR barrier* OR block* OR challeng* OR difficult* OR facilitat* OR enabl* OR access* OR feasib* OR accept* OR uptake
AND
Research methods	qualitative* OR interview* OR FGD OR ‘focus group*’ OR survey* OR questionnaire* OR mixed-method* OR ‘mixed method*’ OR ethnograph* OR theme OR thematic* OR ‘grounded theory’ OR ‘content analysis’ OR field-work OR ‘field work’ OR selfreport* OR self-report* OR ‘self report*’ OR view* OR experience* OR hermeneutic OR phenomenolog*
AND
Geographic setting	‘United Kingdom’ OR UK OR England OR Ireland OR Irish OR Scot* OR Wales OR Britain OR British OR NHS OR ‘National Health Service*’ OR UKHSA OR ‘United Kingdom Health Security Agency’ OR ‘UK Health Security Agency’ OR ‘Channel Island*’ OR London OR Birmingham OR Liverpool OR Manchester OR Cardiff OR Belfast OR Edinburgh OR Glasgow

The databases to be searched include PubMed, Scopus and the WHO COVID-19 Research Database. The search strategy aims to identify both published and unpublished studies (including grey literature), as well as reports and guidance documentation. Research, published from 1 January 2020 onwards and in English, will be included in the review if they focus on one or more of the following three behaviours: undertaking a test, reporting a test result and self-isolating following a positive result, symptoms or a positive contact (see table 4 for search limits and eligibility criteria).

Table 4

Summary of the search parameters and limits as well as the inclusion and exclusion criteria, categorised according to the ‘population, context, concept’ search framework^{34 36}

	Inclusion	Exclusion
Search limits
Language	Published in English	Published in languages other than English
Dates	Published between 1 January 2020 and the search date (the database search was conducted on 7 November 2022 and the UKHSA documents were received throughout September to December 2022)	Published before 2020
Methods	Qualitative or mixed-methods studies; quantitative surveys	Quantitative studies only reporting associations between demographic variables and behavioural outcomes
Eligibility
Literature	Journal articles, peer-reviewed material, articles under review, published books and book chapters, other academic research, research commissioned by governments, unpublished reports	Opinion or statement pieces, magazine articles, blog posts
Population	England, Northern Ireland, Scotland, Wales and the islands making up the British Isles; multicountry studies were included if they included one of these settings	Countries outside the UK, including the Ireland
Concept (key activities)	Descriptions of individuals’ behaviour and/or barriers to testing and/or facilitators to testing with regard to the following key activities: - Antigen testing for COVID-19 (with a focus on LFDs, but including LAMP and PCR testing) - Reporting test results - Isolating (with a focus on isolating due to a positive COVID-19 test result, but including isolating after being identified as a close contact of a COVID-19-positive case) The description of behaviours included associations of survey responses with behaviour or intention to test, report or isolate	Descriptions of testing, reporting or self-isolation but not the behaviour associated with them, for example, describes the sensitivity of a specific test Descriptions of testing for antibodies Descriptions of the barriers or facilitators to self-isolation in the context of social distancing, self-isolation if symptomatic or traveller isolation (hotel quarantine) Descriptions of associations of demographic factors with behaviour or intention to test, report or isolate Testing, reporting results or self-isolation after a positive result in the context of other diseases Descriptions of facilitators or barriers to other COVID-19-related behaviours, such as vaccination or social distancing Descriptions of the impact of testing/reporting/self-isolation on behaviour Knowledge, attitudes or perceptions in relation to COVID-19 itself

LAMP, loop-mediated isothermal amplification; LFDs, lateral flow devices; UKHSA, UK Health Security Agency.

Stakeholder-identified supplementary data

UKHSA has identified a repository of data and documentation, comprising more than 5000 reports and internal documents, which may be of potential relevance to the evaluation. On commencement of the evaluation, and if a review of the documents highlights further potentially relevant sources, additional documentation will be requested by the evaluation consortium. This documentation will be searched for evidence that supports the consortium’s efforts to understand how the testing services were intended to work, how they were experienced and any prior measurement of their effectiveness. Supplementary documents to be provided by UKHSA could include:

• Testing guidance published by UKHSA.

• Testing process documentation.

• Business cases.

• Primary qualitative or quantitative research (including behavioural studies).

• Documentation involving reporting, managing or measuring the testing programme.

• Previous evaluations of testing services.

Once all of the publicly available data have been screened, these stakeholder-identified sources will also be reviewed for inclusion.

Stakeholder consultation

Stakeholder engagement is considered useful for adding methodological rigour to scoping studies.³⁵ Therefore, stakeholders from the evaluation sponsor team within UKHSA will be consulted to help identify additional sources of published and unpublished evidence (described above in ‘Stakeholder-identified supplementary data’ section) and offer perspective and validate the findings. Additional sources identified in this way will be included in the scoping review as ‘stakeholder-identified studies’,³⁸ and insights from these conversations with stakeholders will be anonymised and incorporated into the discussion of the scoping review results. Due to the rapid turnover of testing programme staff members, the availability of stakeholders that the research team will be able to interview is limited. However, the research team has been provided with the names of at least ten policy staff per testing service to interview as part of the stakeholder consultation. This list of names was provided by the funder.

Ethics committee approval for this research has been granted on the basis that informed consent be sought prior to engaging with any UKHSA stakeholders and that anonymity will be preserved in our write-up of the findings. To preserve anonymity, robust measures will be implemented, including the development of pseudonyms or codes for participants, secure storage of data on UKHSA laptops and careful handling of information during data analysis and reporting. The anonymity preservation strategy has been discussed and refined in collaboration with the study sponsor (UKHSA) and their stakeholders, prior to ethics approval being granted.

Data extraction, charting and synthesis

Data extracted from each evidence source will include study metadata (authors, title, year of publication/dissemination, publication stage, country, participant characteristics and methods), the setting (testing service and key activity) and information about the perceptions, experiences and the barriers and facilitators to each of the key activities (testing, reporting and isolating).

All sources will be collated, uploaded into Rayyan and duplicates removed.³⁹ Following an initial screening pilot to refine the eligibility criteria, titles and abstracts will then be screened by two reviewers for assessment against the refined inclusion criteria. A sample of ≥20% will be reviewed by a third reviewer to ensure consistency of inclusion.³⁵ The inter-rater agreement will then be calculated for the final list using Gwet’s first-order agreement coefficient (AC1).⁴⁰ Potentially relevant sources will be extracted fully and then assessed against the inclusion criteria. Any disagreements will be resolved through discussion, and with an additional reviewer if no consensus is reached. The same process will be undertaken for the supplementary data.

Once all the data have been extracted and assessed against the inclusion criteria, we will synthesise the data thematically. This process will be repeated for each testing service. Given the rapid timelines, the aim of the work and the scoping study methodology guidance, the included publications will not be assessed for quality.

Quantitative data analysis: statistical methods

Data collection

Quantitative data will be obtained via the secretariat of the study sponsor within UKHSA, existing UKHSA repositories, the UK Office for National Statistics (ONS), NHS Digital and Public Health Scotland; by applying directly to various holders of non-public datasets; and from other public sources of data where available, such as International Severe Acute Respiratory and Emerging Infection Consortium. General datasets will include SARS-CoV-2 seroprevalence surveys; COVID-19 vaccination data; testing coverage data and COVID-19 cases, hospitalisations and deaths.

Data analysis

The data will be analysed with the aim of (1) providing summaries of indicators identified in the ToCs, in relation to the implementation of each testing service, to better understand the extent and reach of each service and validate findings from the rapid scoping review in relation to behaviours and (2) providing estimates of the impact of each testing service, which will feed into the cost-effectiveness evaluations. Service-specific statistical approaches will be developed for each of the four testing services.

Specified primary outcome indicators will be defined for each testing service, as defined in the ToCs. Outcomes and appropriate counterfactual comparators will be defined, contingent on obtaining access to relevant data sources; these definitions may be refined using an iterative process based on concurrent analysis of qualitative data from the rapid scoping review. For each testing service, a regression-based approach²⁴ will be undertaken together with analysis of data at the local authority level (or, if the data allow, at a finer spatial aggregation), accounting for potential confounding factors such as age, sex and ethnicity profiles, as well as indicators of deprivation, population density and relevant chronic illnesses. If determined to be relevant, in collaboration with UKHSA stakeholders, predictors of engagement with health services, such as vaccination uptake or access to internet services, will also be included.

Where available, reported case and prevalence data at the lower tier local authority level will be combined to determine true case-detection ratios, defined as the percentage of all true cases that were captured by the national testing programme, using the statistical debiasing methodology described by Nicholson et al ⁴¹

Quantitative data analysis: economic evaluation methods

Data collection

The economic analyses will primarily comprise cost-effectiveness analyses, adopting a provider perspective, incorporating costs to the NHS and to local authorities. For the school testing service analysis, a societal perspective will be adopted to quantify potential productivity losses. A rapid literature review of publicly available economic data will be conducted using keyword searches of scientific databases, as well as a search of the grey literature using Google Scholar. Data relating to the volumes of tests distributed to the various testing services and the associated costs will be obtained from UKHSA. Data relating to payments made to individuals who were isolating, and other payments made, will be obtained from DHSC. Costs will be apportioned to the four testing services according to the volumes of tests distributed. The costs of hospitalisations will be obtained from the National Schedule of NHS Costs 2020/2021.⁴²

Data analysis

The data will be analysed with the aim of (1) providing an estimate of the costs for each testing service and (2) providing estimates of the value for money of each testing service. The outcome measure of quality-adjusted life-years (QALYs) gained will be used for the economic analyses. The National Institute for Health and Care Excellence (NICE) defines a QALY as ‘a measure of the state of health of a person or group in which the benefits, in terms of length of life, are adjusted to reflect the quality of life. One QALY is equal to 1 year of life in perfect health. QALYs are calculated by estimating the years of life remaining for a patient following a particular treatment or intervention and weighting each year with a quality-of-life score (on a 0–1 scale)’.⁴³ The cost per QALY gained is a critical value that can be used to determine whether an intervention is cost-effective and is used by NICE to determine whether a proposed new treatment can be covered by the NHS. We will use a value of GBP70 000 as the willingness-to-pay threshold for interventions for COVID-19, based on HM Treasury’s Green Book (2022).⁴⁴ This means that an intervention that costs less than GBP70 000 per QALY gained can be considered cost-effective. In any graphs produced, we will also indicate the NICE willingness to pay threshold of GBP30 000, for reference. QALY weights will be obtained from the relevant literature. Sensitivity analyses will be conducted to test the outcomes against the assumptions, using a ranges of estimates from the statistical analyses and the literature, for each of the testing services.

Data management plan

All individuals on the research team will have UK government Baseline Personnel Security Standard clearance.⁴⁵ All ‘official-sensitive’ data will be accessed via official, UKHSA-approved secure portals and secure UKHSA laptops. All data to be used in the proposed evaluation will be obtained from UKHSA, allied UK government bodies, such as NHS Digital and the ONS, and independent research organisations.

Patient and public involvement

None.

Discussion

The national COVID-19 testing programme in England was rapidly set up, scaled and adapted over time in response to emerging knowledge about COVID-19 transmission, severity and the availability of other public health interventions, such as vaccination. As a result, ongoing monitoring and evaluation of testing as a public health intervention and its effectiveness was not uniformly implemented. This evaluation will, therefore, aim to retrospectively evaluate the national testing programme in England as a whole, with a deep dive into the schools, adult social care, healthcare worker and universal testing services, using a mixed-methods approach. We believe that the combined strengths of quantitative and qualitative approaches are necessary to evaluate such a complex intervention and will allow for a broader spectrum of insights and triangulation to inform future learnings.

For the purpose of this evaluation, alongside a high-level evaluation of the national testing programme as a whole, we have chosen to narrow our scope to conduct deep dives into four testing services to reflect the spectrum of the impact of testing on both high-risk and high-contact populations and to generate learnings that will be of most value, within the constraints of the evaluation time period.

To the best of the authors’ knowledge, this evaluation will be the first of its kind to produce a portfolio of evidence on testing effectiveness and public health impact for a nationally deployed testing service, focusing on the behavioural, economic, equity and public health impacts. The findings of the evaluation are expected to strengthen the evidence base in relation to COVID-19 testing and identify which aspects when deploying a complex public health intervention, such as testing, will be necessary for decision-makers to prioritise when mitigating future pandemic threats.

Ethics and dissemination

The study protocol was granted ethics approval by the UKHSA Research Ethics and Governance Group (reference NR0347). All relevant ethics guidelines will be followed throughout. Findings arising from this evaluation will be used to inform lessons learnt and recommendations for UKHSA on appropriate pandemic preparedness testing programme designs; findings will also be disseminated in peer-reviewed journals, a publicly available report to be published online and at academic conferences. The final report of findings from the evaluation will be used as part of a portfolio of evidence produced for the independent COVID-19 government inquiry in the UK.

Scientific writing assistance and editorial support were provided by Adam Bodley, according to Good Publication Practice guidelines. The authors would like to acknowledge the efforts of the UKHSA Scientific Advisory Group, who provided peer review and approval of the study protocol on behalf of the funder.

Ethics statements

Patient consent for publication

Not applicable.

Footnote

Collaborators Consortium name: EY-Oxford Health Analytics ConsortiumThe full list is included in the file upload as supplementary material for the editors.

Contributors LW and RN conceived the conceptual outline and developed the overall evaluation design. RN developed the mixed-methods approach, obtained ethics approval and led in writing and submitting the protocol for both funder peer-review and BMJ Open submission. BL, MV, KS and PD contributed to the statistical research methods. CMK, MW, SM and BA-W contributed to the rapid scoping review methodology. RS and RH contributed to the economic analysis methodology. LW, RN, SP-D and GY supported the application and management of research funding. TF supported the funder peer review and coordination of the scientific advisory group as well as responses to BMJ Open revisions. All authors contributed to the review and write-up of the protocol.

Funding This study will be funded by the Secretary of State for Health and Social Care acting as part of the Crown through the UK Health Security Agency (UKHSA), reference number C80260/PRO5331.

Competing interests All authors have completed the ICMJE uniform disclosure form at 480 www.icmje.org/coi_disclosure.pdf and declare that all authors had financial support from the UK Health Security Agency (UKHSA) for the submitted work; Ernst & Young LLP London has previously received payment for consultancy and advisory work on the NHS Test and Trace response from the UK Department of Health and Social Care, prior to the establishment of the UK Health Security Agency. There are no other relationships or 485 activities that have or could appear to have influenced the submitted work.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

References

¹ World Health Organization. WHO director-general’s opening remarks at the media briefing on COVID-19 - 16 March 2020. World Health Organization; 2020. Available: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---16-march-2020

² Department of Health and Social Care. Coronavirus (COVID-19): Scaling up our testing programmes 2020. n.d. Available: https://www.gov.uk/government/publications/coronavirus-covid-19-scaling-up-testing-programmes/coronavirus-covid-19-scaling-up-our-testing-programmes

³ Department of Health and Social Care. Government launches NHS test and trace service. HM Government; 2020. Available: https://www.gov.uk/government/news/government-launches-nhs-test-and-trace-service

⁴ UK Health Security Agency. UKHSA annual report and accounts 2021-2022. HMSO; 2023. Available: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1131855/UKHSA_Annual_Report_21_22_FINAL.pdf

⁵ UK Health Security Agency. Test and trace: Overarching privacy notice 2021. n.d. Available: https://www.gov.uk/government/publications/nhs-test-and-trace-privacy-information/test-and-trace-overarching-privacy-notice

⁶ UK Cabinet Office. Our plan to rebuild: The UK Government’s COVID-19 recovery strategy, 2020. Available: https://www.gov.uk/government/publications/our-plan-to-rebuild-the-uk-governments-covid-19-recovery-strategy

⁷ SAGE. Thirty-Second SAGE Meeting on Covid-19. 2020.

⁸ UK Health Security Agency. NHS test and trace Statistics (England): methodology 2022. n.d. Available: https://www.gov.uk/government/publications/nhs-test-and-trace-statistics-england-methodology/nhs-test-and-trace-statistics-england-methodolog

⁹ Jarrom D, Elston L, Washington J, et al. Effectiveness of tests to detect the presence of SARS-Cov-2 virus, and antibodies to SARS-Cov-2, to inform COVID-19 diagnosis: a rapid systematic review. BMJ Evid Based Med 2022; 27: 33–45. doi:10.1136/bmjebm-2020-111511

¹⁰ Teymouri M, Mollazadeh S, Mortazavi H, et al. Recent advances and challenges of RT-PCR tests for the diagnosis of COVID-19. Pathol Res Pract 2021; 221. doi:10.1016/j.prp.2021.153443

¹¹ Leber W, Lammel O, Siebenhofer A, et al. Comparing the diagnostic accuracy of point-of-care lateral flow antigen testing for SARS-Cov-2 with RT-PCR in primary care (REAP-2). EClinicalMedicine 2021; 38. doi:10.1016/j.eclinm.2021.101011

¹² Geddes L. Vaccines work:Gavi, the vaccine alliance. 2022. Available: https://www.gavi.org/vaccineswork/infectious-period-are-cuts-covid-19-isolation533 times-good-idea [Accessed 13 Mar 2023 ].

¹³ Peto T, UK COVID-19 Lateral Flow Oversight Team. COVID-19: rapid antigen detection for SARS-Cov-2 by lateral flow assay: A national systematic evaluation of sensitivity and specificity for mass-testing. EClinicalMedicine 2021; 36. doi:10.1016/j.eclinm.2021.100924

¹⁴ Mercer TR, Salit M. Testing at scale during the COVID-19 pandemic. Nat Rev Genet 2021; 22: 415–26. doi:10.1038/s41576-021-00360-w

¹⁵ Bari MD, Balzi D, Carreras G, et al. Extensive testing may reduce COVID-19 mortality: A lesson from northern italy. Infectious Diseases (except HIV/AIDS) [ Preprint ]. doi:10.1101/2020.04.24.20078709

¹⁶ Rannan-Eliya RP, Wijemunige N, Gunawardana JRNA, et al. Increased intensity of PCR testing reduced COVID-19 transmission within countries during the first pandemic wave. Health Aff (Millwood) 2021; 40: 70–81. doi:10.1377/hlthaff.2020.01409

¹⁷ Liang L-L, Tseng C-H, Ho HJ, et al. Covid-19 mortality is negatively associated with test number and government effectiveness. Sci Rep 2020; 10. doi:10.1038/s41598-020-68862-x

¹⁸ Liverpool University. In: Liverpool Covid-SMART Community Testing Pilot. Liverpool: Liverpool University, 2021.

¹⁹ Thorneloe RJ, Clarke EN, Arden MA. Adherence to Behaviours associated with the test, trace, and isolate system: an analysis using the theoretical domains framework. BMC Public Health 2022; 22: 567. doi:10.1186/s12889-022-12815-8

²⁰ Public Health England. Disparities in the Risk and Outcomes of COVID-19. London: Public Health England, 2020.

²¹ Du Z, Pandey A, Bai Y, et al. Comparative cost-effectiveness of SARS-Cov-2 testing strategies in the USA: a Modelling study. The Lancet Public Health 2021; 6: e184–91. doi:10.1016/S2468-2667(21)00002-5

²² Iacobucci G, Coombes R. Covid-19: government plans to spend £100Bn on expanding testing to 10 million a day. BMJ 2020; 370. doi:10.1136/bmj.m3520

²³ Petticrew M. When are complex interventions ‘complex’? When are simple interventions ‘simple Eur J Public Health 2011; 21: 397–8. doi:10.1093/eurpub/ckr084

²⁴ HM Treasury. Magenta Book - Central Government Guidance on Evaluation. London: HM Treasury, 2020.

²⁵ Skivington K, Matthews L, Simpson SA, et al. Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update. Health Technol Assess 2021; 25: 1–132. doi:10.3310/hta25570

²⁶ Skivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of medical research Council guidance. BMJ 2021; 374: 2061. doi:10.1136/bmj.n2061

²⁷ Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: medical research Council guidance. BMJ 2015; 350. doi:10.1136/bmj.h1258

²⁸ Fulbright-Anderson K, Auspos P, eds. Aspen Institute Roundtable on community change. In: Community Change: Theories, Practice, and Evidence. Washington D.C: Aspen Institute, 2006.

²⁹ De Silva MJ, Breuer E, Lee L, et al. Theory of change: a theory-driven approach to enhance the medical research Council’s framework for complex interventions. Trials 2014; 15. doi:10.1186/1745-6215-15-267

³⁰ Blamey A, Mackenzie M. Theories of change and realistic evaluation:peas in a pod or apples and oranges? Evaluation 2007; 13: 439–55. doi:10.1177/1356389007082129

³¹ Funnell SC, Rogers PJ. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Wiley, 2011.

³² Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods. SAGE Publications, 2014.

³³ Tricco AC, Lillie E, Zarin W, et al. A Scoping review on the conduct and reporting of Scoping reviews. BMC Med Res Methodol 2016; 16: 15. doi:10.1186/s12874-016-0116-4

³⁴ Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology 2005; 8: 19–32. doi:10.1080/1364557032000119616

³⁵ Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010; 5: 69. doi:10.1186/1748-5908-5-69

³⁶ Peters MDJ, Godfrey CM, Khalil H, et al. Guidance for conducting systematic Scoping reviews. Int J Evid Based Healthc 2015; 13: 141–6. doi:10.1097/XEB.0000000000000050

³⁷ Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for Scoping reviews (PRISMA-SCR): checklist and explanation. Ann Intern Med 2018; 169: 467–73. doi:10.7326/M18-0850

³⁸ Hanneke R, Asada Y, Lieberman L. The Scoping Review Method: Mapping the Literature in “Structural Change” Public Health Interventions. Available: https://methods.sagepub.com/case/scoping-review-mapping-literature-structural-change-public-interventions

³⁹ Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile App for systematic reviews. Syst Rev 2016; 5. doi:10.1186/s13643-016-0384-4

⁴⁰ Gwet KL. Computing inter-Rater Reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 2008; 61: 29–48. doi:10.1348/000711006X126600

⁴¹ Nicholson G, Lehmann B, Padellini T, et al. Improving local prevalence estimates of SARS-Cov-2 infections using a causal Debiasing framework. Nat Microbiol 2022; 7: 97–107. doi:10.1038/s41564-021-01029-0

⁴² NHS England. National cost collection data publication 2022. 2020. Available: https://www.england.nhs.uk/publication/2020-21-national-cost-collection-data-publication

⁴³ NICE. Glossary. 2023. Available: https://www.nice.org.uk/glossary

⁴⁴ HM Treasury. The green book (2022). 2022. Available: https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent/the-green-book-2020

⁴⁵ Office C. HMG Baseline Personnel Security Standard. 2018.