A quasi-experimental effectiveness evaluation of the ’Incredible Years Toddler’ parenting programme on children’s development aged 5: A study protocol

Abstract

Child behavioural and mental health problems have become a public health crisis. The consequences of poor mental health in childhood have large economic costs and consequences for the individual, their families, and for society. Early intervention through parenting programmes can reduce the onset of poor mental health in childhood, hence evaluating the effectiveness of parenting programmes is critical. The ‘Incredible Years Toddler’ parent programme is an education and training intervention designed to enhance the social and emotional wellbeing of children aged 1–3 years. Whilst previous studies show Incredible Years Toddler to provide promising effects on child outcomes in the short term, the research samples have lacked ethnic diversity and representation from socioeconomically deprived families. This quasi-experimental study is registered on ISRCTN (ISRCTN49991769). We will investigate the effectiveness of Incredible Years Toddler being delivered in three neighbourhoods in inner city Bradford, England. These neighbourhoods contain a socially and ethnically diverse population with 84% living in the poorest decile for England and Wales. Parents with a child aged 1–3 years old who are enrolled in Born in Bradford’s Better Start interventional family cohort study are eligible for this study. Intervention participants will be matched to a demographically comparable control group using propensity score matching. This study will use retrospective and prospective data from participants who attended Incredible Years groups between September 2018 and April 2024. The required minimum sample is n = 1336 (ratio 1:3) to detect a small effect (odds = 1.5, d = .20) on the Early Years Foundation Stage profile total score at age 5; a measure of early child development that is routinely collected by teachers. We will also establish whether these effects are moderated by child age at entry to intervention, programme delivery mode, socioeconomic status, and ethnicity. We will also estimate the cost of the intervention and conduct a cost-consequence analysis.

Introduction

Child behavioural and mental health problems have become a public health crisis. One in eight children (aged 2–19 years) in England are estimated to experience a mental health disorder [1]. The prevalence of this increased with the COVID-19 pandemic, with the incidence rising from 11% in 2017, to 16% in July 2020 for children aged 5–16 years [2] In 2022, 18% of children aged 7 to 16 years were estimated to have a probable mental disorder [3]. Further, mental health problems are exacerbated for children living in socioeconomically disadvantaged circumstances [4, 5].

When children with mental health issues reach adulthood, they are more likely to be unemployed and to have contact with the criminal justice service [6]. Poor child mental health has large economic costs and consequences for both the individual and for society, for example, the average annual costs to mental health services for 12–15-year-olds with conduct disorders are £1,789, and £1,353 for those with emotional disorders [6].

The first three years of a child’s life are a crucial time for their development, with disruptions in early neurodevelopment being a significant risk for development of later mental health problems [7, 8]. Early intervention through parenting programmes has the potential to prevent or reduce the personal and societal costs of poor child mental health [9], since some child behavioural and mental health issues may be attributable to insecure attachments and low parental sensitivity [10, 11]. Whilst group parenting programmes have been found to be effective for reducing child conduct problems and improving child socio-emotional wellbeing for parents of children aged three years or older [12, 13], evidence remains weak for the short and long-term effectiveness of parenting programmes for parents of children in the critical first three years of life, as well as in specific subgroups of parents [9, 12, 14].

The incredible years toddler programme

One such series of programmes that may address this paucity of evidence are The Incredible Years (IY) parent programmes (www.incredibleyears.com), which are parent education and training group-based interventions informed by social learning theory and designed to enhance the social and emotional wellbeing of children aged 0–12 years. These manualised programmes are typically delivered by trained facilitators to groups of 10–12 parents for two hours a week for 12–20 weeks. This protocol outlines our plans to evaluate the Incredible Years Toddler (IY-T) programme for parents of 1-3-year-olds. In IY-T, parents learn effective strategies to help the child learn how to better manage their emotions and behaviour [15]. Whilst IY parent programmes for older children have evidence of positive short-term outcomes [14, 16], the more recently developed IY-T has less evidence in comparison.

There are only two previous studies which have investigated the effectiveness of IY-T [17, 18]. Perrin et al. [2014] conducted an RCT with 89 parents in the USA, and delivered IY Toddler through primary care (paediatric practices), rather than community settings. Significant improvements were found in parenting practices and child disruptive behaviours. Hutchings et al. [2017] conducted a community based RCT with 153 parents in Wales, and relied on geographical targeting to disadvantaged ‘Flying Start’ areas to deliver IY Toddler. Significant improvements were found in child development, home environment, and parental depression. In addition, a recent RCT of the ‘E-SEE Steps’ model (a proportionate universal approach delivering various intensities of the IY Baby and IY-T) found no significant differences in parent and child outcomes 18 months after randomisation [19]. However, the E-SEE trial was not designed or powered to establish the effectiveness of IY-T as a standalone programme.

It is of interest to know which sociodemographic groups of parents may benefit most from parenting programmes. Ethnic minorities represented 29% of the study population of the trial based in the USA [18] and no information regarding ethnicity was reported for the study based in Wales [17]. With regards to socioeconomic status, parents in the study based in Wales were living in a disadvantaged area, however, the recruited participants were less disadvantaged than parents in a comparable trial (Welsh Sure Start trial), where community disadvantage rates were lower [17, 20]. Parents in the study based in the USA had household incomes ranging between $20,000 and $100,000 [18]. The IY-T intervention therefore has limited evidence of effect in the most vulnerable populations such as those from an ethnic minority or living in areas of socioeconomic deprivation. A meta-analysis of IY parent programmes indicated that the intervention effects are not moderated by socioeconomic or ethnic groups for child conduct problems, suggesting that the programme is similarly effective for families from different backgrounds [21]. However, this meta-analysis only compared ‘White’ groups to all other ethnic minorities. This means we cannot be certain of the meaning of this finding for different ethnic minority groups, since there may be much heterogeneity regarding the programme effectiveness across different ethnic minority groups. It would, therefore, be beneficial to test whether the finding that IY parent programmes are consistently effective across different ethnic groups generalises specifically to IY-T, and generalises to other long-term outcomes.

Due to the high cost of collecting long-term data, effectiveness trials are not usually funded to explore long-term outcomes over 12 months after the intervention took place. The aforementioned evaluations of IY-T examined parent and child outcomes up to 12 months after baseline [17, 20]—which is relatively long-term compared to other effectiveness trials, however, more long-term outcome studies are needed to explore ongoing effectiveness as well as economic and societal impacts. There is some evidence that children whose parents received the IY ‘parent training and child treatment’ programme maintain positive behavioural improvements for oppositional defiant disorder and conduct disorder several years post-follow-up [22], however, studies are needed to confirm the long-term results of all IY interventions for various ages, diagnoses, and demographic populations [14].

Methods for evaluating child and family preventative services in practice

Whilst RCTs are the most rigorous method of establishing the effectiveness of an intervention [23], they are susceptible to several issues [24, 25]. RCTs can suffer from a lack of generalisability, as participants that volunteer to participate might not be representative of the population being studied. Parents who are at the point of ‘readiness to change’ are unlikely to agree to take part in a study in which they might be randomised not to get help, resulting in recruitment bias [23, 25]. A further problem is that it is usually not possible to control the opportunities through which a parent can access an intervention; since they can be made available through third sector organisations, the internet or through books. A parent who has heard about a programme during their recruitment to a study may decide to enrol in the programme through another route (even if they are in the control group of the RCT) [25]. Finally, RCTs are susceptible to a high number of participants being lost to follow up [23], especially when a study aims to explore long term effects of the intervention.

A quasi-experimental study cannot provide the level of certainty in establishing causal effectiveness that an RCT does, as it lacks random assignment to treatment group [24, 26]. However, it has advantages in terms of efficiency and pragmatism, and can overcome the RCT design limitations outlined above, such as providing ecological validity by evaluating an intervention when delivered in its natural (non-trial) context [24]. This quasi-experimental study evaluates IY-T as delivered in the Better Start Bradford programme by Barnardo’s. Parents at the point of ‘readiness to change’ are free to enrol on this intervention immediately (or not), thus reducing recruitment bias. The participants are enrolled in the Born in Bradford’s Better Start (BiBBS) interventional family cohort study, which runs in parallel to Better Start Bradford [27]. By utilising the BiBBS cohort, this study has linkage to a routinely collected and long-term outcome, and this mitigates the risk of losing participants to follow up. Finally, the BiBBS cohort contains an ethnically diverse population (61% Pakistani heritage; 12% White British; 8% other South Asian and 6% Central and Eastern European ethnicity), with the majority (84%) living in the lowest decile of the Index of Multiple Deprivation [28]. This study therefore provides a unique opportunity to advance our understanding of the effectiveness of IY-T in an ethnically diverse and socioeconomically deprived population.

Rationale and objectives for this study

The long-term effectiveness of IY-T has not yet been investigated using a predominantly disadvantaged and ethnically diverse cohort sample with routinely linked data. This study protocol outlines our plans to be the first to provide insights into the longitudinal effectiveness of IY-T on children’s social-emotional development and early years school skills at 5 years, approximately 2–4 years after their parents’ attendance at IY-T groups. The study objectives are to:

1. Establish if IY-T is effective in improving child social and emotional development at age 5 (primary outcome) when compared to services as usual

2. Establish if IY-T is effective in improving children’s early years school skills at age 5 (secondary outcome) when compared to services as usual

3. Establish whether the impact of IY-T on the above outcomes is moderated by (a) child age at time of intervention, (b) programme delivery mode (online versus face to face), and (c) socioeconomic status

4. Estimate the cost of the intervention (including delivery and training to deliver IY-T) and combine them with the effectiveness evidence through a cost-consequence analysis.

Methods

Registration

We have used the STROBE (2021) guidelines for reporting methods in this protocol and will use the guidelines for the publication of our results. This study has been registered retrospectively on ISRCTN, a clinical database for RCTs and effectiveness studies (ISRCTN49991769).

Design

This effectiveness evaluation will use a quasi-experimental design. A quasi-experimental design lacks random assignment and uses self-selection (by which participants choose treatment or a lack of treatment for themselves) [26]. The analyses will use propensity score matching, where multiple control cases will be matched to intervention cases (see statistical methods for further details).

Setting

This study is set in Bradford; a city in Northern England with high levels of socioeconomic deprivation and ethnic diversity [29]. IY-T is being delivered by Barnardo’s as part of the Better Start Bradford initiative (https://www.barnardos.org.uk/what-we-do/services/incredible-years-happy; https://www.betterstartbradford.org.uk/). Better Start Bradford is an initiative funded by the National Lottery Community Fund (previously the Big Lottery fund), which provide services for expectant families and families with children aged 0–3 in Bowling and Barkerend, Bradford Moor and Little Horton (three areas within Bradford).

Born in Bradford’s Better Start (BiBBS) is an interventional birth cohort study, which runs in parallel to Better Start Bradford, and is designed to evaluate the effectiveness of Better Start Bradford’s early life interventions [27]. BiBBS recruits pregnant women and their babies who live in the Better Start Bradford areas, regardless of intervention participation. Women complete an in-depth baseline questionnaire during pregnancy, and consent to routine linkage to their and their child’s health and education records and Better Start Bradford intervention participation. BiBBS recruits 54% of the eligible population and is generally representative of the local pregnant population, therefore the intervention and control participants in this study are also representative of this area [28].

Ethical and safety considerations

The informed consent procedure in BiBBS includes women providing written informed consent to the evaluation of Better Start Bradford interventions using quasi-experimental methods for themselves and their child, including the use of cohort participants to create intervention and control groups, and use of routine data to evaluate interventions.

The protocol for BiBBS recruitment and collection of routine outcome data was approved by Bradford Leeds NHS Research Ethics Committee (15/YH/0455). Research governance approval was gained from Bradford Teaching Hospitals NHS Foundation Trust (BTHFT). The sponsor is the BTHFT Research Management and Support Office who are authorised to conduct independent auditing of sponsored studies. The existing ethics includes approval for the evaluation of Better Start Bradford interventions using quasi-experimental methods, including the use of cohort participants to create control groups.

IY-T is delivered as a part of usual practice. Ethical considerations with regards to the delivery of the programme can be found in the IY-T service design plan, and issues arising are managed by the service delivery team and Better Start Bradford as service commissioners, in accordance with local guidance, e.g., such as safeguarding procedures, access to appropriate mental health support. Key members of the research team are trained in good clinical practice. There are unlikely to be harms to individuals from taking part in the evaluation as it is non-invasive, and the practitioners delivering the intervention are trained in safeguarding.

Participants

Eligibility criteria.

To be eligible for this evaluation, participants must be enrolled in the BiBBS cohort. Whilst not all IY-T attendees will be BiBBS participants, all observations in the study sample, i.e., analysed intervention and matched comparison group, will be BiBBS participants.

Eligibility criteria for the intervention group are:

* Consented to participate in BiBBS

* Parent consented to a referral and enrolled in IY-T

Eligibility for the matched comparison group are:

* Consented to participate in BiBBS

* Parent of a child aged between 12 and 36 months at any time in the duration of service delivery (to ensure that matched cases are comparable)

* Parent has not been referred or enrolled in IY-T at any timepoint

* Parent has been matched to an individual in the intervention group (see statistical methods section)

Intervention.

IY-T is the intervention. Parents living in the BSB area with a child aged 1–3 are eligible to enrol in IY-T. IY-T receive most of their enrolled participants through their own engagement with the community directly, and also through engaging with schools and nurseries. IY-T aim to recruit approximately 12 parents to each group that they run.

Within IY-T, parents learn how to help their toddlers feel loved and secure, encourage social and emotional development, and establish strategies for developing routines, handling separation, and managing misbehaviour [15]. Parents/carers receive 3 promotional contacts prior to the beginning of the group via assertive outreach, consisting of telephone contact and at least one home visit. The initial telephone contact introduces the parents to the project and Group Facilitators, and aims to build the participants confidence in attending. The home visits are intended to create a sense of rapport between the family and Group Facilitators and alleviate any barriers that families might have in accessing the group such as crèche, language difficulties, or concerns about what the group might involve. The intervention will be delivered in a combination of face-to-face and virtual formats (dependent on lockdown rules in place during the COVID-19 pandemic at the time of the study and participant needs) by trained Group Facilitators who receive regular supervision from an accredited IY mentor.

The 13 IY Toddler Basic sessions cover eight topics:

* Part one –Child-Directed Play Promotes Positive Relationships

* Part two–Promoting Toddler’s Language with Child-Directed Coaching

* Part three–Social and Emotion Coaching

* Part four–The Art of Praise and Encouragement

* Part five–Spontaneous Incentives for Toddlers

* Part six–Handling Separations and Reunions

* Part seven–Positive Discipline-Effective Limit Setting

* Part eight–Positive Discipline-Handling Misbehaviour

Further detail on the content and delivery of the IY-T intervention, and materials, can be found here: https://incredibleyears.com/programs/. The IY-T theory of change and logic model developed by Barnardo’s for Bradford delivery states that IY-T will result in improvements in child social and emotional development over the medium and long term. The logic model is provided in a S2 File.

Outcome (early years foundation stage profiles)

The primary and secondary outcomes are obtained through the child’s Early Years Foundation Stage Profile Profile (EYFSP) which will be linked to individual BiBBS children through their routine education records. The EYFSP is completed by teachers for children when they reach age five, at the end of their last term in reception. It contains 17 Early Learning Goals (ELGs) relating to different areas of learning. Previously, children were assigned a score of either “Emerging” (not reaching expected levels), “Expected” (reaching expected levels), or “Exceeding” for each ELG [30, 31]. For the new EYFSP (post September 2021), children are assigned a score of either “Emerging” or “Expected” for each ELG. See Table 1 for an overview of both the new and old EYFS ELGs.

[Figure omitted. See PDF.]

The EYFSP is an important predictor of later academic attainment and developmental difficulties [33, 34]. Whilst previous studies and The Department for Education have used the ‘good level of development’ (GLD) measure, this is a binary outcome and reduces all goals into one composite measure. An alternative to the GLD is to instead assign numerical scores to each category in the EYFSP (e.g., zero for emerging, one for expected, and two for exceeding), and sum these scores into a ‘total score’ composite (ranging between 0–34).

For each child, every goal met (or exceeded) will be assigned a one, and each goal scored as ‘emerging’ will be assigned a zero. The outcomes will be summed in the following ways:

1. For the primary outcome, we will examine only the most relevant ELG’s to child social and emotional development. These are the ‘communication and language’ and ‘personal, social, and emotional’ ELGs, with total 5 goals (see Table 2). The binary responses for each of these 5 goals will be summed to generate a summary score between 0 and 5.

2. For the secondary outcome (child early years development), the binary responses for each of the 17 ELGs will be used to generate a summary score, where a total score between 0 and 17 will indicate how many of all the ELGs were assigned a score of ‘expected’.

[Figure omitted. See PDF.]

Participants who entered reception prior to or during September 2020 will be assessed according to the old version (though this is anticipated to be a small number of participants), and participants in IY-T who enter reception after or during September 2021 will be assessed according to the new version. The measures will be made comparable by collapsing all participants who met or exceeded goals to be assigned a one, and each goal scored as emerging will be assigned a zero.

Sample size calculations

This study has a fixed sample size dependent on the projects capacity and recruitment rates, and this section explores the required sample size to obtain 80% power. The EYFSP will be modelled as an ordinal outcome, using a proportional odds model. We have calculated the numbers at which we will have 80% power to detect an odds ratio of 1.5, which is thought to be equivalent to a Cohen’s d of .20 (small effect size) [35]. We have used an allocation ratio of 1:3 (intervention:control). The Stata command artcat was used for all sample size calculations.

1. For the primary outcome using five ELGs, we will be powered to detect a difference with 1336 participants total (334 intervention participants and 1002 control participants).

2. For the secondary outcome using all 17 ELGs, we will be powered to detect a difference with 950 participants total (237 intervention participants and 713 control participants).

Data timeline

This study will use all available data since IY-T began (1^st September 2018) up until the end of their service delivery period (expected to be 30^th April 2024). Fig 1 provides the participant pathway through BiBBS cohort recruitment, participation in IY-T (or non-participation), and school entry with the outcome collected during reception year. Parents may enrol in the intervention when their child is aged anywhere between 1 and 3 years old, which means outcome data may be collected anywhere between two and four years after intervention participation. The routine education outcome in the EYFSP, which is reported by teachers at the end of children’s reception year in primary school (when child is aged 5 years old, equivalent to 60 months). The final analyses will be after August 2028, when the youngest children within the delivery period reach age 5 years.

[Figure omitted. See PDF.]

Note: Caregivers are eligible for IY-T when their child is aged 12–36 months old. Participation in the IY-T programme lasts approximately 12 weeks.

Statistical methods

A detailed statistical analysis plan will be preregistered on the Open Science Framework (OSF), this section is provided as a summary. Our analyses will use the propensity score matching method, which estimates the probability of a subject to receive a treatment conditional on the set of covariates. Matching patients with a similar estimated propensity score will create approximate balance for baseline characteristics [36]. The propensity score method will use one-to-many matching, by matching individual treatment cases to multiple control cases [37]. After matched groups have been created, ordinal logistic regression analyses will be run using group assignment as the independent variable, and the primary and secondary outcomes as the dependent variables.

Variables

The Directed Acyclic Graph (DAG) in Fig 2 identifies the confounding variables that will be used as matching variables in the propensity score [38, 39]. We will match on the 7 characteristics on the left-hand side of the graph (prenatal attachment, prenatal depression, prenatal anxiety, ethnicity, socioeconomic status, language status, and immigration status) to estimate the effect of treatment (IY-T) on the primary outcome (child social and emotional development).

[Figure omitted. See PDF.]

Notes: 1. SES = socioeconomic status. 2. The secondary outcome (children’s early years school skills) is not depicted here.

The justification for the selection of these variables is given in Table 2 below. Variables which affect program participation but not the outcomes should not be used in propensity score matching, as this increases the error in the estimated values [40]. Matching variables were therefore selected if they were thought to have an association with both uptake of treatment, and with the outcomes (child early years development), and there was evidence to suggest this was the case. See Table 2 for a summary of the evidence for using a matching variable, and the tested moderators of the effect. We have proposed that we will test the effects of four moderators: child age, programme delivery mode, family socioeconomic status, and ethnicity, however, this will only be possible if a sufficient sample size is reached.

All variables are obtained from the BiBBS baseline questionnaire, which takes place before the child is born (see https://borninbradford.github.io/datadict/bibbs/ for details).

Included participants

We will conduct both Intention to Treat (ITT) analyses to ascertain the effects of enrolling in the intervention [50, 51], and per protocol analyses to ascertain the effects of receiving the treatment [52]. The ITT analyses will therefore include all who enrolled in the intervention after an initial home visit (even if they did not go on to participate in it), and the per protocol analyses will only include ‘completers’, defined as those who take part in at least 8 out of the 13 sessions. This will give us more accurate estimates of the average treatment effect on the treated participants.

Parents are eligible for the evaluation regardless of whether they are the mother, father, or another type of co-parent. Whilst we anticipate that the majority of parents in this study will be mothers, we will describe the number of fathers and other co-parents if possible.

Missing data methods

If missing data are higher than 5% in any of the matching variables or moderators, then multiple imputation will be used to handle missing data [53]. Missing data rules for within measures will be followed where available (e.g., within the PHQ-8). Counts of children missing the outcome will be presented with reasons where possible, and tabulations of those who are missing the outcome will be created by the matching variables to explore whether there are any differences in characteristics between those with outcome data and those without.

Robustness testing

Histograms of the distribution of the EYFSP variable will be created and the analysis be run with the EYFSP as both a categorical and continuous variable. The model fit and parameters will be compared between the models. The Stata user written command ‘overfit’ will be used to measure the amount of overfitting generated by each model.

Software

Stata-17 will be used for data handling and analyses. The packages teffects and psmatch2 will be used for matching, and ologit will be used for regression analysis.

Economic analysis

The objectives for the economic analysis are to:

1. Estimate the cost of the intervention (including delivery and training to deliver IY-T),

2. seek to combine these costs with the effectiveness data to inform a cost-consequence analysis, and

3. to explore extrapolating the effectiveness data generated by this analysis to longer-term outcomes through an existing simulation model.

The costing analysis will seek to apply an existing framework for conducting micro-costing [54] in addition to previous costings conducted in a previous randomised trial of IY (the E-SEE study) [19]. Where possible, we will generate a cost per participant in each group, estimate the cost of the intervention, and who that cost falls upon. These costs will be reported alongside the effectiveness outcomes described in the ‘Outcome’ sections of this protocol to inform a cost-consequence analysis, including potential sub-group analysis stratified by socioeconomic status. Finally, by applying ongoing research mapping the EYFSP to the Strengths and Difficulties Questionnaire (SDQ) [Mooney et al., in preparation] to a life course dynamic microsimulation model [55] we will explore the extrapolation of findings of this study over the lifetime of the children who both receive IY-T and the matched control group.

Pilot data

An interim profile of BiBBS for all pregnant women recruited between January 2016 and November 2019, with an expected due date between 1st April 2016 and 8th March 2020, indicated that the BiBBS cohort sample is representative of the Better Start Bradford pregnant population. Furthermore, a high proportion (85%) of BiBBS families had engaged in one or more of the Better Start Bradford interventions, including in IY-T [28]. This data demonstrates the feasibility of conducting the IY-toddler evaluation in a disadvantaged population.

Process evaluation

A process evaluation will explore aspects of implementation fidelity including dose (number of sessions attended) and adherence (self-reported Group facilitator checklists). Parent and practitioner perspectives on the impact of IY-T, and barriers and facilitators to implementation will be explored. Fathers and other co-parents will be involved in the process evaluation where possible. The plans for this study will be specified in a separate process evaluation protocol.

Data management

As per the BiBBS protocol, record matches for routine data linkage, including health, education and IY-T data will be validated based on NHS number plus multiple non-unique identifiers (e.g., surname, date of birth) where possible. Identifiable data is only accessible by a small number of the research team (not the authors of this study), and is only done for approved processes such as withdrawals; address updates etc. Data for evaluations is processed and fully anonymised before being shared with researchers. The central database, hosted by BTHFT, will store data obtained from all sources. Data from each source will be linked at the BiBBS person level and will be structured and maintained by BiBBS data managers as a long-term strategic store to service cohort data capture and analysis. The entire database schema and data will be backed up nightly. All data will be stored within the cohort-specific Relational Database Management System (RDBMS) and managed using Microsoft SQL Server 2018 Management Studio.

Discussion

This quasi-experimental evaluation will establish the effectiveness (or not) of the IY-T programme on children’s social and emotional wellbeing and early years school skills two to four years post parent attendance in IY-T groups. It will provide a unique insight into the effects of the IY-T programme in an ethnically and socioeconomically diverse population. This study will harness the BiBBS cohort to enable links to routine data, allowing an exploration of the effects of IY-T on children’s EYFSP scores. The intervention is offered in a real-world context, in a setting where the intervention would exist whether we evaluated it or not. We will utilise an existing data set to conduct a quasi-experimental evaluation; hence it allows for a low-cost, efficient, and timely evaluation compared to a full RCT. The findings regarding intervention effectiveness of IY-T will be generalisable to other urban areas that are ethnically diverse with high levels of socioeconomic deprivation. They may also be generalisable beyond IY-T to other parenting programmes more widely, though this will depend on the contents and implementation of the individual programmes.

Beyond this study, it will be possible to follow the very long-term outcomes of the children whose parents have and have not received IY-T in the BiBBS cohort. Whilst this planned study aims to first examine the impact of IY-T on children’s social and emotional wellbeing and school skills at age 5, it will also eventually be possible to examine any impacts of IY-T into adolescence (e.g. through routine education data) and adulthood (e.g. through access to medical records to infer incidence of mental health problems). Again, this highlights the utility of a quasi-experimental study nested within an interventional cohort.

Study limitations

There is a paucity of routinely collected data within the early years of a child’s life, which limited our choice of an outcome measure. Whilst the EYFSP is a well-timed measure and represents a construct of interest to this study, other measures relating to outcomes that IY-T aims to effect would have been beneficial (e.g. of parent-child bonding) [56]. Further, as collection of EYFSP data was paused during the COVID pandemic, there will inevitably be missing outcome data for a small number of children. However, it is worth noting that EYFSP and other routinely collected health and education outcome data is regularly used as the basis for most policy-making decisions and this study reflects real-world conditions.

A further limitation of this evaluation is that it will not consider the involvement of fathers and other co-parents in the IY-T groups. Whilst fathers and co-parents of children are eligible for both BiBBS and IY-T, the number of enrolled fathers and co-parents in both the cohort and IY-T is unlikely to be sufficient to conduct any meaningful analyses on these groups. Whilst it can be more challenging to recruit fathers to parenting research studies [57, 58], we acknowledge that much further research and efforts is needed on the involvement of fathers and co-parents in parenting interventions, and will aim to explore this in our process evaluation of IY-T.

Conclusion

This study will contribute to the growing evidence base regarding the effectiveness of the IY-T programme [17–19], and more broadly, the literature regarding effectiveness of parenting education programmes in underserved communities [9, 12, 14, 21]. This quasi-experimental study will provide useful evidence for policy makers and commissioners regarding IY-T and parenting education programmes, and it may ultimately improve outcomes for children living in a disadvantaged area.

Supporting information

S1 File. Strengthening the reporting of observational studies in epidemiology statement checklist.

https://doi.org/10.1371/journal.pone.0291557.s001

(DOCX)

S2 File. Logic model of incredible years (Toddler).

https://doi.org/10.1371/journal.pone.0291557.s002

(DOCX)

Acknowledgments

The integration of research and practice in Bradford has only been possible because of the enthusiasm and commitment of staff and volunteers across children’s services in Bradford. We are grateful to all Born in Bradford staff, the Better Start Bradford partnership and staff, all Better Start Bradford project teams, health professionals, local authority and voluntary and community sector organisations who have supported the integration of research into practice. We are grateful to all the families taking part in BiBBS and all members of the Community Research Advisory Group.

Citation: Mooney KE, Bywater T, Hinde S, Richardson G, Wright J, Dickerson J, et al. (2023) A quasi-experimental effectiveness evaluation of the ’Incredible Years Toddler’ parenting programme on children’s development aged 5: A study protocol. PLoS ONE 18(9): e0291557. https://doi.org/10.1371/journal.pone.0291557

About the Authors:

Kate E. Mooney

Roles: Conceptualization, Methodology, Project administration, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Department of Health Sciences, University of York, York, United Kingdom, Bradford Institute for Health Research, Bradford Royal Infirmary, Bradford, United Kingdom

ORICD: https://orcid.org/0000-0003-4231-1643

Tracey Bywater

Roles: Conceptualization, Methodology, Writing – review & editing

Affiliation: Department of Health Sciences, University of York, York, United Kingdom

Sebastian Hinde

Roles: Methodology, Writing – review & editing

Affiliation: Centre for Health Economics, University of York, York, United Kingdom

Gerry Richardson

Roles: Methodology, Writing – review & editing

Affiliation: Centre for Health Economics, University of York, York, United Kingdom

John Wright

Roles: Conceptualization, Writing – review & editing

Affiliation: Bradford Institute for Health Research, Bradford Royal Infirmary, Bradford, United Kingdom

Josie Dickerson

Roles: Conceptualization, Methodology, Writing – review & editing

Affiliation: Bradford Institute for Health Research, Bradford Royal Infirmary, Bradford, United Kingdom

Sarah L. Blower

Roles: Conceptualization, Methodology, Writing – review & editing

Affiliation: Department of Health Sciences, University of York, York, United Kingdom

References

1. Sadler Tim Vizard, Ford Tamsin, Marcheselli Franziska, Pearce Nicola, Mandalia Dhriti, et al. Mental Health of Children and Young People in England, 2017. 2018 [cited 2022 Nov 21]; Available from: https://gss.civilservice.gov.uk/wp-

2. Newlove-Delgado T, McManus S, Sadler K, Thandi S, Vizard T, Cartwright C, et al. Child mental health in England before and during the COVID-19 lockdown. Lancet Psychiatry [Internet]. 2021 May 1 [cited 2022 Nov 21];8(5):353–4. Available from: http://www.thelancet.com/article/S2215036620305708/fulltext pmid:33444548

3. NHS Digital. Mental Health of Children and Young People in England 2022—wave 3 follow up to the 2017 survey [Internet]. 2022 [cited 2023 Apr 12]. Available from: https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2022-follow-up-to-the-2017-survey/part-1—mental-health

4. Moulton V, Goodman A, Nasim B, Ploubidis GB, Gambaro L. Parental Wealth and Children’s Cognitive Ability, Mental, and Physical Health: Evidence From the UK Millennium Cohort Study. Child Dev [Internet]. 2021 Jan 16 [cited 2021 Mar 17];92(1):115–23. Available from: https://onlinelibrary.wiley.com/doi/10.1111/cdev.13413 pmid:32939765

5. Kirby N, Wright B, Allgar V. Child mental health and resilience in the context of socioeconomic disadvantage: results from the Born in Bradford cohort study. Eur Child Adolesc Psychiatry [Internet]. 2020 Apr 1 [cited 2022 Nov 17];29(4):467–77. Available from: https://link.springer.com/article/10.1007/s00787-019-01348-y pmid:31243580

6. Knapp M, Ardino V, Brimblecombe N, Evans-Lacko S, Iemmi V, King D, et al. Youth Mental Health: New Economic Evidence. 2016 [cited 2022 Nov 21]; Available from: www.pssru.ac.uk/publication-details.php?id=5160

7. Tierney AL, Charles A. Nelson I. Brain Development and the Role of Experience in the Early Years. Zero Three [Internet]. 2009 Nov 11 [cited 2023 Mar 10];30(2):9. Available from: /pmc/articles/PMC3722610/

8. Newman L, Judd F, Olsson CA, Castle D, Bousman C, Sheehan P, et al. Early origins of mental disorder—risk factors in the perinatal and infant period. BMC Psychiatry [Internet]. 2016;16(1):1–8. Available from: pmid:27473074

9. Hurt L, Paranjothy S, Lucas PJ, Watson D, Mann M, Griffiths LJ, et al. Interventions that enhance health services for parents and infants to improve child development and social and emotional well-being in high-income countries: a systematic review. BMJ Open [Internet]. 2018 Feb 1 [cited 2022 Nov 21];8(2):e014899. Available from: https://bmjopen.bmj.com/content/8/2/e014899 pmid:29439064

10. Deans CL. Maternal sensitivity, its relationship with child outcomes, and interventions that address it: a systematic literature review. https://doi.org/101080/0300443020181465415 [Internet]. 2018 Jan 25 [cited 2021 Aug 2];190(2):252–75. Available from: https://www.tandfonline.com/doi/abs/10.1080/03004430.2018.1465415

11. Ranson KE, Urichuk LJ. The effect of parent–child attachment relationships on child biopsychosocial outcomes: a review. http://dx.doi.org/101080/03004430600685282 [Internet]. 2008 [cited 2021 Aug 2];178(2):129–52. Available from: https://www.tandfonline.com/doi/abs/10.1080/03004430600685282

12. Barlow J, Coren E. The Effectiveness of Parenting Programs. https://doi.org/101177/1049731517725184 [Internet]. 2017 Sep 22 [cited 2022 Dec 20];28(1):99–102. Available from: https://journals.sagepub.com/doi/full/10.1177/1049731517725184

13. Furlong M, McGilloway S, Bywater T, Hutchings J, Smith SM, Donnelly M. Behavioural and cognitive-behavioural group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years. Campbell Systematic Reviews [Internet]. 2012 Jan 1 [cited 2022 Nov 21];8(1):1–239. Available from: https://onlinelibrary.wiley.com/doi/full/10.4073/csr.2012.12 pmid:22336837

14. Pidano AE, Allen AR. The Incredible Years Series: A Review of the Independent Research Base. J Child Fam Stud [Internet]. 2015 Jul 8 [cited 2022 Nov 21];24(7):1898–916. Available from: https://link.springer.com/article/10.1007/s10826-014-9991-7

15. The Incredible Years. Toddler Basic Program | Incredible Years [Internet]. 2013 [cited 2021 Nov 18]. Available from: https://incredibleyears.com/programs/parent/toddler-curriculum/

16. Early Intervention Foundation. Incredible Years Preschool | EIF Guidebook [Internet]. 2016 [cited 2021 Nov 18]. Available from: https://guidebook.eif.org.uk/programme/incredible-years-preschool

17. Hutchings J, Griffith N, Bywater T, Williams ME, Hutchings J, Psych D, et al. Evaluating the Incredible Years Toddler Parenting Programme with parents of toddlers in disadvantaged (Flying Start) areas of Wales. Child Care Health Dev [Internet]. 2017 Jan 1 [cited 2022 Nov 23];43(1):104–13. Available from: https://onlinelibrary.wiley.com/doi/full/10.1111/cch.12415 pmid:27704590

18. Perrin EC, Sheldrick RC, McMenamy JM, Henson BS, Carter AS. Improving Parenting Skills for Families of Young Children in Pediatric Settings: A Randomized Clinical Trial. JAMA Pediatr [Internet]. 2014 Jan 1 [cited 2022 Nov 23];168(1):16–24. Available from: https://jamanetwork.com/journals/jamapediatrics/fullarticle/1766496 pmid:24190691

19. Bywater T, Berry V, Blower S, Bursnall M, Cox E, Mason-Jones A, et al. A randomized controlled trial of a proportionate universal parenting program delivery model (E-SEE Steps) to enhance child social-emotional wellbeing. PLoS One. 2022 Apr 1;17(4 April). pmid:35377882

20. Hutchings J, Gardner F, Bywater T, Daley D, Whitaker C, Jones K, et al. Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial. BMJ [Internet]. 2007 Mar 29 [cited 2022 Nov 23];334(7595):678. Available from: https://www.bmj.com/content/334/7595/678 pmid:17350966

21. Gardner F, Leijten P, Harris V, Mann J, Hutchings J, Beecham J, et al. Equity effects of parenting interventions for child conduct problems: a pan-European individual participant data meta-analysis. Lancet Psychiatry. 2019 Jun 1;6(6):518–27. pmid:31072801

22. Drugli MB, Larsson B, Fossum S, Mørch WT. Five- to six-year outcome and its prediction for children with ODD/CD treated with parent training. Journal of Child Psychology and Psychiatry [Internet]. 2010 May 1 [cited 2022 Nov 21];51(5):559–66. Available from: https://onlinelibrary.wiley.com/doi/full/10.1111/j.1469-7610.2009.02178.x pmid:20015193

23. Hariton E, Locascio JJ. Randomised controlled trials—the gold standard for effectiveness research. BJOG [Internet]. 2018 Dec 1 [cited 2022 Nov 21];125(13):1716. Available from: /pmc/articles/PMC6235704/

24. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ [Internet]. 2021 Sep 30;n2061. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.n2061 pmid:34593508

25. Stewart-Brown S, Anthony R, Wilson L, Winstanley S, Stallard N, Snooks H, et al. Should randomised controlled trials be the “‘gold standard’” for research on preventive interventions for children? J Child Serv. 2011;6(4):228–36.

26. White H, Sabarwal S. Quasi-Experimental Design and Methods. 2014;(8).

27. Dickerson J, Bird PK, McEachan RRC, Pickett KE, Waiblinger D, Uphoff E, et al. Born in Bradford’s Better Start: An experimental birth cohort study to evaluate the impact of early life interventions. BMC Public Health [Internet]. 2016 Aug 4 [cited 2021 Mar 19];16(1):711. Available from: http://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-016-3318-0

28. Dickerson J, Bridges S, Willan K, Kelly B, Moss RH, Lister J, et al. Born in Bradford’s Better Start (BiBBS) interventional birth cohort study: Interim cohort profile. Wellcome Open Research 2022 7:244 [Internet]. 2022 Oct 3 [cited 2022 Nov 21];7:244. Available from: https://wellcomeopenresearch.org/articles/7-244

29. City of Bradford Council. Bradford Council. 2021 [cited 2022 Dec 19]. 2021 Census: Bradford District. Available from: https://ubd.bradford.gov.uk/about-us/2021-census/

30. GOV.UK. Early years foundation stage profile results: 2018 to 2019 [Internet]. 2019 [cited 2022 Nov 17]. Available from: https://www.gov.uk/government/statistics/early-years-foundation-stage-profile-results-2018-to-2019

31. Department for Education. Early years foundation stage profile 2022 handbook. 2021;

32. GOV.UK. Changes to the early years foundation stage (EYFS) framework [Internet]. 2021 [cited 2021 Nov 23]. Available from: https://www.gov.uk/government/publications/changes-to-the-early-years-foundation-stage-eyfs-framework/changes-to-the-early-years-foundation-stage-eyfs-framework

33. Atkinson AL, Hill LJB, Pettinger KJ, Wright J, Hart AR, Dickerson J, et al. Can holistic school readiness evaluations predict academic achievement and special educational needs status? Evidence from the Early Years Foundation Stage Profile. Learn Instr. 2022 Feb 1;77:101537.

34. Wright B, Mon-Williams M, Kelly B, Williams S, Sims D, Mushtaq F, et al. Original research: Investigating the association between early years foundation stage profile scores and subsequent diagnosis of an autism spectrum disorder: a retrospective study of linked healthcare and education data. BMJ Paediatr Open [Internet]. 2019 Nov 11 [cited 2021 Nov 24];3(1). Available from: /pmc/articles/PMC6863697/

35. Chen H, Cohen P, Chen S. How Big is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies. https://doi.org/101080/03610911003650383. 2010 Apr;39(4):860–4.

36. Benedetto U, Head SJ, Angelini GD, Blackstone EH. Statistical primer: propensity score matching and its alternatives. European Journal of Cardio-Thoracic Surgery [Internet]. 2018 Jun 1 [cited 2021 Oct 12];53(6):1112–7. Available from: https://academic.oup.com/ejcts/article/53/6/1112/4978231 pmid:29684154

37. Barth RP, Guo S, Mccrae JS. Propensity Score Matching Strategies for Evaluating the Success of Child and Family Service Programs. 2008;

38. Williams TC, Bach CC, Matthiesen NB, Henriksen TB, Gagliardi L. Directed acyclic graphs: a tool for causal studies in paediatrics. Pediatr Res. 2018; pmid:29967527

39. Shrier I, Platt RW. Reducing bias through directed acyclic graphs. Vol. 8, BMC Medical Research Methodology. BioMed Central; 2008. p. 1–15.

40. Caliendo M, Kopeinig S. Some practical guidance for the implementation of propensity score matching. J Econ Surv. 2008 Feb;22(1):31–72.

41. Bunten A, Porter L, Gold N, Bogle V. A systematic review of factors influencing nhs health check uptake: Invitation methods, patient characteristics, and the impact of interventions. BMC Public Health. 2020;20(1):1–16.

42. Berry V, Mitchell SB, Blower S, Whittaker K, Wilkinson K, McGilloway S, et al. Barriers and facilitators in the delivery of a proportionate universal parenting program model (E-SEE Steps) in community family services. PLoS One [Internet]. 2022;17(6):e0265946. Available from: pmid:35696375

43. Prady SL, Endacott C, Dickerson J, Bywater TJ, Blower SL. Inequalities in the identification and management of common mental disorders in the perinatal period: An equity focused reanalysis of a systematic review. Vol. 16, PLoS ONE. Public Library of Science; 2021. pmid:33720994

44. Petersen I, Peltola T, Kaski S, Walters KR, Hardoon S. Depression, depressive symptoms and treatments in women who have recently given birth: UK cohort study. BMJ Open [Internet]. 2018 Oct 1 [cited 2021 Aug 2];8(10):e022152. Available from: https://bmjopen.bmj.com/content/8/10/e022152 pmid:30361401

45. Spoth R, Redmond C, Shin C. Modeling factors influencing enrollment in family-focused preventive intervention research. Prevention Science. 2000;1(4):213–25. pmid:11523749

46. Wallby T, Hjern A. Child health care uptake among low‐income and immigrant families in a Swedish county. 2011.

47. Iqbal H, Golombok S. The Generation Game: Parenting and Child Outcomes in Second-Generation South Asian Immigrant Families in Britain: https://doi.org/101177/0022022117736039 [Internet]. 2017 Oct 19 [cited 2021 Sep 3];49(1):25–43. Available from: https://journals.sagepub.com/doi/full/10.1177/0022022117736039?casa_token=w5i6bjtF0lgAAAAA%3AEEEtv5nJJ0TYmgb0dd8KiFMjmkLPTrOw6W2ATJs6qj2XllQo0x5WlkdDIPL_CB_bazgLfhEK-BE

48. Strand S. Ethnic, socio-economic and sex inequalities in educational achievement at age 16: An analysis of the Second Longitudinal Study of Young People in England (LSYPE2). Report for the Commission on Ethnic and Racial Disparities (CRED) [Internet]. 2021; Available from: https://www.gov.uk/government/publications/the-report-of-the-commission-on-race-and-ethnic-disparities-supporting-research/ethnic-socio-economic-and-sex-inequalities-in-educational-achievement-at-age-16-by-professor-steve-strand

49. Tracey L, Bowyer-Crane C, Bonetti S, Nielsen D, Apice KD’, Compton S. The Impact of the COVID-19 Pandemic on Children’s Socio-Emotional Wellbeing and Attainment during the Reception Year. Research Report. Education Endowment Foundation [Internet]. 2022 May [cited 2023 Mar 28]; Available from: www.educationendowmentfoundation.org.uk

50. Fergusson D, Aaron SD, Guyatt G, Hébert P. Post-randomisation exclusions: The intention to treat principle and excluding patients from analysis. Br Med J. 2002;325(7365):652–4. pmid:12242181

51. Gupta S. Intention-to-treat concept: A review. Perspect Clin Res. 2011;2(3):109. pmid:21897887

52. Tripepi G, Chesnaye NC, Dekker FW, Zoccali C, Jager KJ. Intention to treat and per protocol analysis in clinical trials. Nephrology. 2020;25(7):513–7. pmid:32147926

53. Rubin DB. Multiple Imputation after 18+ Years. J Am Stat Assoc. 1996 Jun 1;91(434):473–89.

54. Charles JM, Edwards RT, Bywater T, Hutchings J. Micro-Costing in Public Health Economics: Steps Towards a Standardized Framework, Using the Incredible Years Toddler Parenting Program as a Worked Example. Prevention Science. 2013 Aug 1;14(4):377–89. pmid:23306596

55. Skarda I, Asaria M, Cookson R. LifeSim: A Lifecourse Dynamic Microsimulation Model of the Millennium Birth Cohort in England. Vol. 14, International Journal of Microsimulation. 2021.

56. Dickerson J, Bird PK, Bryant M, Dharni N, Bridges S, Willan K, et al. Integrating research and system-wide practice in public health: Lessons learnt from Better Start Bradford. BMC Public Health [Internet]. 2019 Mar 4 [cited 2022 Nov 21];19(1):1–12. Available from: https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-019-6554-2

57. Pleck JH. Integrating Father Involvement in Parenting Research. https://doi.org/101080/152951922012683365 [Internet]. 2012 Apr [cited 2023 Aug 16];12(2–3):243–53. Available from: https://www.tandfonline.com/doi/abs/10.1080/15295192.2012.683365

58. Macfadyen A, Swallow V, Santacroce S, Lambert H. Involving fathers in research. Journal for Specialists in Pediatric Nursing. 2011 Jul;16(3):216–9. pmid:21702882