Changing trends in ethnicity and academic performance: observational cohort data from a UK medical school

Abstract

Objective

Quantify differential attainment by ethnicity in undergraduate medical assessments and evaluate whether institutional efforts to reduce the attainment gap have had impact.

Design

Observational cohort study.

Setting

A single UK MBBS medical programme.

Participants

Pseudonymised data of adults aged ≥18 years enrolled in one of the UK MBBS medical programmes between 2012 and 2018. Ethnicity was self-declared during enrolment as White, Asian, Black, mixed and other.

Main outcome measure

Module mark (distinction, merit, pass, fail) graded according to a variety of assessments, including single best answer examinations, objective structured clinical examinations and coursework submissions. All modular assessments are graded as a percentage. Logistic regression models were used to calculate relative risk ratio to study the association between ethnicity and attainment gap over a calendar and scholastic year. Models were adjusted for age, gender, social deprivation and scholastic year of study.

Results

3714 student records were included. In the sample, 2134 students (57%) were non-white. The proportion of non-white students increased from 2007 (49%) to 2018 (70%). Mean age was 18 (IQR 18–21) and 56.6% were females. Higher proportion of non-white students 218 (24.8%) were from more deprived backgrounds versus white 76 (14.8%). Compared with non-white, there were no significant differences in the proportion of students failing assessments. However, white students were more likely to achieve merit (relative risk ratio 1.29 (95% CI 1.08 to 1.45)) or distinction (1.69 (95% CI 1.37 to 2.08)). Differences in attainment gap have remained unchanged over time, and for black students, attainment gap grew between their first and final year of study.

Conclusion

A similar proportion (97%) of non-white and white students had a passing score, but attainment gap for higher grades persists over years despite widespread efforts in medical schools to diminish the attainment gap linked to ethnicity. Our findings are from a single institution, thus affecting generalisability.

PM, MAA, BC and JBG are joint first authors.

HL and SL are joint senior authors.

STRENGTHS AND LIMITATIONS OF THIS STUDY

• This study uses a large cohort to assess the deferential attainment across students.

• More than half of the students analysed come from ethnic minority backgrounds.

• We accounted for possible confounders in our analyses but need to acknowledge that there remains likely unmeasured confounding.

• We focused only on undergraduate students in a 5-year MBBS programme, thus affecting generalisability to other programmes.

Introduction

During the pandemic, there has been substantial publicity around ethnicity and inequality. Differences in health outcomes in almost every level of society can be linked to ethnicity, and medical education is no exception.^1–4 A landmark systematic review by Woolf et al, described the magnitude of differential attainment in medical education at undergraduate and postgraduate levels.⁵ The review demonstrated worse performance for non-white students, with a summary estimate (using Cohen’s d of 0.42) indicating a moderate effect size for the difference.⁶ The effects were comparable for computer-marked assessments and viva-based exams.⁵

Many studies have previously explored the reasons for differential attainment by ethnicity. Although there may be related to prior education attainment, age, gender and socioeconomic position (SEP), these factors do not fully explain the observed differences.^7–10

Over the last decade, there have been concerted efforts to reduce differential attainment by ethnicity in healthcare education. Examples include expanding access to medical school for non-white students at both entry into and through the programme. At the entry stage, the changes include Widening Participation programmes, and during the programme, through a process of (1) rethinking and reframing the medical curricula in collaboration with students (eg, decolonising the curriculum programmes) and (2) training in cultural competency and (3) unconscious bias education for faculty and examiners.^11–17

We included data from one of the largest UK medical schools, with a typical year group size of 400 medical students. In Central London, medical schools attract a diverse population, with high proportions of non-white students. We sought to quantify differential attainment in exam performance by ethnicity in our student population and evaluate if this has reduced over the last 10 years. In parallel, we investigated whether differential attainment changed between entry to, or exit from, medical school.

Methods

Data source

The medical degree course is a 5-year programme, which accepts entrance directly after secondary school as well as graduate entry. To be accepted into the Bachelor of Medicine, Bachelor of Surgery (MBBS) programme, applicants must have school-level science qualifications. UK-resident and international students are accepted into the programme. The first year of study is based on preclinical sciences and is assessed only with computer-marked written exams. Years 2–5 are based on clinically focused content and are assessed through a combination of computer-assessed exams (40% contribution), Objective structured clinical examinations (OSCEs) (40% contribution) and projects (20% contribution). Assessments are graded as a percentage, with grades 70% and above classified as distinction, 60%–69% as merit, 50%–59% as pass and below 50% as fail.

Study population

Pseudonymised data were extracted from the student records system for individuals who were enrolled on one of the UK MBBS programme. Data were available from September 2012 to July 2019, irrespective of year of study for individual students. These years corresponded to students enrolled from 2007 (who would only contribute their final year of examination data) to 2018 (who would only contribute their first year of examination data). If students intercalated during their programme, their results linked only to the MBBS programme were included.

We did not include results from 2020 due to the COVID-19 pandemic.

Self-declared ethnicity status was defined by the 2011 UK census five main ethnic groups classification.¹⁸ That is Asian (Bangladeshi, Indian, Pakistani, Chinese and other Asian); black (Black African, Black Caribbean and other Black); mixed (white and black Caribbean/white and black African/white and Asian/any other mixed or multiple ethnic backgrounds); other (Arab/any other ethnic group) and white (white, white Irish and other white).

Outcomes

The primary outcome of our analyses was exam grade boundaries (categorical variable: distinction, merit, pass, fail). A secondary outcome explored was the summary year module marks (continuous variable). Module marks relate to a variety of assessments, and include single best answer examinations, OSCEs and coursework submissions. All these modular assessments are graded as a percentage. Data were not separated by type of assessment within each year as this information was not accessible to our research group. We included only results from first attempts at any assessment (ie, not resit examinations).

Predictors

The first predictor we investigated was a calendar year of study, to investigate whether the attainment gap has changed over time. The second predictor was a scholastic year in a study for individual students, to evaluate whether attainment gaps change during their medical course.

Covariates

Confounders included in our models were age on course entry, gender and SEP. The SEP information was available from 2013 onwards. SEP was defined by entry via the Widening Participation programme. Students were listed as being in the Widening Participation group if, at the time of application to the course, they: (1) lived in geographic regions with historically low participation in higher education; (2) lived in regions described as less prosperous in the Acorn geodemographic segmentation system (https://acorn.caci.co.uk/) and (3) had been a participant in a locally organised university outreach programme. One of the main aims of the Widening Participation programme in England is to encourage students from different socioeconomic backgrounds to apply for higher education.¹⁹

Statistical analyses

Descriptive comparisons by ethnicity were tabulated with tests for statistically significant differences (χ² for categorical variables; Mann-Whitney U for non-parametric and continuous variables).^{20 21}

Multinomial logistic regression²² was performed to estimate the association between ethnicity and grade boundary attainment across all assessments, to provide context for our main analyses. Linear regression²³ was used to estimate the association between ethnicity and examination marks. We used linear regression to describe the change in attainment gap by calendar year and scholastic year. Models incorporate unique student number as a clustering variable to account for multiple results per student. Models were adjusted for age, gender, widening participation and scholastic year of study. Results were presented graphically and as marginal means (with 95% CIs) from the regression outputs.

Sensitivity analyses explored the ethnicity subcategories. Our analyses used a complete-case approach. Missing data categories are described in the results tables. No imputation for missing data was performed as the assumptions of missingness could not be verified. Analyses were performed Stata V.17.0 (StataCorp).

Patient and public involvement

There was no patient or public involvement in the design, conduction, reporting and dissemination plans of this study.

Results

A total of 3825 students were enrolled in the 5-year MBBS programme from 2007 to 2018 and undertook assessments between September 2012 and July 2019. Only 3714 students had known ethnicity information and these students were included in further analyses: this comprising 19 247 individual assessment outcomes.

Baseline characteristics are described by ethnicity in table 1, with a breakdown by individual subgroups in online supplemental table 1. A total of 2134/3714 students (57%) were non-white. The proportion of non-white students increased over time, from 49% of those enrolled in 2007 to 70% in 2018, with the trend more apparent in recent years (see figure 1). Median age across the cohort was 19 (IQR 18–21). A smaller proportion of non-white students 67 (3.1%) were in older age groups compared with white students 143 (9.1%). A total of 2097 (56.6%) of all students were females, with no differences across ethnic groups. More non-white 218 (24.8%) than white students 76 (14.8%) were enrolled via the Widening Participation programme.

Figure 1. Percentage of white and non-white students enrolment. Percentage of the white and non-White students enrolled in the 5-year MBBS programme from 2007 to 2018. MBBS, Bachelor of Medicine, Bachelor of Surgery.

Table 1

Baseline characteristics

	Total	Non-white	White	P value
	N=3714	N=2134	N=1580
Age: median (IQR)	19 (18–21)	19 (18–20)	19 (18–22)	<0.001
Age band: n (%)				<0.001
17–20	2711 (73.0)	1683 (78.9)	1028 (65.1)
21–24	793 (21.4)	384 (18.0)	409 (25.9)
25 and over	210 (5.7)	67 (3.1)	143 (9.1)
Female: n (%)				0.13
No	1617 (43.5)	952 (44.6)	665 (42.1)
Yes	2097 (56.5)	1182 (55.4)	915 (57.9)
Widening participation: n (%)				<0.001
No	1097 (78.9)	660 (75.2)	437 (85.2)
Yes	294 (21.1)	218 (24.8)	76 (14.8)
Missing widening participation information: n	2323	1256	1067

Baseline characteristics of students enrolled in the 5-year MBBS programme from 2007 to 2018 and undertook assessments between September 2012 to July 2019.

MBBS, Bachelor of Medicine, Bachelor of Surgery .

The regression analyses to estimate the association between ethnicity and assessment outcomes (see table 2) confirmed the presence of an attainment gap, which was driven by differences in students achieving the highest grades. There was no difference by ethnicity in proportion of students who were unsuccessful in their assessment attempts, neither at White vs non-White, or when comparing by ethnicity subgroups (see online supplemental table 2).

Table 2

The associations between ethnicity and assessment results

	Total	Non-white	White	P value
	N=19 247	N=11 334	N=7913
Module Result: n (%)				0.32
Fail	523 (3)	319 (3)	204 (3)
Pass	18 724 (97)	11 015 (97)	7709 (97)
Grade boundaries: n (%)				<0.001
Fail	445 (2.8)	277 (2.9)	168 (2.5)
Pass	2368 (14.7)	1577 (16.8)	791 (11.8)
Merit	5639 (35.0)	3428 (36.5)	2211 (33.0)
Distinction	7647 (47.5)	4114 (43.8)	3533 (52.7)
aRRR (95% CI)
Fail			0.93 (0.59 to 1.45)	0.75
Pass			Referent group
Merit			1.29 (1.08 to 1.54)	<0.001
Distinction			1.69 (1.37 to 2.08)	<0.001
Module mark: median (IQR)	69 (62–75)	68 (61–74)	70 (64–76)	<0.001
Linear regression coefficient (95% CI)			1.91 (1.19 to 2.62)	<0.001

RRR and coefficients represent estimates for White compared with non-white group. We used linear regression for this analysis. Models incorporate unique student numbers as a clustering variable to account for multiple results per student. The number of students in the grading boundaries (fail, pass, merit and distention), which is (16 099), is different from the total number of student with module results (19 247) as some teaching modules are marked with only a pass/fail grade boundary. The aRRR included adjustment for age, gender, widening participation and scholastic year.

aRRR, adjusted relative risk ratio; CI, Confidence interval.

Compared with non-white students, white students were more likely to achieve a merit (adjusted relative risk (aRR) 1.29 (95% CI 1.08 to 1.45) or distinction (aRR 1.69 (95% CI 1.37 to 2.08)). The associations were determined by differences for the Asian and Black ethnic subgroups, with mixed and other ethnic groups not performing significantly differently from the white group. Non-white student scores were 1.9% lower in assessments compared with white students. This equates to a standardised mean difference of 0.19 (95% CI 0.12 to 0.26).

The analysis of the changing attainment gap over calendar year is shown in figure 2. Over the time studied, there is evidence of average mark improvement in all groups. Yet, across every calendar year there were significantly lower marks awarded to non-white students. The absolute difference between the groups grew over time. There was no evidence that the pattern of change over time varied within the ethnicity subgroups.

Figure 2. Attainment gap over the calendar year for students with different ethnic backgrounds. Attainment gap change over the calendar year for students enrolled in the 5-year MBBS programme from 2007 to 2018 and undertook assessments between September 2012 and July 2019. We used linear regression for this analysis. Models incorporate unique student numbers as a clustering variable to account for multiple results per student. Models were adjusted for age, gender, widening participation and scholastic year of study. Results are presented as mean in circles and 95% CI in vertical lines. MBBS, Bachelor of Medicine, Bachelor of Surgery.

The analysis of performance within scholastic year of study in figure 3 shows that the attainment gap is visible in year 1 of study and persists throughout the course. The breakdown by ethnicity subgroup shows a pattern that is different for the black student group, for whom the attainment gap worsens during the 5 years of study. The differential attainment increases across the years of study (but interpretation of the exact cause is somehow speculative, as explanations of attainment differences were not explored in this study).

Figure 3. Attainment over the scholastic year for students with different ethnic backgrounds. Attainments over the scholastic year for students enrolled in the 5-year MBBS programme from 2007 to 2018 and undertook assessments between September 2012 and July 2019. We used linear regression for this analysis. Models incorporate unique student number as a clustering variable to account for multiple results per student. Models were adjusted for age, gender, widening participation and scholastic year of study. Results are presented as mean in circles and 95% CI in vertical lines. MBBS, Bachelor of Medicine, Bachelor of Surgery .

Discussion

The attainment gap between white and non-white students has not changed over time, despite institutional efforts to reduce inequality in assessment outcomes associated with ethnicity using data from a single institution. In addition, we have demonstrated that during their medical school studies, the gap in attainment for students of black ethnicity worsens.

The possible explanations for the attainment gap itself are subject of many previous publications^8–10 and were not the focus of this study. Our goal was to evaluate temporal changes over calendar year and by scholastic year of study.

The lack of any discernible reduction in attainment gap over time is cause for concern. There are several possible explanations for the failure of the current strategies.

First, we need to acknowledge that the efforts to reduce attainment gaps started prior to 2012(our data span 2012–2019). It is notably that the attainment gap we have described is lower than what has been reported previously.⁵ In our analysis, the standardised mean difference was 0.19 (95% CI 0.12 to 0.26) compared with the pooled standardised mean difference of 0.42 (0.35–0.49) from Woolf et al, which used data from 16 reports of UK undergraduate medical degree courses spanning 1970–2009.⁵ We cannot rule out the historical attainment gap being larger in magnitude and then narrowing prior to data available for this study. However, it is clear that in the last decade there has been no further substantive reducing in the gradient with non-white students scoring on average 2% lower, and around 40% less likely to be awarded a distinction.

Second, it is possible that the current strategies are not sufficiently effective to have rea-world impact. Unconscious bias training has been an important role in addressing inequality in education and was rolled out to all staff between 2014 and 2016. However, a large meta-analysis, of interventions to reduce implicit bias, demonstrated only weak effects, with mainly short-term changes being seen.²⁴ A further systematic review focusing specifically on interventions to reduce implicit prejudice reached similar conclusions.²⁵

The other major focus of our analysis explored the change in the attainment gap during the medical degree course. A single previous study has reported on risk factors for poor performance during a medical degree course. Using an adjusted regression model it identified non-white ethnicity as a significant factor and highlighted that the effects were greatest during clinical assessments such as OSCEs.⁹

As students move into more senior years during their medical studies the methods of assessment change, with a shift from computer marked assignments to projects and OSCEs. The nature of these assessments, for which markers are not blinded to student ethnicity, increases the potential for unconscious bias to influence grades, although not all studies indicate the same.²⁶ In addition, during clinical years interaction (both nature and quality) with teaching staff may differ. Teaching becomes more dependent on clinical placements and there is a greater emphasis on small-group teaching. Experiences of discrimination that may disadvantage non-white students, whether conscious or unconscious, may be more pronounced in clinical years.²⁷

Limitations

Our research has some important limitations. Our study is from a single institution in London and our focus has been only on undergraduate students in a 5-year MBBS programme. Also, as attainment gap causes are complicated and multifactorial, thus, our findings are not necessary generalisable to other universities or healthcare programmes. Our adjustment for confounders was limited, and further, the information on SEP was a single metric, Widening Participation status, and was missing for some of the year groups. It is important to acknowledge that widening participation is an imperfect surrogate for deprivation. Unfortunately, we had no other metric available.

We were unable to control for prior attainment (secondary school grades), although this may have been of limited value because medical school entry requires uniformly high A-level outcomes. Finally, we acknowledge that the categorisation of students by ethnic groups into broad categories may mask important differences in people from ethnic minority communities.

Conclusions

We present robust quantitative evidence of persistent attainment gaps that are associated with ethnicity. The root causes of the attainment gap between White and non-white students are complex and multifactorial. Clearly not all the solutions will lie within the educational system. Strategies to actively address differential attainment are urgently needed in collaboration with all stakeholders including students and patients. Structural approaches such as anonymous marking or written submissions, video recording and monitoring of OSCEs, and positive discrimination strategies should be explored. In addition, further qualitative research has been conducted in parallel to this study, which sheds further detailed light on the possible causes of the attainment gap from the perspective of students and medical faculty staff, and possible solutions.

Data availability statement

Data are available upon reasonable request. Data used in this study were collected from students records and are under the control of the data holders at the medical school. Only aggregated data are available upon reasonable request. Personal data cannot be shared due to confidentiality reasons. All data relevant to the study are included in the article. All figures and tables included in this article are original.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The study was reviewed by a local research ethics committee, and as pseudonymised data were being used, no formal ethics approval was required.

Footnote

Twitter @No, @bennyleeshuang

Contributors JBG, PM, BC, TM, JT, HL and SL. involved in the design. JBG, MAA, SN and MDR analysed the data. All authors interpreted the results. JBG, PM, MAA and BC wrote the report with contributions from all other authors. PM, MAA, BC and JBG are joint first authors. HL and SL are joint senior authors. JBG, SN, MAA and BC accessed and verified the underlying data. MAA and JBG had final responsibility for the decision to submit for publication. JBG accepts full responsibility for the finished work and the conduct of the study, had access to the data, and controlled the decision to publish. JBG attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding This work was supported by the King’s Together Multi and Interdisciplinary Research Scheme (Wellcome Trust Institutional Strategic Support Fund) grant reference (204823/Z/16/Z). All funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. JBG affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Isik U, Wouters A, Croiset G, et al. "What kind of support do I need to be successful as an ethnic minority medical student?" A qualitative study. BMC Med Educ 2021; 21: 6. doi:10.1186/s12909-020-02423-8 http://www.ncbi.nlm.nih.gov/pubmed/33402191

² Leyerzapf H, Abma TA, Steenwijk RR, et al. Standing out and moving up: performance appraisal of cultural minority physicians. Adv Health Sci Educ Theory Pract 2015; 20: 995–1010. doi:10.1007/s10459-014-9577-6 http://www.ncbi.nlm.nih.gov/pubmed/25549932

³ Pan D, Martin CA, Nazareth J, et al. Ethnic disparities in COVID-19: increased risk of infection or severe disease? The Lancet 2021; 398: 389–90. doi:10.1016/S0140-6736(21)01428-8

⁴ Woolf K. Differential attainment in medical education and training. BMJ 2020; 368: m339. doi:10.1136/bmj.m339 http://www.ncbi.nlm.nih.gov/pubmed/32047006

⁵ Woolf K, Potts HWW, McManus IC. Ethnicity and academic performance in UK trained doctors and medical students: systematic review and meta-analysis. BMJ 2011; 342: d901. doi:10.1136/bmj.d901 http://www.ncbi.nlm.nih.gov/pubmed/21385802

⁶ Larner AJ. Effect size (Cohen's D) of cognitive screening instruments examined in pragmatic diagnostic accuracy studies. Dement Geriatr Cogn Dis Extra 2014; 4: 236–41. doi:10.1159/000363735 http://www.ncbi.nlm.nih.gov/pubmed/25177332

⁷ Shah R, Ahluwalia S. The challenges of understanding differential attainment in postgraduate medical education. Br J Gen Pract 2019; 69: 426–7. doi:10.3399/bjgp19X705161 http://www.ncbi.nlm.nih.gov/pubmed/31467003

⁸ Mountford-Zimdars A, Sabri D, Moore J, et al. Causes of differences in student outcomes (HEFCE) 2015.

⁹ Yates J, James D. Risk factors for poor performance on the undergraduate medical course: cohort study at Nottingham university. Med Educ 2007; 41: 65–73. doi:10.1111/j.1365-2929.2006.02648.x http://www.ncbi.nlm.nih.gov/pubmed/17209894

¹⁰ Woolf K, McManus IC, Potts HWW, et al. The mediators of minority ethnic underperformance in final medical school examinations. Br J Educ Psychol 2013; 83: 135–59. doi:10.1111/j.2044-8279.2011.02060.x http://www.ncbi.nlm.nih.gov/pubmed/23369179

¹¹ Williamson S, Foley M. Unconscious Bias Training: The ‘Silver Bullet’ for Gender Equity? Australian Journal of Public Administration 2018; 77: 355–9. doi:10.1111/1467-8500.12313

¹² Stevenson J. An Exploration of the Link between Minority Ethnic and White Students’ Degree Attainment and Views of Their Future ‘Possible Selves’. Higher Education Studies 2012; 2: 103–13. doi:10.5539/hes.v2n4p103

¹³ Ali K, McColl E, Tredwin C, et al. Addressing racial inequalities in dental education: decolonising the dental curricula. Br Dent J 2021; 230: 165–9. doi:10.1038/s41415-020-2598-z http://www.ncbi.nlm.nih.gov/pubmed/33574542

¹⁴ Linton S. Taking the difference out of attainment. BMJ 2020; 368: m438. doi:10.1136/bmj.m438 http://www.ncbi.nlm.nih.gov/pubmed/32051115

¹⁵ Patterson R, Price J. Widening participation in medicine: what, why and how? MedEdPublish 2017; 6: 184. doi:10.15694/mep.2017.000184

¹⁶ Baugh RF, Baugh AD. Cultural influences and the objective structured clinical examination. Int J Med Educ 2021; 12: 22–4. doi:10.5116/ijme.5ff9.b817 http://www.ncbi.nlm.nih.gov/pubmed/33507878

¹⁷ White-Davis T, Edgoose J, Brown Speights JS, et al. Addressing racism in medical education an interactive training module.

¹⁸ Ethnicity-facts-figures.service.gov.uk. List of ethnic groups, 2022. Available: https://www.ethnicity-facts-figures.service.gov.uk/style-guide/ethnic-groups

¹⁹ Chowdry H, Crawford C, Dearden L, et al. Widening participation in higher education: analysis using linked administrative data. J R Stat Soc Ser A Stat Soc 2013; 176: 431–57. doi:10.1111/j.1467-985X.2012.01043.x

²⁰ Hart A. Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ 2001; 323: 391–3. doi:10.1136/bmj.323.7309.391 http://www.ncbi.nlm.nih.gov/pubmed/11509435

²¹ McHugh ML. The chi-square test of independence. Biochem Med 2013; 23: 143–9. doi:10.11613/BM.2013.018 http://www.ncbi.nlm.nih.gov/pubmed/23894860

²² de Jong VMT, Eijkemans MJC, van Calster B, et al. Sample size considerations and predictive performance of multinomial logistic prediction models. Stat Med 2019; 38: 1601–19. doi:10.1002/sim.8063 http://www.ncbi.nlm.nih.gov/pubmed/30614028

²³ Schneider A, Hommel G, Blettner M. Linear regression analysis: part 14 of a series on evaluation of scientific publications. Dtsch Arztebl Int 2010; 107: 776–82. doi:10.3238/arztebl.2010.0776 http://www.ncbi.nlm.nih.gov/pubmed/21116397

²⁴ Forscher PS, Lai CK, Axt JR, et al. A meta-analysis of procedures to change implicit measures. J Pers Soc Psychol 2019; 117: 522–59. doi:10.1037/pspa0000160 http://www.ncbi.nlm.nih.gov/pubmed/31192631

²⁵ FitzGerald C, Martin A, Berner D, et al. Interventions designed to reduce implicit prejudices and implicit stereotypes in real world contexts: a systematic review. BMC Psychol 2019; 7: 29. doi:10.1186/s40359-019-0299-7 http://www.ncbi.nlm.nih.gov/pubmed/31097028

²⁶ Wass V, Roberts C, Hoogenboom R, et al. Effect of ethnicity on performance in a final objective structured clinical examination: qualitative and quantitative study. BMJ 2003; 326: 800–3. doi:10.1136/bmj.326.7393.800 http://www.ncbi.nlm.nih.gov/pubmed/12689978

²⁷ Claridge H, Stone K, Ussher M. The ethnicity attainment gap among medical and biomedical science students: a qualitative study. BMC Med Educ 2018; 18: 325. doi:10.1186/s12909-018-1426-5 http://www.ncbi.nlm.nih.gov/pubmed/30594175