Pan-Canadian study of psychiatric care (PCPC): protocol for a mixed-methods study

Abstract

Introduction

The Canadian population has poor and inequitable access to psychiatric care despite a steady per-capita supply of psychiatrists in most provinces. There is some quantitative evidence that practice style and characteristics vary substantially among psychiatrists. However, how this compares across jurisdictions and implications for workforce planning require further study. A qualitative exploration of psychiatrists’ preferences for practice style and the practice choices that result is also lacking. The goal of this study is to inform psychiatrist workforce planning to improve access to psychiatric care by: (1) developing and evaluating comparable indicators of supply of psychiatric care across provinces, (2) analysing variations and changes in the characteristics of the psychiatrist workforce, including demographics and practice style and (3) studying psychiatrist practice choices and intentions, and the factors that lead to these choices.

Methods and analysis

A cross-provincial mixed-methods study will be conducted in the Canadian provinces of British Columbia, Manitoba, Ontario and Nova Scotia. We will analyse linked-health administrative data within three of the four provinces to develop comparable indicators of supply and characterise psychiatric services at the regional level within provinces. We will use latent profile analysis to estimate the probability that a psychiatrist is in a particular practice style and map the geographical distribution of psychiatrist practices overlayed with measures of need for psychiatric care. We will also conduct in-depth, semistructured qualitative interviews with psychiatrists in each province to explore their preferences and practice choices and to inform workforce planning.

Ethics and dissemination

This study was approved by Ontario Tech University Research Ethics Board (16637 and 16795) and institutions affiliated with the study team. We built a team comprising experienced researchers, psychiatrists, medical educators and policymakers in mental health services and workforce planning to disseminate knowledge that will support effective human resource policies to improve access to psychiatric care in Canada.

Correspondence to David Rudoler; [email protected]

STRENGTHS AND LIMITATIONS OF THIS STUDY

• This study will use population-level health administrative data from multiple provinces from 2012/2013 to 2021/2022 to analyse variations and changes in the supply of psychiatric care and generate comparable indicators of the supply of psychiatric care across provinces.

• We will compare changes in the supply of psychiatric care and the need for psychiatric care before and after the COVID-19 pandemic across provinces.

• We will conduct semistructured interviews with a large sample of psychiatrists from various practice backgrounds across provinces to understand the factors that shape their practice choices and preferences.

• The study will analyse administrative data from three of the four provinces as Nova Scotia does not have comparable data; however, the qualitative data gathered from Nova Scotia will augment our findings.

• Our study will measure trends over time of the treated prevalence for mental and substance use disorders as a proxy for need; however, treated prevalence is underestimated as it is limited to those who accessed publicly funded care.

Introduction

Unmet psychiatric need in the Canadian population

The global socioeconomic burden of mental illness is estimated to be in line with physiological diseases like cardiovascular disease, and access to mental health services, particularly during a crisis, is a challenge faced by people around the world.^{1 2} About 5 million Canadians access mental health services each year.³ The rates of outpatient visits have been increasing for psychiatric care.⁴ While psychiatrists are only a part of the mental health system—in fact most mental health services are provided by other health professionals—most other mental health professionals are not covered by provincial health insurance plans. As a result, people relying on publicly funded services have few options when seeking specialised mental healthcare. Access to psychiatrists has been challenging for many, but particularly for those with the least means and the greatest need.^{5 6} A Canadian multiprovince report showed that 25%–50% of youth have their first mental health treatment contact in the emergency department rather than in the community.⁷ In Ontario, about 70% of people do not see a psychiatrist in the 30 days following a psychiatric hospitalisation discharge,^{5 8} and about 60% do not see one within 6 months of a suicide attempt.⁹ Poor access to psychiatric care could lead to relapse, rehospitalisation, avoidable costs to the healthcare system and death in the case of low post-suicide follow-up.^10–12

The supply of psychiatric care is not keeping up with increasing patient demand.^{13 14} Effective health human resource policy is an important part of ensuring that patients have access to psychiatric care when it is needed. One potential solution is to recruit more psychiatrists; it has been argued that psychiatrist shortages have caused shortfalls in psychiatric care,^{15 16} though per-capita supply of psychiatric care has remained constant in most provinces.¹⁷

Factors that may affect access to psychiatrists

Practice characteristics and practice style

One explanation for shortfalls in supply is variation in practice characteristics and practice style.^{4 5 9 18} Here, we use practice style as shorthand for practice volume, practice intensity (ie, annual psychiatric services per patient), practice location, practice specialisation and patient case-mix. In a recent study, our team used administrative data to demonstrate that full-time psychiatrists in Ontario fall into three distinct practice styles: (1) urban practices providing high intensity care to small numbers of patients; (2) urban practices providing low intensity care to high numbers of patients and (3) rural practices providing low intensity care to high numbers of patients.⁴ We also found that psychiatrists in practice style (1) tended to see relatively healthier and wealthier patients. Moreover, members of our team found that about a quarter of psychiatrists in Ontario see fewer than 100 patients per year, and on average, see them 16 times per year; such low volume practices limit the availability of psychiatrists to a population.⁴ High intensity care may be beneficial following a crisis event like a suicide attempt, but we have found that these services are less likely to be accessed by people with high needs.^{5 6} In Ontario, the psychiatric workforce is ageing and is increasingly concentrated in urban communities,⁴ which also presents challenges for ensuring equitable access to publicly insured mental health and addiction services.

The analyses that members of our team have conducted in Ontario are the only population-level research that explores the supply of publicly insured mental health services in Canada, and there is genuine uncertainty about how psychiatric care is supplied in other provinces. There are several factors that could contribute to variation in psychiatric care across provinces and make it difficult to generalise findings from Ontario. Regional variation may be more pronounced in provinces with fewer large urban centres; providers tend to be concentrated in these areas leading to greater maldistribution in less urbanised provinces. In response to maldistribution, other provinces have implemented health human resource and recruitment policies (eg, special licensing from the Royal College and provincial colleges to practice in a specific geographic area) that may have impacted the supply of psychiatric care in ways that have not been observed in Ontario. In addition, fee schedules for psychiatric care vary across provinces and may have important impacts on psychiatrist practice styles that may, in turn, impact access to care.^{19 20}

Practice style may also be affected by the environment in which physicians work. For instance, differences in the physician fee schedules or payment models (eg, alternative payment plans) may produce different incentives and, ultimately, different practice patterns. Opportunities for different kinds of psychiatrist practice, including the density of psychiatrists practising within a given geographical area may present more opportunities for practice style selection.⁵ More information is needed to understand how practice opportunities and political environments shape decision-making, and whether differences in policy environments across Canada influence practice patterns and practice location.

Furthermore, policy discourse concerning the supply of psychiatric care has been largely absent outside of the Ontario context where it has garnered some attention.²¹ Local analysis of the supply of psychiatric care may inform wider policy debate. Finally, the COVID-19 pandemic has had an impact on how psychiatric care is delivered and will likely have effects that persist. Thus, the pandemic motivates further analysis of psychiatric care in all regions to understand its impact, including the benefits and unintended consequences of increased remote and online access.

Personal characteristics

There is evidence that a variety of factors related to personal characteristics and training experiences are related to the choice of psychiatry as a specialty among medical students.^22–28 Our quantitative work has shown that personal factors (eg, age, sex/gender and location of training) and structural factors (eg, community size and billing schedules) contribute to variation in practice style and access to psychiatric care.⁴ A growing body of evidence also suggests that sex/gender influences practice patterns (eg, hours worked, practice setting and patient population).^{29 30}

To design interventions to improve the supply of and access to psychiatric care, decision-makers need to understand the factors and preferences that shape practice style selection. However, little research exists on factors shaping choice of practice style within psychiatry. There is an opportunity to use qualitative methods to explore variation in practice style and why psychiatrists choose to practice in different ways and in different settings. A qualitative exploration focusing on these and other emerging factors can deepen our understanding of how psychiatrists’ training, work and personal experiences and local context inform beliefs, attitudes and patterns of practice. The range and variety of practice styles raise questions about the appropriate role of the specialty within our healthcare system. It will be important to determine what psychiatrists view as their role, and whether these perspectives vary across different policy environments, regions or by physician characteristics (years in practice, location of training).

Need for indicators to inform the psychiatric workforce

Thus far, efforts to harmonise and study pan-Canadian indicators of mental healthcare have been limited; while they establish clear and growing access challenges, they do not inform patterns of provision of psychiatric care within or across provinces.^{7 31 32} Meanwhile, several government agencies have called for improved access to mental health services. For instance, a report commissioned by the government of Manitoba highlights the challenges of mental health service delivery and the importance of developing a plan to improve access.³³ In Nova Scotia, the Department of Health and Wellness Blueprint for Mental Health and Addictions³⁴ calls for intervening and treating earlier, shorter waits and better care. Our study will help address these calls for improved access by generating indicators that will inform health human resource planning.

Goal and objectives

The overarching goal of this study is to inform psychiatrist workforce planning by generating new information about the supply of publicly insured mental health services and psychiatrist practice style in Canada. Specific objectives are to:

• Develop and evaluate comparable indicators of supply of psychiatric care across provinces.

• Analyse variation and changes in the characteristics of the psychiatrist workforce, including demographics and practice style.

• Understand psychiatrist preferences for practice style, and their perspectives on factors that lead to practice style selection.

Methods and analysis

We will employ a cross-provincial, concurrent mixed-methods design^{35 36} integrating quantitative and qualitative data from four provinces: British Columbia (BC), Manitoba (MB), Ontario (ON) and Nova Scotia (NS). These provinces were selected to maximise geographic diversity, population density and feasibility of cross-provincial comparison. We will investigate Objective 1 and Objective 2 using quantitative methods; data for these objectives consist of population-level linked administrative data available in three of the four provinces (BC, MB and ON) as these provinces have comparable data on mental health services. Objective 1 will describe the supply of psychiatric care; it will inform us on how specialised mental health services are distributed within and across provinces. Objective 2 will develop harmonised measures to investigate variation in practice style. We will investigate Objective 3 by conducting qualitative interviews with psychiatrists in all four provinces who are in various stages of career and practice settings. The interviews will further explore the unique practice styles and increase our understanding of the factors that shape practice style selection. While NS does not have comparable administrative data on mental health services, the province will be included in the qualitative arm of this study. Quantitative and qualitative data collection began in January 2023. We anticipate all data analysis and interviews to be completed by July 2024.

Linked administrative data

We will use linked databases developed and housed separately in each province: Population Data BC (PopDataBC), Manitoba Centre for Health Policy (MCHP) and ICES (formerly the Institute for Clinical Evaluative Sciences) in Ontario. Within each province, deidentified data with unique patient and physician identifiers will enable us to connect records across data sets and over time. It is neither possible nor necessary to combine record-level data across provinces, but parallel analysis using comparable variable definitions and identical analytic procedures will permit us to explore patterns across the three provinces. The following types of databases will be accessed in each province:

Physician registry files: This includes data on physician demographics, specialty, year of graduation, practice location and payment methods.

Physician payment information: This includes data on all fee-for-service and shadow-billed medical services claims. It describes services used and includes a patient diagnosis code for each encounter.

Patient registry files for provincial insurer: This includes a record for all provincial residents who are eligible to receive publicly funded healthcare services, including their demographic information.

Hospital separation files: This includes records of all inpatient discharges and in-hospital deaths for provincial residents.

Emergency department contact information: This includes records of patient contacts in emergency departments.

Vital statistics (deaths): This includes records of the date and cause of death.

It will also be possible to obtain some information on prescription drugs dispensed, including coverage under plans for people with low income or lacking coverage for psychiatric medicines, but completeness of prescription data varies by province.

We will request data covering a decade of healthcare delivery, starting in 2012/2013 through to 2021/2022, or the most recent data available. A look-back period for 2007/08 will be used for some indicators.

Sex-gender-based analysis considerations

The quantitative data sources we will use capture binary variables labelled ‘sex’, but these are not biological measures. This information is self-reported by patients and physicians at the time of registration, and whether legal sex, sex assigned at birth or gender is being reported cannot be confirmed. Recognising this lack of specificity in our measures, and that socially constructed gender roles most plausibly influence practice and healthcare seeking behaviour, we will describe these variables as sex/gender within our methods. We recognise that the cis-normative conflation of sex and gender is a limitation of this analysis.

Despite limitations of quantitative data, we have considered gender in our research design in several ways. In quantitative analysis, we will examine accesses and practice characteristics stratified by sex/gender. In qualitative analysis, we will aim to sample psychiatrists of different gender identities and explore how roles and gender identities of psychiatrists shape practice intentions and choices, as well as the values and preferences that underpin them.

Objective 1. Develop and evaluate comparable indicators of supply of psychiatric care across provinces

We will develop comparable measures of supply of psychiatric care using the administrative data that can be implemented across provinces. Creating measures requires data fields from the provincial databases as inputs. In cases where measures must rely on non-comparable fields, we will focus on differences between psychiatrists within, but not between, provinces. This is a limitation of using different administrative data sources but does not undermine our primary objective of describing the supply of psychiatric care across Canada to inform health workforce policy within each province. These measures have been developed in Ontario in previous work by members of our research team.⁴ We will investigate psychiatric care supplied to adults between 18 and 105 years of age. These indicators will be reported as a comparative study of the supply of outpatient psychiatric care in three study provinces. Our descriptive analysis of supply will include the following indicators:

Population supply indicators

• Service Propensity (probability of one or more visits with a psychiatrist): We will capture all unique individuals in each year who received at least one service from a psychiatrist. This will be divided by the general population eligible for provincial health insurance and by the number of adults who have received mental health and substance use services (MHSU) in the emergency department, hospital or during a physician visit in each province.

• Psychiatric Consultation (probability of one or two outpatient visits with a psychiatrist): It is possible for psychiatrists to offer consultations over one or two visits without providing ongoing psychiatric care. As a proxy for the probability of psychiatric consultation, we will capture all unique individuals in each year who received one or two outpatient services from a psychiatrist. This will be divided by the general population eligible for provincial health insurance and by the number of adults who have received MHSU services in each province.

• Ongoing Psychiatric Care (probability of three or more outpatient visits with a psychiatrist): As a proxy for the probability of ongoing psychiatric care, we will identify all unique individuals in each year with three or more outpatient visits to the same psychiatrist. This will be divided by the general population eligible for provincial health insurance and by the number of adults who have received MHSU services in each province.

• Service Intensity (number of visits with a psychiatrist per patient): We will capture the number of outpatient visits to a psychiatrist. This will be divided by the general population eligible for provincial health insurance and by the number of adults who have received MHSU services¹ in each province.

Within each province, all indicators will be stratified by metropolitan (census metropolitan areas and census agglomerations) versus non-metropolitan geographic areas, neighbourhood income quintile, low-income drug programme (where available), age and sex/gender. We will also describe supply at the regional level within provinces (16 Health Service Delivery Areas in BC, five health regions in MB and 34 public health units in ON). We will also assess the impact of the COVID-19 pandemic on the supply of psychiatric care by comparing measures of supply before and after COVID-19 emerged in Canada (March 2020). We will stratify this analysis by subpopulation to see if there have been differential effects on subpopulations. We will present our results using descriptive statistics and visualisations. Inferential statistics and standardised differences will be used to compare all indicators across strata.

Objective 2. Analyse variation and changes in the characteristics of the psychiatrist workforce, including demographics and practice style

We will develop measures of psychiatrist supply and practice style within the administrative billings data that can be implemented in comparable ways across provinces.

Psychiatrist personal and practice characteristics

We will describe personal and practice characteristics of clinically active psychiatrists between 2012/2013 and 2021/2022 in three study provinces. Personal characteristics include age, sex/gender, years since graduation and full-time status (>30th percentile in billings). Practice characteristics include total number of inpatients, outpatients, outpatient visit frequencies, number of new outpatients, psychotherapy visits, focused psychotherapy practice,¹⁸ mental health versus addiction visits, number of electroconvulsive therapy sessions, number of group therapy sessions, number of home visits and virtual care. We will plot mean and median changes in these characteristics over the 10 year study period and compare distributional changes within provinces using relative distributions.^37–39 The relative distribution method will allow us to account for changes in practice over time that may not be reflected in measures of central tendency.

Relationship between psychiatrist supply and patient need

We will describe per-capita supply of clinically active psychiatrists between 2012/2013 and 2021/2022 by dividing the number of full-time psychiatrists (>30th percentile billings) by residents in each province and subregion. A proxy for patient need will be generated by calculating regional rates (using residents in each region) of services for substance use, schizophrenia spectrum disorders, and all other mental disorders using ICD-9 and ICD-10 codes (note: DSM-IV/V codes will be used for hospitalisations recorded in the Ontario Mental Health Reporting System) in emergency department, hospital and outpatient physician services.⁵

Psychiatrist practice style

We will use distributions of psychiatrist service volume (number of unique patient contacts) and service intensity (number of visits per patient) to identify distinct psychiatrist practice styles. Following Rudoler et al,⁶ we will use a latent profile analysis^{40 41} that will estimate the probabilities that a psychiatrist is in a particular practice style. Based on our previous work, we hypothesise that there will be three distinct psychiatrist practice styles in each province (high intensity urban, low intensity urban and low intensity rural). To test this hypothesis, we will apply the model developed with Ontario data in the other study provinces. If we find that our hypothesis does not hold outside of Ontario, we will fit province-specific models.

Regardless of the model, we will use the results of the latent profile analysis to describe the personal and practice characteristics of high and low intensity practice styles as well as outpatient case-mix. Patient case mix will be measured using previous psychiatric hospitalisation, Charlson Comorbidity Index, neighbourhood income quintile, low-income drug programme (where available) and diagnosis of schizophrenia spectrum disorder using a validated algorithm.⁴² We will also summarise regional supply in each province, stratified by province-specific practice style (high/low intensity).

Relationship between psychiatrist supply, practice style and supply of psychiatric care

This analysis will integrate our findings from objectives 1 and 2 by studying the relationship between regional psychiatrist supply, psychiatrist practice style (high/low intensity) and regional supply of psychiatric care. The aim is to provide decision-makers with an overview of the regional distribution of psychiatrist practice and practice style to inform human resource planning. The analysis will be conducted using data aggregated to region level (16 regions in BC, 34 regions in ON and 5 regions in MB). Data visualisation and geographic mapping will be used to show the concentration of psychiatrist practice, the quantity and intensity of psychiatric care provided and the breakdown of practice style within each region.

Objective 3. Study psychiatrist preferences for practice style, and the factors that lead to practice style selection

Semistructured, in-depth interviews

Semistructured, in-depth interviews with psychiatrists will be conducted to build on existing knowledge and explore factors shaping practice style choices. Some potential factors have been identified in our previous work,^{4 6 43} in studies exploring choice of psychiatry.^22–28 Factors to be explored in interviews include preferences for specific aspects of clinical work (eg, patient population, types of services provided and interest in particular topics), structural factors (eg, billing schedules and healthcare organisations) and personal factors (eg, age, sex/gender and location of training). Interviews will also explore the degree to which participants’ practice patterns have changed retrospectively and how they anticipate practice patterns will evolve over time, and why.

We will conduct all interviews by telephone or video call at a time that is convenient and acceptable for study participants. Telephone, in-depth interviews have been shown to produce similar data richness for a lower cost when compared with in-person interviews,^{44 45} can be more readily integrated into psychiatrists’ schedules, and allow data collection to proceed in the context of COVID-19. We expect each interview to last between 60 and 90 min. An honorarium will be offered to compensate for time and lost income. Interviews will be digitally recorded (audio-only in the case of video calls) and transcribed.

As is usual in qualitative research, our sampling strategy is designed to generalise to concepts and theories rather than to populations.^{46 47} Accordingly, we will purposefully recruit psychiatrists based on key characteristics identified in prior research, including personal characteristics, characteristics of residency training and practice setting. We will add to and modify our purposeful sampling strategy as we collect data and learn more about the factors that shape decision-making.^{46 48}

To ensure as much variation as possible, study participants will be recruited in all four provinces using multiple recruitment approaches. We will also use the team’s contacts with professional psychiatric associations to access email lists and circulate recruitment information on social media. Potential study participants will be asked to complete a screening questionnaire to ensure that they meet the study inclusion criteria, to inform our purposeful sampling decisions and to document the success of various recruitment methods. We will draw on the details collected on an individual’s screening questionnaire to inform their semistructured interview.

We have planned for sample sizes of 10–15 participant interviews in each province, BC, MB and NS (n=30–45). A larger sample of n=15–20 will be captured in ON due to the potential for greater variability in psychiatrist practice and larger population. We expect that this sample size will allow us to reach saturation⁴⁹ in each province, as well as provide us with enough data to investigate potential differences by subgroups and by province.⁴⁸ This sample size is consistent with recommendations for minimum samples for similar study designs and recommendations for study-specific increases over minimum sample sizes to reflect multiple variations within the study, such as our four provinces.^{22 23 25}

Interview data will be analysed using thematic and framework analysis.^50–52 Thematic analysis will occur concurrently with qualitative data collection. Analysis will consist of a line-by-line examination of interview transcripts to identify key themes.⁵⁰ Constant comparison will also be used to compare and contrast themes from our data with concepts already in the literature.⁵³ The thematic analysis will be used for detailed description of practice choice⁵³ and as the initial stage of our framework analysis.^{51 52} Patterns within and between cases and themes will be used to develop a framework or a typology of factors shaping practice choice. The typology will then be used to classify each interviewee and look for patterns among physicians within provinces and by years in practice.^{51 52} We will explore variations by province and other subgroups as appropriate based on the data. NVivo V.12 software will be used to facilitate analyses and record analytical decisions and definitions. Multiple researchers from the four provinces will be involved with coding and will participate in regular group analytic discussions to help ensure the final analysis meets trustworthiness, validity and reliability criteria for qualitative research.^{54 55}

Patient and public involvement

There will be no patient or public involvement in this study.

Ethics and dissemination

We received ethics approval for both quantitative and qualitative arms of this study from Ontario Tech University (REB numbers 16637 and 16795, respectively), Simon Fraser University/University of British Columbia (harmonised REB number H21-03694), University of Manitoba (REB number HS25310 (H2022:010) and Nova Scotia Health (REB number 1028069).

We plan to publish the results from this research project in peer-reviewed scientific journals and share our findings at conferences to contribute to scientific knowledge. We also plan to publish key findings in non-academic reports targeted for our partners and stakeholders. This will include the geographic mapping of psychiatrist supply. To inform decision-making, outputs will be disseminated to a range of audiences, including our partners at the BC Ministry of Health, Manitoba Department of Health, Seniors and Active Living, Ontario Health, Nova Scotia Department of Health and Wellness and Nova Scotia Health.

Ethics statements

Patient consent for publication

Not required.

Footnote

Contributors DR, MRL, JZ and EGM conceptualised the study. DR wrote the introduction, quantitative method and ethics and dissemination. JZ and EGM developed qualitative methods. PK, AK, AG, JB, KG, SE, BM, JM, SP, RK and CM advised on the study design and methodology. DR, MRL, JZ, EGM, KPPG, PK, BM, JM, RK, PT, AG, AK, JB, SE, CM and SP provided feedback and revised the final version for submission.

Funding This work was supported by the Canadian Institutes of Health Research (PJT-178110).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

Provenance and peer review Not commissioned; peer reviewed for ethical and funding approval prior to submission.

References

¹ Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. Lancet Psychiatry 2016; 3: 171–8. doi:10.1016/S2215-0366(15)00505-2

² Lamb D, Davidson M, Lloyd-Evans B, et al. Adult mental health provision in England: a national survey of acute day units. BMC Health Serv Res 2019; 19: 866. doi:10.1186/s12913-019-4687-8

³ Public health agency of Canada. Report from the Canadian chronic disease surveillance system: mental illness in Canada, 2015. 2015. Available: https://www.canada.ca/en/public-health/services/publications/diseases-conditions/report-canadian-chronic-disease-surveillance-system-mental-illness-canada-2015.html [Accessed 30 Sep 2020 ].

⁴ Kurdyak P, Zaheer J, Cheng J, et al. Changes in characteristics and practice patterns of Ontario psychiatrists. Can J Psychiatry 2017; 62: 40–7. doi:10.1177/0706743716661325

⁵ Kurdyak P, Stukel TA, Goldbloom D, et al. Universal coverage without universal access: a study of psychiatrist supply and practice patterns in Ontario. Open Med 2014; 8: e87–99.

⁶ Rudoler D, de Oliveira C, Zaheer J, et al. Closed for business? Using a mixture model to explore the supply of psychiatric care for new patients. Can J Psychiatry 2019; 64: 568–76. doi:10.1177/0706743719828963

⁷ Butler A, Adair C, Jones W, et al. Toward quality mental health services in Canada: a comparison of performance indicators across 5 provinces. Vancouver: Centre for Applied Research in Mental Health & Addiction, 2017. Available: https://www.sfu.ca/content/dam/sfu/carmha/resources/2017-toward-quality-mh/CARMHA%20REPORT_29%20Aug_Final.pdf [accessed 31 Mar 2021 ].

⁸ Anderson KK, Kurdyak P. Factors associated with timely physician follow-up after a first diagnosis of psychotic disorder. Can J Psychiatry 2017; 62: 268–77. doi:10.1177/0706743716673322

⁹ Rudoler D, de Oliveira C, Cheng J, et al. Payment incentives for community-based psychiatric care in Ontario, Canada. CMAJ 2017; 189: E1509–16. doi:10.1503/cmaj.160816

¹⁰ Caton CLM, Koh SP, Fleiss JL, et al. Rehospitalization in chronic schizophrenia. J Nerv Ment Dis 1985; 173: 139–48. doi:10.1097/00005053-198503000-00002

¹¹ Katherine MB, Samuel AM, Sikerbol K. Early rehospitalization. Can J Psychiatry 1991; 36: 743–5. doi:10.1177/070674379103601010

¹² Lavergne MR, Loyal JP, Shirmaleki M, et al. The relationship between outpatient service use and emergency department visits among people treated for mental and substance use disorders: analysis of population-based administrative data in British Columbia, Canada. BMC Health Serv Res 2022; 22: 477. doi:10.1186/s12913-022-07759-z

¹³ Canadian Mental Health Association. Mental health in the balance: ending the health care disparity in Canada. Canadian Mental Health Association; 2018. Available: https://cmha.ca/wp-content/uploads/2018/09/CMHA-Parity-Paper-Full-Report-EN.pdf

¹⁴ Mental Health Commission of Canada. Making the case for investing in mental health in Canada. Mental Health Commission of Canada. Available: https://www.mentalhealthcommission.ca/sites/default/files/Investing_in_Mental_Health_FINAL_Version_ENG_0.pdf

¹⁵ Canadian Medical Association Canadian Psychiatric Association. Joint statement on access to mental health care from the Canadian Medical Association and Canadian psychiatric Association. Canadian Medical Association & Canadian Psychiatric Association; 2016.

¹⁶ Ontario Psychiatric Association. Ontario needs psychiatrists: chronic psychiatry shortage contributing to Canada’s mental health crisis. Available: https://www.eopa.ca/sites/default/uploads/files/Ontario%20Needs%20Psychiatrists%20FINAL%20-%20August%207%2C%202018.pdf [Accessed 15 Sep 2020 ].

¹⁷ Canadian Institute for Health Information. National physician database 2017-2018 - data release. 2019. Available: secure.cihi.ca/estore/productFamily.htm?pf=PFC4052&lang=en$media=0 [Accessed 22 Jul 2020 ].

¹⁸ Kurdyak P, Zaheer J, Carvalho A, et al. Physician-based availability of psychotherapy in Ontario: a population-based retrospective cohort study. CMAJ Open 2020; 8: E105–15. doi:10.9778/cmajo.20190094

¹⁹ Clemens J, Gottlieb JD. Do physicians’ financial incentives affect medical treatment and patient health Am Econ Rev 2014; 104: 1320–49. doi:10.1257/aer.104.4.1320

²⁰ Douven R, Mocking R, Mosca I. The effect of physician remuneration on regional variation in hospital treatments. Int J Health Econ Manag 2015; 15: 215–40. doi:10.1007/s10754-015-9164-2

²¹ Strauss M. Ontario’s proposed cuts welcomed by critics of "boutique psychiatrists" who over-treat wealthy patients. National post. Available: https://nationalpost.com/health/boutique-psychiatrists

²² Choudry A, Farooq S. Systematic review into factors associated with the recruitment crisis in psychiatry in the UK: students’, trainees’ and consultants’ views. BJPsych Bull 2017; 41: 345–52. doi:10.1192/pb.bp.116.055269

²³ McManus IC, Lefford F, Furnham AF, et al. Career preference and personality differences in medical school applicants. Psychol Health Med 1996; 1: 235–48. doi:10.1080/13548509608402221

²⁴ Curtis-Barton MT, Eagles JM. Factors that discourage medical students from pursuing a career in psychiatry. Psychiatrist 2011; 35: 425–9. doi:10.1192/pb.bp.110.032532

²⁵ Nielsen AC, Eaton JS. Medical students’ attitudes about psychiatry: implications for psychiatric recruitment. Arch Gen Psychiatry 1981; 38: 1144–54. doi:10.1001/archpsyc.1981.01780350078009

²⁶ Yager J, Lamotte K, Nielsen A, et al. Medical students’ evaluation of psychiatry: a cross-country comparison. Am J Psychiatry 1982; 139: 1003–9. doi:10.1176/ajp.139.8.1003

²⁷ Goldenberg MN, Williams DK, Spollen JJ. Stability of and factors related to medical student specialty choice of psychiatry. AJP 2017; 174: 859–66. doi:10.1176/appi.ajp.2017.17020159

²⁸ Lyons Z. Attitudes of medical students toward psychiatry and psychiatry as a career: a systematic review. Acad Psychiatry 2013; 37: 150–7. doi:10.1176/appi.ap.11110204

²⁹ Hedden L, Barer ML, Cardiff K, et al. The implications of the feminization of the primary care physician workforce on service supply: a systematic review. Hum Resour Health 2014; 12: 32. doi:10.1186/1478-4491-12-32

³⁰ Lavergne MR, Gonzalez A, Ahuja MA, et al. The relationship between gender, parenthood and practice intentions among family medicine residents: cross-sectional analysis of national Canadian survey data. Hum Resour Health 2019; 17: 67. doi:10.1186/s12960-019-0396-y

³¹ Mental Health Commission of Canada. Mental health Commission of Canada. MEASURING PROGRESS: resources for developing a mental health and addiction performance measurement framework for Canada. 2018. Available: https://www.mentalhealthcommission.ca/sites/default/files/2018-07/MHPMF_final_report_eng_0.pdf

³² Canadian Institute for Health Information. Selecting Pan-Canadian indicators for access to mental health and addiction services, and to home and community care: progress report. Ottawa, ON: CIHI; 2018. Available: https://www.cihi.ca/sites/default/files/document/shp-interim-progress-rep-en.pdf [Accessed 7 Sep 2020 ].

³³ Virgo Planning and Evaluation Consultants Inc. Improving access and coordination of mental health and addiction services: a provincial strategy for all Manitobans: Toronto. 2018. Available: https://www.gov.mb.ca/health/mha/docs/mha_strategic_plan.pdf [Accessed 8 Oct 2020 ].

³⁴ Government of Nova Scotia. Together we can: the plan to improve mental health and addictions care for Nova Scotians. Halifax. Available: https://novascotia.ca/dhw/mental-health/reports/Mental-Health-and-Addictions-Strategy-Together-We-Can.pdf [Accessed 10 Oct 2020 ].

³⁵ Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd edn. Los Angeles: SAGE, 2018.

³⁶ Leech NL, Onwuegbuzie AJ. A typology of mixed methods research designs. Qual Quant 2009; 43: 265–75. doi:10.1007/s11135-007-9105-3

³⁷ Handcock MS, Morris M. Relative distribution methods in the social sciences. New York: Kluwer Academic, 2002.

³⁸ Rudoler D, Laporte A, Barnsley J, et al. Paying for primary care: a cross-sectional analysis of cost and morbidity distributions across primary care payment models in Ontario Canada. Soc Sci Med 2015; 124: 18–28. doi:10.1016/j.socscimed.2014.11.001

³⁹ Contoyannis P, Wildman J. Using relative distributions to investigate the body mass index in England and Canada. Health Econ 2007; 16: 929–44. doi:10.1002/hec.1240

⁴⁰ Oberski DL. Modern statistical methods for HCI. In: Robertson J, Kaptein M, eds. Mixture models: latent profile and latent class analysis. Cham: Springer, 2016: 275–87. doi:10.1007/978-3-319-26633-6

⁴¹ Deb P, Trivedi PK. Demand for medical care by the elderly: a finite mixture approach. J Appl Econ 1997; 12: 313–36. doi:10.1002/(SICI)1099-1255(199705)12:3<313::AID-JAE440>3.0.CO;2-G

⁴² Kurdyak P, Lin E, Green D, et al. Validation of a population-based algorithm to detect chronic psychotic illness. Can J Psychiatry 2015; 60: 362–8. doi:10.1177/070674371506000805

⁴³ Lavergne MR, Goldsmith LJ, Grudniewicz A, et al. Practice patterns among early-career primary care (ECPC) physicians and workforce planning implications: protocol for a mixed methods study. BMJ Open 2019; 9: e030477. doi:10.1136/bmjopen-2019-030477

⁴⁴ Burke LA, Miller MK. Phone interviewing as a means of data collection: lessons learned and practical recommendations. Forum Qual Sozialforschung 2001; 2. doi:10.17169/fqs-2.2.959

⁴⁵ Drabble L, Trocki KF, Salcedo B, et al. Conducting qualitative interviews by telephone: lessons learned from a study of alcohol use among sexual minority and Heterosexual women. Qual Soc Work 2016; 15: 118–33. doi:10.1177/1473325015585613

⁴⁶ Patton MQ. Qualitative research & evaluation methods: integrating theory and practice. SAGE Publications, 2014.

⁴⁷ Green J, Thorogood N. Qualitative methods for health research. SAGE, 2013.

⁴⁸ Sandelowski M. Sample size in qualitative research. Res Nurs Health 1995; 18: 179–83. doi:10.1002/nur.4770180211

⁴⁹ Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and Operationalization. Qual Quant 2018; 52: 1893–907. doi:10.1007/s11135-017-0574-8

⁵⁰ Aronson J. A pragmatic view of thematic analysis. TQR 1995; 2: 1–3. doi:10.46743/2160-3715/1995.2069

⁵¹ Gale NK, Heath G, Cameron E, et al. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013; 13: 117. doi:10.1186/1471-2288-13-117

⁵² Ritchie J, Spencer L. Qualitative data analysis for applied policy research | 14 | analyzing. 1994. Available: https://www.taylorfrancis.com/chapters/edit/10.4324/9780203413081-14/qualitative-data-analysis-applied-policy-research-jane-ritchie-liz-spencer [accessed 19 Jul 2022 ].

⁵³ Sandelowski M. Whatever happened to qualitative description. Res Nurs Health 2000; 23: 334–40. doi:10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g

⁵⁴ Lincoln YS, Guba EG, Pilotta JJ. Naturalistic inquiry. Int J Intercult Relat 1985; 9: 438–9. doi:10.1016/0147-1767(85)90062-8

⁵⁵ Mays N, Pope C. Rigour and qualitative research. BMJ 1995; 311: 109–12. doi:10.1136/bmj.311.6997.109