Unravelling the determinants of medical practice variation in referrals among primary care physicians: insights from a retrospective cohort study in Southern Israel

Abstract

Objectives

Reducing medical practice variation (MPV) is a central theme of system improvement because it is associated with poor health outcomes, increased costs and disparities in care. This study aimed to estimate the extent to which each determinant (patient, physician, clinic) explains MPV among primary care physicians and to identify the characteristics of health services with a greater explained variance.

Methods

A retrospective cohort study of primary care physicians practising in non-private clinics of Clalit Health Services in Southern Israel, for longer than a year between 2011 and 2017 and with more than 100 adult patients per practice. We assessed the variation in referral rates among 17 health services and the proportion explained by each domain (patient, physician and clinic). We used generalised linear negative binomial mixed models and the Nakagawa’s R², computing the marginal r².

Results

The study included 243 physicians working in 295 practices and 139 clinics. The mean-explained variance was 28.5%±10.0%, where physician characteristics explained 4.5% of the variation. The intrapractice variation (within a single physician between the years) was explained better than the interphysician (between physicians). Health services with high explained variation were blood tests characterised by both low intrapractice variation (Rs=−0.65, p value=0.005) and high referral rates (Rs=0.46, p value=0.06).

Conclusion

Over 70% of MPV is not explained by the patient, clinic and physician demographic and professional characteristics. Future research should focus on the fraction of MPV that is explained by the physicians’ psychological characteristics, and thus potentially identify psychological targets for behavioural modifications aimed at reducing MPV.

Correspondence to Dr Sagi Shashar; [email protected]

STRENGTHS AND LIMITATIONS OF THIS STUDY

• The study is based on a large and comprehensive administrative database with a large sample size across 17 diverse healthcare services and referred by 250 physicians in 140 clinics, over 7 years of practice.

• The study provides essential insights for developing approaches to reduce unwarranted variation in medical practice and aims to propose potential targets for focused behavioural interventions to improve health outcomes and reduce disparities in care.

• Study’s limitations include the inherent limitations of the administrative database, such as the inability to determine patient utilisation of referrals or assess patient/physician preferences.

• Also the inability to estimate system-level factors such as resource constraints, process, workflow issues, funding, services accessibility and availability and the lack of assessment of patient-oriented health outcomes associated with underuse or overuse of diagnostic testing.

Introduction

Medical practice variation (MPV) is associated with poor health outcomes, increased costs, disparities in care and burden on medical systems.^{1 2} Unwarranted MPV is the type of variation that cannot be explained by patient illness or preferences.³ Reducing the variation is a central theme of system improvement and has been recently adopted in medical practice.⁴ The reduction of the unwarranted MPV and devising appropriate interventions⁵ requires cause analysis. Causes of MPV can be divided into three main domains: patient, physician and healthcare system characteristics.⁶ The traditional MPV research infrequently assessed all three determinants within the same investigational framework.^{7 8} Moreover, MPV research often focused on the secondary and tertiary sectors of care and analysed a single health service.^{6 9}

In our previous study,¹⁰ we conducted an extensive analysis of MPV in primary care, revealing substantial variation among physicians and individual practices over time. Interestingly, we observed that the intraphysician variation was lower compared with the interphysician variation. This intriguing finding led us to interpret that physicians exhibit consistent clinical behaviour within their own practices, suggesting the presence of a distinct and personalised protocol or approach to patient care.

In the current study, our objective was to examine the factors contributing to MPV among primary care physicians in the Southern region of Israel. Specifically, we aimed to assess the variation in referrals, which served as our primary outcome, and determine the extent to which patient, physician and clinic characteristics explain this variation. To ensure a comprehensive evaluation of the factors influencing MPV, we selected the Southern District of Israel, the ‘Negev’, as the study location. This region offered a diverse mix of physicians, patients and clinics, providing an ideal environment for investigating the impact of physician and patient characteristics on MPV. By including physicians and patients from varied backgrounds and contexts, we aimed to capture a wide range of primary care practices and patient populations, thereby enhancing the generalisability and robustness of our study findings.

By examining the variation in referrals and assessing the influence of patient, physician and clinic characteristics, we aim to identify the underlying causes of MPV and develop targeted interventions to reduce unwarranted variation. Through this research, we hope to improve the quality of care and enhance healthcare outcomes for patients.

Methods

Study design and population

In this retrospective cohort study, we included primary care physicians practising in non-private clinics of Clalit Health Services (CHS) in the Southern region of Israel for more than a year between 2011 and 2017, with more than 100 adult patients per practice. CHS is the largest health insurance organisation in Israel (4.5 million insurees) and the largest healthcare provider in Southern Israel, covering approximately 70% of its 730 000 residents.

Data collection

Data was collected from the computerised medical records of CHS, including physician, patient, clinic and referral data. The unit of analysis, the data’s structure, was the annual practice, that is, physician/clinic/year, to address physicians working simultaneously in more than one clinic. For instance, a physician working between years 2013 and 2017 in 2 clinics would be considered as working in two practices and would be represented in the data by 10 rows, 5 for each year per practice. ‘Practice’ refers to a primary care unit consisting of a primary care physician and their associated patient population within a specific clinic.

The physician data included the age, gender, seniority (length of time practising, in years), number of years employed in CHS, country of birth, specialty and practice size. The annual patient data (age>18) for each physician per clinic included mean age, socioeconomic status (SES), marital status, per cent of male patients, Bedouin Arabs, patients with diabetes, hypertension, malignancies and bedridden patients. The annual clinic data included the type of clinic (primary/primary and professional/rural), SES, the total number of physicians and patients and the number of visits per practice.

Outcomes

The referral data, the primary outcomes, included 17 health services (HSs) in primary care. The annual referral rates were calculated as the number of referrals per year divided by the total number of insured patients under the physician’s care, and then multiplied by 1000 to standardise the rates per 1000 patients. We chose HSs that involve clinical scenarios with discretionary decisions where the physician has the freedom of action to decide whether to use them.¹¹ The referrals are divided into five domains:

• Four imaging tests: bone scintigraphy, CT of the brain and spine, X-ray of the chest (CXR) and MRI.

• A composite of cardiac tests: 24-hour Holter electrocardiography, stress test and echocardiography.

• Six laboratory tests: vitamin B12, vitamin D, thyroid-stimulating hormone, haemoglobin, carcinoembryonic antigen and prostate-specific antigen.

• Three specialist consultations visits: rheumatology, pulmonary and neurology.

• Three emergency department (ED) referrals: all referrals to the ED, referrals due to chest pain and referrals due to back pain.

Statistical analysis

We described quantitative variables with normal distribution using mean and SD, variables with non-normal distributions using the median and IQR, and categorical variables using frequencies and percentages.

To assess the explained variance by each of the three domains (patient, physician, clinic), we first computed three regression models for each HS, including covariates related to a single domain (patient, physician, clinic). For instance, for the CXR referral rate outcome, we computed three regression, one including only patient characteristics, one including physician characteristics and one including clinic characteristics, all as fixed effects covariates separately. We used generalised linear negative binomial mixed models with an unstructured correlation matrix, since our outcomes were a fraction value (referral annual rate). The annual HSs’ referrals per practice were defined as outcomes, and the annual insured patients per practice volume as the outcome’s offset. We used mixed models as data was clustered by physicians, clinics and years and accordingly they were defined as random effect clusters in the regression.

Then, for each regression model, we calculated Nakagawa’s R², computing the marginal r², which reflects the percentage of variance explained only by the fixed effects. Thus, we were able to calculate the precise percentage of explained variance in referral rates, separately for each HS’s, by the three domains: patient, physician and clinic. Using intraclass correlation coefficient (ICC) was not suitable in this context because our analysis involved mixed effects modelling, which includes both mixed and fixed effects. It was necessary to separate the explained variance by these two types of effects. Therefore, we used Nakagawa’s R² to accurately assess the explained variance specifically attributable to the fixed effects. ICC does not provide the same level of distinction and was not appropriate for our analysis.¹²

Furthermore, we calculated the interpractice and intrapractice coefficient of variations (COV) before and after the adjustment to assess which type of variation was explained more by the determinants. COVs were calculated as: COV=SDmean×100. We calculated interpractice COV using practices’ averaged referrals, while for intrapractice we first calculated singular COVs for each practice and then averaged them to derive the HS’s overall intrapractice COV. We further calculated the proportional change in variation (PCV) using the formula: ${\displaystyle \mathrm{P}\mathrm{C}\mathrm{V}=\frac{\mathrm{V}\mathrm{n}1-\mathrm{V}\mathrm{n}2}{\mathrm{V}\mathrm{n}1}}$,¹³ where Vn1=crude COV and Vn2=adjusted COV. We presented the results per 1000 patients.

We then assessed the correlation between the proportion of the explained variance and the interpractice and intrapractice variations and referral rates and ICC, using Spearman’s test. We obtained the ICC from regression models inclusive all the independent covariates analysed in the study, where higher ICC indicates that within-group variation is larger than the between.

We used the ‘glmmTMB’ and ‘performance’ R packages,¹⁴ V.1.0.136, and IBM SPSS, V.24. P values<0.05 were considered significant.

Patient and public involvement

Patients and the public were not involved in any way in the study.

Results

Study population

Table 1 depicts the practice characteristics (2014). The study included 243 physicians working in 295 practices, a total of 139 clinics and an overall of 1 864 714 patient-years. More than half of the physicians (132) were board-certified specialists in primary care medicine. 48% (116) were men; the mean age±SD was 52.1±8.4 with the length of practice at a median of 27 years (IQR 16–33). Overall, 27% of the clinics were in rural areas, the median number of patients per clinic was 4278.0 (IQR 753.0–6701.0) and the median number of physicians per clinic was 4.

Table 1

Patient, physician and clinic characteristics per practice (2014)

	Total practices (n=295)
Physician characteristics
Male gender, % (n)	52.9% (156)
Age, years, mean±SD	52.5±8.1
Specialists in primary care, % (n)	56.3% (166)
Seniority, years, median (IQR)	28.0 (16–32)
Seniority in Clalit Health Service, years median (IQR)	17.6 (10.0–23.5)
Born in Israel, % (n)	26.4% (78)
Physicians working in more than one clinic, % (n)	17.6% (52)
Patient characteristics
SES, median (IQR)	5.1 (1.0–8.4)
Age, mean±SD	44.8±7.4
Per cent males, mean±SD	47.3%±5.2%
Per cent married, mean±SD	55.5%±9.4%
Per cent patients with diabetes, mean±SD	16.2%±9.3%
Per cent patients with hypertension, mean±SD	27.5%±17.7%
Per cent bedridden patients, mean±SD	10.9%±7.4%
Per cent patients with malignancy, mean±SD	8.8%±9.7%
Patient census, median (IQR)	1003.0 (460.0–1276.0)
Average annual number of visits per patient, mean±SD	5.3±1.4
Total visits, median (IQR)	4278.0 (753.0–6701.0)
Clinic characteristics
Rural clinics, % (n)	26.7% (79)
Number of physicians median (IQR)	4 (1.0–5.0)
SES, median (IQR)	5.1 (1.1–8.2)

SES, socioeconomic status.

The median annual total patient population was 269 569, and the median number of patients per practice was 1003.0 (IQR 460.0–1276.0) with mean age±SD 44.8±7.4 and median SES of 5.1 (IQR 1.0–8.4), on a 20-point scale. The annual mean per cent of patients per practice with diabetes, hypertension and malignancy was 16.2%±9.3%, 27.5%±17.7% and 8.8%±9.7%, respectively.

Table 1 presents the patient, physician and clinic characteristics for all 295 practices, as of 2014 (the middle year of the study period). Covariates with normal distribution were presented as means and SD, numerical variables without a distribution diverging from the normal were presented as medians and IQRs and categorical variables as frequencies and percentages. Practice refers to a primary care unit consisting of a primary care physician and their associated patient population within a specific clinic. Table 1 depicts the practice characteristics.

Explained variance

Online supplemental table 1 shows the crude variances per HS and the proportion of variance explained by patient, physician and clinic characteristics.

The overall mean-explained variance was 28.5%±10.0. The mean-explained variance was the highest for patient characteristics (15.3%±8.5%) followed by clinic characteristics (8.6%±4.8%) and the lowest for physician characteristics (4.5%±2.4%). Figure 1 shows the proportions of the explained variation by the three determinants per HS and the total unexplained variance.

Figure 1. Explained versus unexplained variation. CEA, carcinoembryonic antigen; HGB, haemoglobin; PSA, prostate-specific antigen; TSH, thyroid-stimulating hormone.

The mean interpractice COV prior to and following the adjustment for patient, physician and clinic characteristics was 81.9%±36.5% and 74.3%±31.5%, respectively. The mean intrapractice COV prior to and following the adjustment was 45.0%±23.7% and 12.6%±7.2%, respectively. The mean PCV for interpractice variation was 6.0%±6.3% and 67.5%±15.0% for intrapractice variation.

Online supplemental table 1 Proportional change in the variance; 3 domains across 17 health services

Online supplemental table 1 shows the crude variances per HS and the proportion of variance explained by patient, physician and clinic characteristics. It also presents the interpractice and intraphysician COVs prior to and following the adjustment for patient, physician and clinic characteristics. COVs were calculated as: ${\displaystyle \mathrm{C}\mathrm{O}\mathrm{V}=\frac{\mathrm{S}\mathrm{D}}{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}\times 100}$. The adjusted COVs present the PCV using the formula: ${\displaystyle \mathrm{P}\mathrm{C}\mathrm{V}=\frac{\mathrm{V}\mathrm{n}1-\mathrm{V}\mathrm{n}2}{\mathrm{V}\mathrm{n}1}}$, where Vn1=crude COV and Vn2=adjusted COV.

Figure 1 shows the proportions of the explained variation by the three determinants (patient, physician demographic and occupational and clinic) per HS, as well as the total unexplained variance.

Correlation analysis

Figures 2 and 3 illustrate the correlations between the HSs’ proportion of explained variance and referral rates, interpractice variation and intrapractice variation. Significant correlations were found between the HSs’ explained variance and high referral rates (Rs=0.46 p value=0.06, figure 2) and low intrapractice variation (Rs=−0.65, p value=0.005, figure 3). Additionally, correlation between the HSs’ explained variance was also found with the ICC=0.46 (p value=0.07), meaning that HSs with higher explained variance had higher interpractice variation than the intrapractice variation.

Figure 2. Intrapractice variation, referral rates and explained variance. Rs: p value, Intrapractice: -0.652 to 0.005. Rates: 0.458-0.06. CEA, carcinoembryonic antigen; HGB, haemoglobin; PSA, prostate-specific antigen; TSH, thyroid-stimulating hormone.

Figure 3. Interpractice variation, referral rates and explained variance. Rs: p value. Intrapractice, -0.260 to 0.314, Rates: 0.458-0.06. CEA, carcinoembryonic antigen; HGB, haemoglobin; PSA, prostate-specific antigen; TSH, thyroid-stimulating hormone.

Figure 2 illustrates the HSs by their intrapractice variation and explained variance (x,y axis, respectively), and the referral rates (size of the circles): low intrapractice variation HSs with high explained variance are positioned at the upper left. In contrast, high intravariation HSs with low explained variance at the lower right. The larger the circle, the higher the referral rate. Significant correlations were found between the HSs’ explained variance and high referral rates (Rs=0.46, p value=0.06) and low intrapractice variation (Rs=−0.65, p value=0.005)

Figure 3 illustrates the HSs by their interpractice variation and explained variance (x,y axis, respectively), and the referral rates (size of the circles): low interpractice variation HSs with high explained variance are positioned at the upper left, while high intervariation HSs with low explained variance at the lower right. The circle size correlates with the referral rate. Significant correlations were found between the HSs’ explained variance and high referral rates (Rs=0.46, p value=0.06).

Discussion

In this study, we found that patient, physician and clinic characteristics explain only a third of the variation in 17 main diagnostic activities of primary care physicians. The intrapractice variation was explained better than the interpractice variation. Finally, we showed that blood tests had the highest explained variation and were characterised by low intrapractice variation, high interpractice variation and high referral volume.

Unwarranted variation

Research to date has predominantly focused on patient characteristics as the primary determinant of variation in medical practice volume (MPV), suggesting that this variation falls within the domain of warranted MPV,^{5 15} driven by patient clinical differences or preferences.¹⁶ While our study does not directly discern the proportion of explained variance that specifically represents unwarranted variation, the significant contribution of physician and clinic characteristics, which accounted for half of the explained variance, suggests the presence of unwarranted variation. Unwarranted variation can stem from system-related factors, such as resource accessibility, or physician-related factors, including opinions, preferences, knowledge and seniority.¹⁷ The existence of unwarranted variation has been linked to suboptimal outcomes and increased healthcare costs.¹⁷ Future research can delve into the specific determinants of unwarranted variation and employ additional methodologies to differentiate between warranted and unwarranted variation more accurately. By continuing to explore and understand unwarranted variation, we can develop targeted interventions and strategies to reduce it, such as system-level improvements or behaviour modification strategies, to mitigate its impact and enhance the consistency and quality of care delivery.

Unexplained variation

Similar to other studies, and despite our efforts to assess and explain the sources of variation in referral rates, a substantial portion remains unexplained,¹⁸ which implies that other factors were not adequately estimated and researched so far.¹⁹ Dealing with unexplained variation is a critical aspect of improving healthcare delivery and reducing unwarranted variation. To address this challenge, further research is needed to identify and investigate the factors contributing to the unexplained variation. This may involve exploring additional variables, such as patient preferences, healthcare system factors or contextual influences, that were not captured in our study. Additionally, qualitative research methods, such as interviews or surveys, could provide insights into the subjective factors influencing referral decisions. By better understanding the underlying causes of unexplained variation, we can strive to optimise healthcare practices, improve patient outcomes and enhance the efficiency and effectiveness of healthcare systems.

Physicians’ psychological characteristics

The low percentage of variation explained by physician characteristics and the fact that the interpractice variation remains six times greater than the intrapractice variation suggest that differences in practice styles among physicians may be attributed to physicians’ personal characteristics. While the physician’s professional (even differences in medical school backgrounds) and demographic characteristics were studied,²⁰ this research did not yield an adequate explanation for MPV. It was hypothesised that physicians psychological characteristics may be the missing link.²¹ The physician’s psychological characteristics include personality, attitudes or behaviour.²² These characteristics are more difficult to estimate as they are very diverse and subjective and thus difficult to measure accurately.¹⁹

Assessing the extent of the variance that the physicians’ psychological characteristics can explain is essential because (1) this type of variation is unwarranted and (2) it may guide us to develop targeted behaviour modification tools that may successfully bring to MPV reduction.²³ Personal behavioural characteristics studied to date in the context of unwarranted MPV included risk attitudes,²⁴ adherence to treatment guidelines, empathy and fear of malpractice litigation. We suggest focusing on the physicians’ behaviour because it is validated and more measurable and quantifiable than personality and attitudes.²⁵

Computerised decision support (CDS)

The final objective of the current study was to identify common factors among HSs with high explained variance. To the best of our knowledge, this is the first study that compares the explained variance across diverse HSs. We observed that certain HSs, particularly blood tests, exhibited a higher likelihood of being explained, characterised by low intrapractice variation, high interpractice variation and greater referral rates. The low intrapractice variation suggests that physicians within each practice have a consistent approach in determining when blood tests should be used. Furthermore, some blood tests have specific restrictions on testing frequency, such as vitamin B12 and vitamin D tests being limited to once in 2 years or a year, respectively. In light of these findings, CDS systems-based interventions emerge as potential tools to reduce MPV in diagnostic testing. Successfully implemented CDS can aid in standardising the decision-making process, improving physicians’ knowledge, facilitating the adoption of new evidence and effectively managing beliefs, assumptions and uncertainties associated with diagnostic testing. By leveraging CDS, healthcare providers can strive to achieve more standardised and evidence-based practices, ultimately leading to better patient care and resource utilisation.²⁶

Limitations

This study has a number of limitations. Due to the inherent shortcomings of the administrative database, while all the data on the referrals given by the physician were available, we could not determine whether the patient used the referral. Therefore, we could not assess the overall costs incurred by each physician. However, as demonstrated before, most of the referrals (~80%) are used by patients.²⁷ In addition, since we used a computerised database and aggregated data at the annual practice level, more comprehensive data at the patient level, such as patient/physician preferences, distance from the hospital and patient adherence were not assessed. The aggregated data carries a risk of an ecological fallacy when drawing conclusions about causations. It is crucial to interpret the findings within the context of aggregated data and not directly generalise them to individual patients.

Our study setting precludes the estimation of system-level factors such as resource constraints, process, workflow issues, funding, services accessibility and availability. However, Israel has universal health coverage, and the system-level variation between healthcare networks in Israel is limited as the law controls their services’ provision and accessibility. Furthermore, healthcare providers in Israel are not compensated by ordering more testing and documenting more medical procedures and diagnoses. Therefore, our results may be generalisable to other countries with private medicine-based health system and practice intensity depended on remuneration.

Lastly, the aim of the current study was not to deduce what kind of physician is practising ‘better’ medicine—one who gives too many referrals or one who gives few. In other words, we have not assessed patient-oriented health outcomes such as mortality, hospitalisations and life-threatening events that might be associated with underuse or overuse of the diagnostic testing. However, since MPV has been previously shown to be associated with poorer health outcomes,¹ we believe that focusing on MPV determinants provides essential insight for developing MPV reduction approaches.

Conclusion

In this study, we demonstrated that the majority of MPV is unexplained and that the interpractice variation exceeds its intrapractice, indicating that the individual behavioural characteristics largely determine the practice patterns. Future research should focus on the fraction of MPV explained by the physicians’ personal behavioural characteristics, and thus potentially identify psychological targets for behavioural modifications aimed at reducing MPV.

Data availability statement

Data may be obtained from a third party and are not publicly available. The datasets used and/or analysed during the current study are available following local ethics committee approval.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The study was approved by the Soroka University Medical Center Institutional Ethics Committee (0063-14-SOR).

Footnote

Twitter @MoriahEllen

Contributors SS: substantial contributions to the design of the work, analysis, drafting the work. ME: substantial contributions to the design of the work and interpretation of data. SC and ED: substantial contributions to the conception. VN: the guarantor, substantial contributions to the conception, design, interpretation of data, substantively revised the work.

Funding Supported by the research grant from the Israel Health Policy Institute (number of grant: 2014/134).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Duddy C, Wong G. Explaining variations in test ordering in primary care: protocol for a realist review. BMJ Open 2018; 8: e023117. doi:10.1136/bmjopen-2018-023117

² Shashar S, Codish S, Ellen M, et al. Determinants of medical practice variation among primary care physicians: protocol for a three phase study. JMIR Res Protoc 2020; 9: e18673. doi:10.2196/18673

³ Wennberg JE. Unwarranted variations in Healthcare delivery: implications for academic medical centres. BMJ 2002; 325: 961–4. doi:10.1136/bmj.325.7370.961

⁴ Mullan F, Wennberg J. Interview: wrestling with variation: an interview with Jack Wennberg. Health Affairs 2004; 23: VAR–73 doi:10.1377/hlthaff.var.73

⁵ Goodman DC, Goodman AA. Medical care epidemiology and unwarranted variation: the Israeli case. Isr J Health Policy Res 2017; 6: 9. doi:10.1186/s13584-017-0135-6

⁶ O’Sullivan JW, Stevens S, Oke J, et al. Practice variation in the use of tests in UK primary care: A retrospective analysis of 16 million tests performed over 3.3 million patient years in 2015/16. BMC Med 2018; 16: 229. doi:10.1186/s12916-018-1217-1 Available: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-018-1217-1

⁷ Orueta JF, García-Alvarez A, Grandes G, et al. The origin of variation in primary care process and outcome indicators: patients, professionals, centers, and health districts. Medicine (Baltimore) 2015; 94: e1314. doi:10.1097/MD.0000000000001314

⁸ Reschovsky JD, Rich EC, Lake TK. Factors contributing to variations in physicians' use of evidence at the point of care: A conceptual model. J Gen Intern Med 2015; 30: S555–61. doi:10.1007/s11606-015-3366-7

⁹ Murchie P, Chowdhury A, Smith S, et al. General practice performance in referral for suspected cancer: influence of number of cases and case-mix on publicly reported data. Br J Cancer 2015; 112: 1791–8. doi:10.1038/bjc.2015.110

¹⁰ Shashar S, Ellen M, Codish S, et al. Medical practice variation among primary care physicians: 1 decade, 14 health services, and 3,238,498 patient-years. Ann Fam Med 2021; 19: 30–7. doi:10.1370/afm.2627 Available: http://www.annfammed.org/lookup/doi/10.1370/afm.2627

¹¹ Shashar S, Codish S, Ellen M, et al. Determinants of medical practice variation among primary care physicians: A study design and protocol (Preprint). JMIR Res Protoc 2020; 9: e18673. doi:10.2196/18673

¹² Nakagawa S, Schielzeth H. Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists. Biol Rev Camb Philos Soc 2010; 85: 935–56. doi:10.1111/j.1469-185X.2010.00141.x

¹³ Merlo J, Chaix B, Yang M, et al. A brief conceptual Tutorial of Multilevel analysis in social epidemiology: linking the statistical concept of clustering to the idea of Contextual phenomenon. J Epidemiol Community Health 2005; 59: 443–9. doi:10.1136/jech.2004.023473 Available: http://multilevel

¹⁴ Nakagawa S, Johnson PCD, Schielzeth H. The coefficient of determination R 2 an^d intra-class correlation coefficient from generalized linear mixed-effects models Revisited and expanded. J R Soc Interface 2017; 14: 20170213. doi:10.1098/rsif.2017.0213 Available: https://royalsocietypublishing.org/doi/10.1098/rsif.2017.0213

¹⁵ Corallo AN, Croxford R, Goodman DC, et al. A systematic review of medical practice variation in OECD countries. Health Policy 2014; 114: 5–14. doi:10.1016/j.healthpol.2013.08.002

¹⁶ Sándor J, Nagy A, Jenei T, et al. Influence of patient characteristics on preventive service delivery and general practitioners’ preventive performance indicators: A study in patients with hypertension or diabetes mellitus from Hungary. Eur J Gen Pract 2018; 24: 183–91. doi:10.1080/13814788.2018.1491545

¹⁷ Cook DA, Pencille LJ, Dupras DM, et al. Practice variation and practice guidelines: attitudes of generalist and specialist physicians, nurse practitioners, and physician assistants. PLoS One 2018; 13: e0191943. doi:10.1371/journal.pone.0191943

¹⁸ Seymour CW, Iwashyna TJ, Ehlenbach WJ, et al. Hospital-level variation in the use of intensive care. Health Serv Res 2012; 47: 2060–80. doi:10.1111/j.1475-6773.2012.01402.x

¹⁹ Korenstein D. Medical Overuse as a physician cognitive error: looking under the hood. JAMA Intern Med 2019; 179: 26–7. doi:10.1001/jamainternmed.2018.5136

²⁰ Bouck Z, Ferguson J, Ivers NM, et al. Physician characteristics associated with ordering 4 low-value screening tests in primary care. JAMA Netw Open 2018; 1: e183506. doi:10.1001/jamanetworkopen.2018.3506

²¹ Lipitz-Snyderman A, Sima CS, Atoria CL, et al. Physician-driven variation in Nonrecommended services among older adults diagnosed with cancer. JAMA Intern Med 2016; 176: 1541–8. doi:10.1001/jamainternmed.2016.4426

²² Meidani Z, Mousavi GA, Kheirkhah D, et al. Going beyond audit and feedback: towards behaviour-based interventions to change physician laboratory test ordering behaviour. J R Coll Physicians Edinb 2017; 47: 339–44. doi:10.4997/JRCPE.2017.407

²³ Sood R, Sood A, Ghosh AK. Non-evidence-based variables affecting physicians’ test-ordering tendencies: a systematic review. 1992.

²⁴ Pedersen AF, Carlsen AH, Vedsted P. Association of Gps’ risk attitudes, level of empathy, and burnout status with PSA testing in primary care. Br J Gen Pract 2015; 65: e845–51. doi:10.3399/bjgp15X687649

²⁵ Nelson-Piercy C, Vlaev I, Harris K, et al. What factors could influence physicians’ management of women of childbearing age with chronic inflammatory disease? A systematic review of behavioural determinants of clinical inertia. BMC Health Serv Res 2019; 19. doi:10.1186/s12913-019-4693-x Available: https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-019-4693-x

²⁶ Palen TE, Sharpe RE, Shetterly SM, et al. Randomized clinical trial of a clinical decision support tool for improving the appropriateness scores for ordering imaging studies in primary and specialty care ambulatory clinics. AJR Am J Roentgenol 2019; 213: 1015–20. doi:10.2214/AJR.19.21511

²⁷ Nelson-Piercy C, Vlaev I, Harris K, et al. What factors could influence physicians’ management of women of childbearing age with chronic inflammatory disease? A systematic review of behavioural determinants of clinical inertia. BMC Health Serv Res 2019; 19: 863. doi:10.1186/s12913-019-4693-x