Diagnostic accuracy in coronary CT angiography analysis: artificial intelligence versus human assessment

Abstract

Background

Visual assessment of coronary CT angiography (CCTA) is time-consuming, influenced by reader experience and prone to interobserver variability. This study evaluated a novel algorithm for coronary stenosis quantification (atherosclerosis imaging quantitative CT, AI-QCT).

Methods

The study included 208 patients with suspected coronary artery disease (CAD) undergoing CCTA in Perfusion Imaging and CT Coronary Angiography With Invasive Coronary Angiography-1. AI-QCT and blinded readers assessed coronary artery stenosis following the Coronary Artery Disease Reporting and Data System consensus. Accuracy of AI-QCT was compared with a level 3 and two level 2 clinical readers against an invasive quantitative coronary angiography (QCA) reference standard (≥50% stenosis) in an area under the curve (AUC) analysis, evaluated per-patient and per-vessel and stratified by plaque volume.

Results

Among 208 patients with a mean age of 58±9 years and 37% women, AI-QCT demonstrated superior concordance with QCA compared with clinical CCTA assessments. For the detection of obstructive stenosis (≥50%), AI-QCT achieved an AUC of 0.91 on a per-patient level, outperforming level 3 (AUC 0.77; p<0.002) and level 2 readers (AUC 0.79; p<0.001 and AUC 0.76; p<0.001). The advantage of AI-QCT was most prominent in those with above median plaque volume. At the per-vessel level, AI-QCT achieved an AUC of 0.86, similar to level 3 (AUC 0.82; p=0.098) stenosis, but superior to level 2 readers (both AUC 0.69; p<0.001).

Conclusions

AI-QCT demonstrated superior agreement with invasive QCA compared to clinical CCTA assessments, particularly compared to level 2 readers in those with extensive CAD. Integrating AI-QCT into routine clinical practice holds promise for improving the accuracy of stenosis quantification through CCTA.

RB and NSN are joint first authors.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Visual assessment of coronary CT angiography is influenced by reader experience and is prone to variability, which may lead to overestimation of stenosis and unnecessary interventions. Artificial intelligence (AI)-based algorithms, such as AI-QCT, have demonstrated high accuracy against invasive quantitative coronary angiography (QCA), but their performance compared with readers with different levels of experience remains understudied.

WHAT THIS STUDY ADDS

• This study shows that AI-QCT achieves superior agreement with invasive QCA compared with levels 2 and 3 clinical readers in detecting obstructive coronary stenosis (≥50%) when compared with QCA. The findings highlight the consistency and reliability of AI-QCT, particularly in patients with higher plaque volumes.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Integrating AI-QCT into clinical practice could improve diagnostic accuracy, reduce interobserver variability and minimise unnecessary downstream testing. Its application is especially beneficial in settings without access to highly experienced readers, paving the way for more standardised and efficient coronary artery disease assessment.

Introduction

Due to its ability of non-invasive imaging of coronary atherosclerosis and stenosis, recent guidelines have established coronary CT angiography (CCTA) as a first-line test for coronary artery disease (CAD).^{1 2}

Visual assessment of CCTA is time-consuming, highly dependent on the reader’s experience and is hallmarked by a large interobserver variability, as reported in previous studies.^{3 4} In 4347 patients undergoing CCTA imaging in the PROMISE trial, certified site readers classified 41% more patients as having significant CAD than a central core laboratory for CCTA analysis.⁴ Other studies have confirmed overestimation of coronary stenosis severity by readers with less experience.⁵ When applying CCTA more routinely, as is advocated by the guidelines,² there will be an inevitable increase in unnecessary downstream procedures with the current visual standard of care for CCTA interpretation. Therefore, increased accuracy and reproducibility of CCTA analysis is highly warranted.

With the introduction of artificial intelligence-based analysis of CCTA, there is an alternative to the traditional visual analysis of CCTA by human readers. A novel AI-guided algorithm for assessment of coronary stenosis (atherosclerosis imaging quantitative CT [AI-QCT]) has shown high accuracy against invasive quantitative coronary angiography (QCA) and other modalities.^5–8 However, it has not been reported to date how AI-based quantification of coronary stenosis performs in direct comparison to different levels of physician CCTA readers.

Therefore, the present study sought to compare the diagnostic accuracy of AI-QCT with visual CCTA assessment by readers with various levels of experience in a post hoc analysis of the Prospective Comparison of Cardiac PET/CT, SPECT/CT Perfusion Imaging and CT Coronary Angiography With Invasive Coronary Angiography (PACIFIC-1) study.⁹

Methods

Study population

The study population consisted of 208 patients with stable new-onset chest pain and suspected CAD from the PACIFIC-1 study.⁹ The PACIFIC-1 study was a prospective controlled clinical single-centre study conducted from 23 January 2012 to 25 October 2014, at the Amsterdam University Medical Centers, Amsterdam, the Netherlands (online supplemental file 1).⁹

CCTA acquisition

As previously reported, patients underwent CCTA on a 256-slice CT scanner (Philips Brilliance iCT, Philips Healthcare, Best, the Netherlands) with a collimation of 128×0.625 mm and a tube rotation time of 270 ms.⁹ Prior to imaging, sublingual nitroglycerin spray was administered to all patients and, if necessary metoprolol, aimed at achievement of a heart rate <65 beats/min. The scans were triggered using an automatic bolus-tracking technique of 100 mL of iodinated contrast agent (Xenetix 350; 5.7 mL/s) with a region of interest placed in the descending thoracic aorta. A tube current between 200 and 360 mAs at 120 kV was used, adjusting primarily the mAs based on body habitus. Prospective ECG-gating (Step & Shoot Cardiac, Philips Healthcare, Best, the Netherlands) at 75% of the R-R interval was performed in order to minimise radiation burden.

Visual image analysis

The CCTA images were interpreted independently by human readers blinded to each other’s interpretations, patient characteristics and QCA results. Every scan was interpreted by a level 3 radiologist following the site protocol for CCTA assessment. Level 3 readers were experienced radiologists who clinically assess CT scans with >5 years of work experience. Additionally, all scans were analysed by two independent level 2 readers from a separate institution (MS and MU—see acknowledgements). Level 2 readers were cardiologists in training. For every major epicardial vessel (right coronary artery (RCA), left main/left anterior descending coronary artery and left circumflex coronary artery), per cent stenosis and stenosis category were scored according to the recently updated the Coronary Artery Disease Reporting and Data System (CAD-RADS) expert consensus.¹⁰ A total of 29 (4.6%) vessel reads were unevaluable for level 2 reader 1, 48 (8.0%) readings were missing for level 2 reader 2 and 62 (9.9%) were missing for the level 3 reader.

Invasive coronary angiography

The invasive QCA was performed using a standard protocol in at least two orthogonal directions per evaluated coronary artery segment. Prior to contrast injection, 0.2 mL of intracoronary nitroglycerine was administered to induce epicardial coronary vasodilation. QCA measurements were performed by two experienced interventional cardiologists blinded to the non-invasive cardiac imaging results.

QCA was used as a reference standard for plaque characteristics. QCA is an established reference standard for determining the presence or absence of significant stenosis.¹¹ Society of Cardiovascular Computed Tomography (SCCT) guidelines note that CCTA is known to predict quantitative invasive angiography to within ±25% at the best—and incorporated in current guidelines such as CAD-RADS 2.0 stenosis.¹² The QCA reference standard was missing for 2 (0.3%) vessels, due to the incorrect coronary artery being evaluated.

AI-QCT analysis

The CCTA images were analysed using AI-QCT, AI-based software service from Cleerly (Denver, Colorado, USA), which has received clearance from the US Food and Drug Administration. The software uses an array of validated convolutional neural networks to assess image quality, segment and label coronary regions, evaluate the lumen wall, determine vessel contours and categorise plaque. AI-QCT has previously been validated through multicentre trials against expert consensus, QCA, fractional flow reserve and intravascular ultrasound (IVUS).^5–8 Initially, AI-QCT creates contouring for the centreline, lumen and outer vessel wall for all available phases, then selects the two best series for further analysis. After the system automatically segments and labels every vessel, plaques are identified and quantified based on HU attenuation values. To ensure accuracy, a qualified radiologic technologist reviews the AI analysis before approval.

The AI-QCT diameter stenosis is calculated using an interpolated reference diameter at the site of stenosis, which is different from the proximal reference diameter standard for visual interpretation and traditional quantitative coronary angiographic stenosis determination. In detail, per cent diameter stenosis within a segment was represented by 1 minus the ratio of the lumen diameter at the site of maximal obstruction divided by the estimated normal lumen diameter at this site by interpolation of the normal proximal and normal distal reference vessel×100. This method for determining stenosis severity has been previously reported to correlate better with invasively determined fractional flow reserve than traditional quantitative methods using a proximal reference.^{5 13} 95.2% of the scans were of Excellent or Good Quality. AI-QCT performance has been shown to be consistent throughout different scan qualities.¹⁴

Coronary segments with a diameter equal to or greater than 1.5 mm were considered using the adapted 18-segment model from the SCCT.¹² Segments less than 1.5 mm were not analysed. The degree of coronary stenosis was determined on a per-vessel basis according to the guidelines of the SCCT and then classified by the CAD-RADS.¹⁰ The AI-QCT diameter stenosis is calculated using an interpolated reference diameter at the site of stenosis. In detail, per cent diameter stenosis within a segment was represented by the ratio of the maximum lumen diameter at the site of maximal obstruction divided by the estimated lumen diameter at the site of maximal stenosis by interpolation of the normal proximal and normal distal reference vessel×100.^{5 6 15} Plaque volumes (mm³) were calculated for each coronary lesion and then summated to compute the total plaque volume at the patient level. AI-QCT is fully automated, with adjustments only in the quality check phase.

Statistical analysis

We evaluated the external performance of AI-QCT derived from full quantitative CCTA atherosclerosis evaluation (AI-QCT) versus level 2 and level 3 human readers to predict obstructive CAD when compared with blinded core lab QCA with ≥50% and ≥70% stenosis thresholds as reference standards. Receiver operating characteristic (ROC) analysis was employed to evaluate the discriminatory power of AI-QCT, the level 3 reader and the two level 2 readers for the gold standard of QCA.

Predictive performances were assessed using an ROC area under the curve (AUC) analysis in a per-patient and per-vessel analysis. For the per-patient analysis, the stenosis of the most obstructive of the three vessels was used. The diagnostic performance of the different readers was compared with the reference standard (QCA ≥50% and ≥70% stenosis) using an AUC analysis. AUCs were calculated using the Mann-Whitney U test statistic with associated Wald 95% CI. Furthermore, sensitivity, specificity, negative predictive value, positive predictive value and diagnostic accuracy were reported as simple frequencies as percentages with two-sided bootstrapped CIs on a per-patient level and two-sided bootstrapped 95% CIs with cluster sampling on a per-vessel level. Vessels with a missing QCA value were excluded from the analysis. For the current analysis, vessels with missing AI-QCT or reader values were considered positive for stenosis (intention to diagnose). Additionally, we stratified the dataset into patients above and below median total plaque volume based on plaque volume measured by AI-QCT, which has been shown to strongly correlate with IVUS measurements,¹⁶ to compare the performance of readers within these subsets.

Data are presented as mean±SD for normally distributed variables or median with IQR for non-normally distributed data. The normality of data distribution was assessed using histograms and probability plots. Categorical variables are expressed as absolute numbers and percentages. Independent sample t-tests, Wilcoxon tests, Mann-Whitney U-tests and Kruskal-Wallis tests were used where appropriate. All statistical analyses were performed using RStudio software V.4.3.0 (R Foundation, Vienna, Austria).

Results

Study population

The study population consisted of 208 patients who had a mean age 58±9 years, 37% were female. A total of 96 (46%) patients had hypertension, 99 (48%) patients were current smokers, 83 (40%) patients had hypercholesterolaemia, 33 (16%) patients had diabetes mellitus, while body mass index (BMI) was 27.0±3.7 (table 1). A total of 162 (78%) patients used statin therapy at the time of imaging. At baseline, the median total plaque volume for subjects was 218 mm³ (IQR 54–499). The total calcified plaque volume and non-calcified plaque volume were 77 mm³ (IQR 9–195) and 137 mm³ (IQR 38–278), respectively. The median low density plaque volume was 0.1 mm³ (IQR 0–3) and per cent atheroma volume was 9.1 mm³ (IQR 2.3–17.7). Every vessel with a QCA measurement was included in the analysis.

Table 1

Baseline characteristics

Characteristic	N=208
Demographics/risk factors
Age, years	58.1±8.7
Female sex	76 (37%)
Hypertension	96 (46%)
Hypercholesterolaemia	83 (40%)
Type II diabetes	33 (16%)
BMI, kg/m2	27.0±3.7
History of smoking	99 (48%)
Family history of CAD	107 (51%)
Medication use
Aspirin	182 (88%)
Beta blocker	135 (65%)
Calcium blocker	61 (29%)
Statin	162 (78%)
ACE inhibitor	41 (20%)
Nitrate	22 (11%)
AI-QCT characteristics
Total plaque volume, mm3	218.1 (54.6–499.1)
Per cent atheroma volume, %	9.1 (2.3–17.7)
Total calcified plaque volume, mm3	77.4 (8.9–194.8)
Non-calcified plaque volume, mm3	138.9 (37.9–277.7)
Low-density plaque volume, mm3	0.1 (0.0–3.0)
Remodelling index ≥1.10	103 (50%)
Diameter stenosis, %	50 (18.8–75.0)
Area stenosis, %	76.5 (31.5–94.0)
Presence of high-risk plaque	112 (54%)

Data are shown as mean±SD; n (%); median (IQR).

AI-QCT, artificial intelligence quantitative CT; BMI, body mass index; CAD, coronary artery disease; DM2, diabetes mellitus type 2.

Performance of AI-QCT and readers for per-patient presence of obstructive stenosis

To identify a ≥50% stenosis on QCA on a per-patient level, AI-QCT achieved an AUC of 0.91 (95% CI 0.87 to 0.95). The level 3 assessment resulted in an AUC of 0.77 (95% CI 0.70 to 0.83), which was significantly lower than AI-QCT (p<0.001). The two level 2 readers achieved an AUC of 0.79 (95% CI 0.72 to 0.85) and 0.76 (95% CI 0.69 to 0.83), which were also significantly lower than AI-QCT (both p<0.001; figure 1A). Sensitivity was 86%, 88%, 49% and 65% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, respectively, while specificity was 81%, 62%, 85% and 82% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, respectively (table 2). To identify a ≥70% stenosis on QCA on a per-patient level, AI-QCT achieved an AUC of 0.91 (95% CI 0.87 to 0.95). The level 3 assessment resulted in an AUC of 0.77 (95% CI 0.71 to 0.84), which was significantly lower than AI-QCT (p<0.001). The two level 2 readers achieved an AUC of 0.75 (95% CI 0.69 to 0.82) and 0.73 (95% CI 0.66 to 0.80), which was also lower than AI-QCT (both p<0.001; figure 1B). Sensitivity was 78%, 84%, 31% and 59% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, respectively, while specificity was 88%, 68%, 91% and 78% for AI-QCT, level 3, level 2 reader 1 and level 2 reader two, respectively (table 2).

Figure 1. Receiver operating characteristic (ROC) curves for the prediction of coronary artery stenosis on a per-patient level. (A) The ROC curves for predicting >=50% stenosis defined by quantitative coronary angiography per patient. (B) The ROC curves for predicting >=70% stenosis defined by QCA per patient. ***p<0.001. AI-QCT, atherosclerosis imaging-quantitative CT; AUC, area under the curve.

Table 2

Diagnostic performance metrics comparing AI-QCT and human readers for detecting coronary artery stenosis on a per-patient level (n=206)

	Test	Sensitivity	Specificity	PPV	NPV	Accuracy	AUC	P value
≥50%	AI-QCT	86 (78–92)	81 (73–87)	79 (70–86)	88 (80–93)	83 (78–88)	0.91 (0.87–0.95)	Ref
	Level 3	88 (80–94)	62 (52–70)	66 (57–74)	86 (77–92)	74 (67–79)	0.77 (0.7–0.83)	<0.001
	Level 2 reader 1	49 (39–59)	85 (77–90)	73 (61–82)	66 (58–74)	68 (62–74)	0.79 (0.72–0.85)	<0.001
	Level 2 reader 2	65 (55–74)	82 (74–88)	75 (65–83)	74 (65–81)	74 (68–80)	0.76 (0.69–0.83)	<0.001
≥70%	AI-QCT	78 (65–87)	88 (82–92)	71 (59–81)	91 (85–95)	85 (79–89)	0.91 (0.87–0.95)	Ref
	Level 3	84 (73–92)	68 (60–75)	51 (41–60)	92 (85–96)	72 (66–78)	0.77 (0.71–0.84)	<0.001
	Level 2 reader 1	31 (21–44)	91 (85–94)	56 (39–72)	77 (70–83)	74 (67–79)	0.75 (0.69–0.82)	<0.001
	Level 2 reader 2	59 (46–70)	78 (70–84)	51 (39–62)	83 (76–88)	72 (66–78)	0.73 (0.66–0.8)	<0.001

The table presents sensitivity, specificity, PPV, NPV, accuracy, AUC the ROC curve and p values for comparisons with AI-QCT as the reference standard. The metrics are provided for two thresholds of stenosis: ≥50% and ≥70%. Each metric is accompanied by a 95% CI. The AI-QCT shows higher sensitivity and accuracy compared with human readers for both stenosis thresholds, with significant p values indicating the differences. Level 2 readers generally demonstrate higher specificity but lower sensitivity.

AI-QCT, artificial intelligence quantitative CT; AUC, area under the curve; NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic.

Performance of AI-QCT and readers for per-vessel presence of obstructive stenosis

AI-QCT and the level 3 assessment demonstrated comparable performance in identifying ≥50% stenosis on QCA at the per-vessel level (AUC 0.86 (95% CI 0.82 to 0.89) and AUC 0.82 (95% CI 0.79 to 0.86), p=0.098). Both level 2 readers exhibited lower AUCs of 0.69 (95% CI 0.65 to 0.74), which were lower than AI-QCT (both p<0.001; figure 2A). Sensitivity was 66%, 75%, 28% and 40% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, while specificity was 89%, 77%, 89% and 87% for AI-QCT, level 3, level 2 reader 1 and level 2–3, respectively (table 3). For a ≥70% stenosis on QCA at the per-vessel level, AI-QCT again performed similarly to the level 3 reader, achieving an AUC of 0.84 (95% CI 0.79 to 0.89) and 0.82 (95% CI 0.79 to 0.86), respectively (p=0.500). Both level 2 readers showed significantly lower AUCs of 0.69 (95% CI 0.63 to 0.75) compared with AI-QCT (both p<0.001; figure 2B). Sensitivity was 56%, 66%, 19% and 41% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, respectively, while specificity was 93%, 84%, 94% and 87% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, respectively (table 3).

Figure 2. Receiver operating characteristic (ROC) curves for the prediction of coronary artery stenosis on a per-vessel level. (A) The ROC curves for predicting >=50% stenosis defined by quantitative coronary angiography per patient. (B) The ROC curves for predicting >=70% stenosis defined by QCA per patient. ***p<0.001. AI-QCT, atherosclerosis imaging-quantitative CT; AUC, area under the curve.

Table 3

Diagnostic performance metrics comparing AI-QCT and human readers for detecting coronary artery stenosis on a per-vessel level (n=622)

	Test	Sensitivity	Specificity	PPV	NPV	Accuracy	AUC	P value
≥50%	AI - QCT	66 (59–73)	89 (86–92)	70 (62–76)	88 (84–90)	83 (80–86)	0.86 (0.82–0.89)	Ref
	Level 3	75 (68–81)	77 (72–80)	55 (48–61)	89 (86–92)	76 (73–79)	0.82 (0.79–0.86)	0.098
	Level 2 reader 1	28 (22–35)	89 (85–91)	47 (38–57)	77 (73–80)	72 (68–75)	0.69 (0.65–0.74)	<0.001
	Level 2 reader 2	40 (33–47)	87 (83–90)	53 (44–61)	79 (76–83)	74 (70–77)	0.69 (0.65–0.74)	<0.001
≥70%	AI-QCT	56 (45–65)	93 (91–95)	57 (47–67)	93 (90–94)	88 (85–90)	0.84 (0.79–0.89)	Ref
	Level 3	66 (55–75)	84 (80–87)	41 (33–49)	94 (91–95)	81 (78–84)	0.82 (0.79–0.86)	0.500
	Level 2 reader 1	19 (12–28)	94 (91–95)	33 (22–47)	87 (84–90)	83 (80–86)	0.69 (0.63–0.75)	<0.001
	Level 2 reader 2	41 (32–51)	87 (84–89)	35 (26–44)	90 (87–92)	80 (77–83)	0.69 (0.63–0.75)	<0.001

The table presents sensitivity, specificity, PPV, NPV, accuracy, AUC the ROC curve and p values for comparisons with AI-QCT as the reference standard. The metrics are provided for two thresholds of stenosis: ≥50% and ≥70%. Each metric is accompanied by a 95% CI. The AI-QCT shows higher sensitivity and accuracy compared with human readers for both stenosis thresholds, with significant p values indicating the differences. Level 2 readers generally demonstrate higher specificity but lower sensitivity.

AI-QCT, artificial intelligence quantitative CT; AUC, area under the curve; NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic.

Concordance with QCA measurements

The frequency distribution of CAD-RADS scores, ranging from 0 to 5, as evaluated by QCA, AI-QCT, and the three human readers, was assessed for differences between the four readers using χ² tests. The results revealed statistically significant differences among the readers (p<0.001 for all comparisons, figure 3). As assessed by AI-QCT, 104 (50%) patients had the obstructive disease (CAD-RADS≥3), compared with 128 (62%), 63 (30%), 82 (39%) for the level 3 readers and level 2 readers, respectively. The highest concordance with QCA for the individual CAD-RADS categories was achieved by AI-QCT (98/206; 48%). For the level 3 reader, the concordance was 30% (62/206). The two level 2 readers achieved a concordance of 19% and 24%, respectively (figure 4). On a per-vessel basis, 161 (26%) patients, as assessed by AI-QCT, had obstructive disease, compared with 173 (28%), 70 (11%) and 79 (13%), for the level 3 readers and two level 2 readers, respectively. The highest concordance with QCA on a per-vessel basis was achieved by AI-QCT (241/622; 39%, p=0.002). For the level 3 reader, the concordance with QCA was 26% (158/622, p<0.001). The two level 2 readers achieved a concordance of 15% and 19% with QCA, respectively (both p<0.001).

Figure 3. Distribution of per-patient CAD-RADS scores for AI-QCT and clinical readers. The histograms display the frequency of CAD-RADS scores assigned by QCA, AI-QCT and level 3 (L3) and level 2 readers (L2). Each panel represents the CAD-RADS score distribution for the particular reader or modality (0: no stenosis/no plaque, 1: 0 reader 24% stenosis, 2: 25%-49% stenosis, 3: 50%-69% stenosis, 4: 70%-99% stenosis, 5: 100% stenosis). AI-QCT, artificial intelligence-QCT; CAD-RADS, Coronary Artery Disease-Reporting and Data System; QCA, quantitative coronary angiography.

Figure 4. Interobserver correlation heatmaps comparing CAD-RADS score assessments. (A) The level of agreement between quantitative coronary angiography (QCA) as the gold standard versus intelligence-guided algorithm for assessment of coronary stenosis (AI-QCT). (B) The level of agreement between QCA as the gold standard versus the level 3 reader (L3). (C) The level of agreement between QCA as the gold standard versus the level 2 reader 1 (L2-1). (D) The level of agreement between QCA as the gold standard versus the level 2 reader 2 (L2-2). In all panels, each cell represents the frequency of agreement counts, shaded according to the legend, across the range of possible CAD-RADS scores (0-5) for each rater against the gold standard. AI-QCT, artificial intelligence quantitative CT; CAD-RADS, Coronary Artery Disease Reporting and Data System; QCA, quantitative coronary angiography.

Performance of AI-QCT and readers in patients with above and below median plaque volume

In discerning a ≥50% stenosis on QCA among patients with total plaque volume above the median, AI-QCT achieved an AUC of 0.88 (95% CI 0.82 to 0.94). In comparison, the level 3 assessment yielded a slightly lower AUC of 0.77 (95% CI 0.68 to 0.86; p=0.015). Both level 2 readers achieved AUCs of 0.72 (95% CI 0.62 to 0.82) and 0.72 (95% CI 0.62 to 0.83), which were lower than AI-QCT (p=0.007 and p=0.004; figure 5A). Sensitivity was 93%, 82%, 50% and 64% for AI-QCT, level 3, level 2 reader 1 and level 2 reader 2, while specificity was 50%, 50%, 82% and 82% for AI-QCT, level 3, level 2 reader 1 and level 2–3, respectively (table 4). For patients with plaque volumes below the median, AI-QCT achieved an AUC of 0.83 (95% CI 0.72 to 0.95), which was similar to the level 3 assessment which achieved an AUC of 0.81 (95% CI 0.69 to 0.94; p=0.575). Both level two readers demonstrated significantly lower AUCs of 0.66 (95% CI 0.52 to 0.80) and 0.63 (95% CI 0.49 to 0.77) compared with AI-QCT (p=0.032 and p=0.007; figure 5B). Sensitivity was 60%, 75%, 20% and 15% for AI-QCT, level 3, level 2 reader 1 and level 2–3, while specificity was 92%, 83%, 94% and 99% for AI-QCT, level 3, level 2 reader 1 and level 2–3, respectively (table 4).

Figure 5. Receiver operating characteristic (ROC) curves for the prediction of coronary artery stenosis for patients with plaque values above and below the median. (A) The ROC curves for predicting >=50% stenosis defined by quantitative coronary angiography on a per-patient basis for patients with total plaque volume above the median (218.1 mm 3 ; A) and below the median (B). *p<0.05; **p<0.01. AI-QCT, atherosclerosis imaging-quantitative CT; AUC, area under the curve.

Table 4

Performance of AI-QCT and readers in patients with above and below median plaque volume on a per-patient level (n=206)

	Test	Sensitivity	Specificity	PPV	NPV	Accuracy	AUC	P value
Above median	AI-QCT	93 (85–98)	50 (33–67)	84 (74–90)	74 (51–89)	82 (73–88)	0.88 (0.82–0.94)	Ref
	Level 3	82 (71–89)	50 (33–67)	82 (71–89)	50 (33–67)	73 (64–81)	0.77 (0.68–0.86)	0.015
	Level 2 reader 1	50 (39–61)	82 (64–93)	88 (75–95)	38 (27–50)	59 (49–68)	0.72 (0.62–0.82)	0.007
	Level 2 reader 2	64 (53–74)	82 (64–93)	91 (80–96)	46 (33–60)	69 (60–77)	0.72 (0.62–0.83)	0.004
Below median	AI-QCT	60 (39–78)	92 (84–96)	63 (41–81)	91 (82–95)	86 (77–91)	0.83 (0.72–0.95)	Ref
	Level 3	75 (53–89)	83 (74–90)	52 (34–69)	93 (85–97)	82 (73–88)	0.81 (0.69–0.94)	0.575
	Level 2 reader 1	20 (7–42)	94 (86–98)	44 (19–73)	83 (74–89)	80 (71–86)	0.66 (0.52–0.80)	0.032
	Level 2 reader 2	15 (4–37)	99 (93–100)	75 (29–97)	83 (74–89)	83 (74–89)	0.63 (0.49–0.77)	0.007

The table presents sensitivity, specificity, PPV, NPV, accuracy, AUC the ROC curve and p values for comparisons with AI-QCT as the reference standard. The metrics are provided for a threshold of stenosis of ≥50% and divided into patients with plaque values above and below the median. Each metric is accompanied by a 95% CI. The AI-QCT shows higher sensitivity and accuracy compared with human readers for both groups, with significant p values indicating the differences.

AI-QCT, artificial intelligence quantitative CT; AUC, area under the curve; NPV, negative predictive value; PPV, positive predictive value; ROC, receiver operating characteristic.

Discussion

The present study aimed to compare the diagnostic accuracy of AI-QCT with visual assessment by human CCTA readers of varying experience levels. AI-QCT showed the highest concordance with invasive QCA and performed similarly to experienced level 3 readers. AI-QCT outperformed level 2 readers in quantifying CAD-RADS. When stratified by plaque burden, AI-QCT outperformed the level 3 readers in those with above-median plaque burden (ΔAUC 0.11), while there was no significant difference between AI-QCT and the level 3 readers in patients with below-median plaque volume. Collectively, these results illustrate the potential, reliable role of AI in standardising and improving the accuracy of CCTA interpretation for stenosis, especially in those with obstructive CAD.

Our study builds on prior work from Choi et al that demonstrated excellent diagnostic accuracy of AI-QCT compared with level 3 readers, with high sensitivity and specificity for coronary stenosis.⁶ In our study, we extend this by comparing AI-QCT to level 2 readers, highlighting consistent AI performance across varying extents of CAD and emphasising the potential benefit of standardised AI-guided CCTA analysis in centres lacking highly experienced readers.⁶ Another study by Liu et al reported that deep learning powered CCTA analysis achieved good diagnostic performance, comparable to intermediate or experienced readers and substantially shortened CCTA postprocessing time for the detection of CAD.¹⁷ Adding to these findings, our study includes a direct comparison between human reading and AI in assessing coronary artery stenosis grade and CAD-RADS classification, providing a more comprehensive evaluation of AI’s role in diagnosing coronary artery stenosis. Finally, examination of the large, multicentre PROMISE study has highlighted both the overestimation of stenosis in over 40% of patients by less experienced readers⁴ and the proficiency increase in CCTA interpretation through reader experience.^18–20 This highlights the need for a multifaceted approach that encompasses both enhanced education as well as incorporation of well-validated AI approaches into the clinical workflow. Our study aligns with this issue by pointing out the distinct difference in diagnostic accuracy between level 2 and level 3 readers. Therefore, the most prominent benefit of AI-QCT implementation through a standardised approach to CCTA analysis may be in centres lacking highly experienced readers.

Importantly, incorporation of AI-guided CCTA analysis may also optimise human reader proficiency and enhance diagnostic confidence. Considering the present findings that AI-QCT accurately diagnoses the presence of obstructive CAD and previous findings showing the prognostic importance of AI-QCT analysis. The recently published CERTAIN (Changes in CAD Diagnosis, Imaging, Intervention and Medication with AI-QCT) trial was a multicentre crossover study of 5 expert sites and 750 consecutive adult patients comparing the clinical utility of routine AI-QCT implementation versus conventional visual interpretation.¹⁴ The study found that AI-QCT associated with change in diagnosis or management in the majority of patients (428; 57.1%; p<0.001) and largely improved physician’s confidence in CCTA results in the implementation of AI-guided quantitative CCTA analysis may increase confidence of human readers facilitating the accurate selection of downstream testing and preventive medical therapy.

The stratification of findings based on plaque volume provided insights into the performance of AI-QCT and human readers within specific CAD subsets. For those with plaque volumes above the median, and thus relatively extensive CAD, AI-QCT demonstrated superior agreement with QCA compared with level 3 and level 2 readers in identifying significant stenosis on QCA at the per-patient level. Among patients with plaque volumes below the median, AI-QCT maintained good diagnostic accuracy, indicating its consistent performance irrespective of varying plaque volumes. However, while outperforming the level 2 readers, AI-QCT performed similarly to level 3 readers in the subset of patients with less extensive CAD. This is consistent with Liu et al’s findings that AI can substantially improve diagnostic accuracy, even in the presence of challenging factors such as image quality and calcification burden. These findings reinforce the notion that in patients with extensive CAD, interobserver and intraobserver variability might have a more profound impact on CCTA assessment than in patients with less disease severity. Also, AI-QCT’s advanced algorithms may be better equipped to handle and integrate the increased variability and heterogeneity seen in larger plaque volumes.

As the diagnostic landscape evolves, the integration of AI not only promises improved accuracy but also signifies a shift towards more standardised and efficient clinical assessments, thereby enhancing patient care outcomes. Future research should delve into exploring the impact of AI integration on long-term patient outcomes, assessing the generalisability of AI across diverse patient populations, and refinement of AI algorithms based on evolving technological advancements. Additionally, ongoing efforts to define and refine competency-based metrics in CCTA interpretation will facilitate further optimisation of diagnostic accuracy and clinical efficacy.

Limitations

Several limitations require further attention. First, although one of the largest multimodality imaging studies to date, PACIFIC-1 was a single-centre study with a relatively limited sample size. Second, due to motion or other artefacts, the clinical readers were not able to assess all coronary vessels for the presence of obstructive stenosis. Level 3 readers had more unevaluable vessels than level 2 readers; we believe that the experienced radiologists take a conservative approach to interpretation. A radiologist may opt to classify a vessel as unevaluable rather than risk a potentially inaccurate assessment. The absence of segment involvement score and segment stenosis score was also a limitation. Prior research has assessed the higher prognostication of AI-QCT when compared with both visual estimation and semiquantitative analysis of segment involvement score.¹⁵

Finally, the study did not address the clinical consequences and downstream effects of different treatment decisions based on the CCTA interpretation. The impact of AI-QCT on the clinical care pathway needs to be addressed in future studies.

Conclusions

Compared with clinical assessments of CCTA, AI-QCT-defined stenosis demonstrated the highest concordance with an invasive QCA reference standard. Especially in patients with extensive CAD, AI-QCT had similar accuracy to level 3 and higher accuracy in comparison to level 2 CCTA readers in prediction of obstructive stenosis. Implementing AI-QCT in routine clinical practice may improve reproducibility and reliability of CCTA assessment, limiting the need for excess downstream testing while improving physician confidence.

We would like to thank Dr. Muhammad Umer and Dr. Matthew Shotwell (University of Louisville, USA) for their analysis of the CCTA scans.

Data availability statement

Data are available on reasonable request. This is a substudy of the PACIFIC trial with ClinicalTrials.gov identifier: NCT01521468.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by Amsterdam UMC Medical Ethics Committee (PACIFIC trial; 2011/209). Participants gave informed consent to participate in the study before taking part.

Footnote

X @rachelbernardo_, @NickNurmohamed

Contributors All authors were involved in planning. RB and NSN processed the experimental data, performed the analysis, drafted the manuscript and designed the figures. ADC aided in interpreting the results and worked on the manuscript. All authors discussed the results and commented on the manuscript. PK is the guarantor.

Funding This research was supported by the European Atherosclerosis Society and Heart Foundation (Grant number: 03-007-2023-0068).

Competing interests NSN reports grants from the Dutch Heart Foundation (Dekker 03-007-2023-0068), European Atherosclerosis Society (2023), research funding/speaker fees from Cleerly, Daiichi Sankyo and Novartis and is co-founder of Lipid Tools. JKM and JE are employees of Cleerly. ADC reports grant support from GW Heart and Vascular Institute, equity in Cleerly and consulting with Siemens Healthineers, Amgen and Cleerly. PK has received research grants from Cleerly and HeartFlow. The other authors report no conflicts of interest.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Kelion AD, Nicol ED. The rationale for the primacy of coronary CT angiography in the National Institute for Health and Care Excellence (NICE) guideline (CG95) for the investigation of chest pain of recent onset. J Cardiovasc Comput Tomogr 2018; 12: 516–22. doi:10.1016/j.jcct.2018.09.001

² Budoff MJ, Lakshmanan S, Toth PP, et al. Cardiac CT angiography in current practice: An American society for preventive cardiology clinical practice statement^✰ Am J Prev Cardiol 2022; 9: 100318. doi:10.1016/j.ajpc.2022.100318

³ Jonas RA, Weerakoon S, Fisher R, et al. Interobserver variability among expert readers quantifying plaque volume and plaque characteristics on coronary CT angiography: a CLARIFY trial sub-study. Clin Imaging 2022; 91: 19–25. doi:10.1016/j.clinimag.2022.08.005

⁴ Lu MT, Meyersohn NM, Mayrhofer T, et al. Central Core Laboratory versus Site Interpretation of Coronary CT Angiography: Agreement and Association with Cardiovascular Events in the PROMISE Trial. Radiology 2018; 287: 87–95. doi:10.1148/radiol.2017172181

⁵ Griffin WF, Choi AD, Riess JS, et al. AI Evaluation of Stenosis on Coronary CTA, Comparison With Quantitative Coronary Angiography and Fractional Flow Reserve: A CREDENCE Trial Substudy. JACC Cardiovasc Imaging 2023; 16: 193–205. doi:10.1016/j.jcmg.2021.10.020

⁶ Choi AD, Marques H, Kumar V, et al. CT Evaluation by Artificial Intelligence for Atherosclerosis, Stenosis and Vascular Morphology (CLARIFY): A Multi-center, international study. J Cardiovasc Comput Tomogr 2021; 15: 470–6. doi:10.1016/j.jcct.2021.05.004

⁷ Lipkin I, Telluri A, Kim Y, et al. Coronary CTA With AI-QCT Interpretation: Comparison With Myocardial Perfusion Imaging for Detection of Obstructive Stenosis Using Invasive Angiography as Reference Standard. AJR Am J Roentgenol 2022; 219: 407–19. doi:10.2214/AJR.21.27289

⁸ Kim Y, Choi AD, Telluri A, et al. Atherosclerosis Imaging Quantitative Computed Tomography (AI-QCT) to guide referral to invasive coronary angiography in the randomized controlled CONSERVE trial. Clin Cardiol 2023; 46: 477–83. doi:10.1002/clc.23995

⁹ Danad I, Raijmakers PG, Driessen RS, et al. Comparison of Coronary CT Angiography, SPECT, PET, and Hybrid Imaging for Diagnosis of Ischemic Heart Disease Determined by Fractional Flow Reserve. JAMA Cardiol 2017; 2: 1100–7. doi:10.1001/jamacardio.2017.2471

¹⁰ Cury RC, Leipsic J, Abbara S, et al. CAD-RADS^TM 2.0 - 2022 Coronary Artery Disease-Reporting and Data System: An Expert Consensus Document of the Society of Cardiovascular Computed Tomography (SCCT). J Cardiovasc Comput Tomogr 2022; 16: 536–57. doi:10.1016/j.jcct.2022.07.002

¹¹ Arbab-Zadeh A, Texter J, Ostbye KM, et al. Quantification of lumen stenoses with known dimensions by conventional angiography and computed tomography: implications of using conventional angiography as gold standard. Heart 2010; 96: 1358–63. doi:10.1136/hrt.2009.186783

¹² Leipsic J, Abbara S, Achenbach S, et al. SCCT guidelines for the interpretation and reporting of coronary CT angiography: a report of the Society of Cardiovascular Computed Tomography Guidelines Committee. J Cardiovasc Comput Tomogr 2014; 8: 342–58. doi:10.1016/j.jcct.2014.07.003

¹³ Nurmohamed NS, Min JK, Anthopolos R, et al. Atherosclerosis quantification and cardiovascular risk: the ISCHEMIA trial. Eur Heart J 2024; 45: 3735–47. doi:10.1093/eurheartj/ehae471

¹⁴ Nurmohamed NS, Cole JH, Budoff M, et al. Impact of Atherosclerosis Imaging-Quantitative Computed Tomography on Diagnostic Certainty, Downstream Testing. Coron Revasc Med Ther 2024. doi:10.1093/ehjci/jeae029

¹⁵ Nurmohamed NS, Bom MJ, Jukema RA, et al. AI-Guided Quantitative Plaque Staging Predicts Long-Term Cardiovascular Outcomes in Patients at Risk for Atherosclerotic CVD. JACC Cardiovasc Imaging 2024; 17: 269–80. doi:10.1016/j.jcmg.2023.05.020

¹⁶ Omori H, Matsuo H, Fujimoto S, et al. Determination of lipid-rich plaques by artificial intelligence-enabled quantitative computed tomography using near-infrared spectroscopy as reference. Atherosclerosis 2023; 386: S0021-9150(23)05284-X. doi:10.1016/j.atherosclerosis.2023.117363

¹⁷ Liu CY, Tang CX, Zhang XL, et al. Deep learning powered coronary CT angiography for detecting obstructive coronary artery disease: The effect of reader experience, calcification and image quality. Eur J Radiol 2021; 142: 109835. doi:10.1016/j.ejrad.2021.109835

¹⁸ Herzog C, Kerl JM, De Rosa S, et al. Influence of observer experience and training on proficiency in coronary CT angiography interpretation. Eur J Radiol 2013; 82: 1240–7. doi:10.1016/j.ejrad.2013.02.037

¹⁹ Choi AD, Blankstein R. Becoming an Expert Practitioner: The Lifelong Journey of Education in Cardiovascular Imaging. JACC Cardiovasc Imaging 2021; 14: 1594–7. doi:10.1016/j.jcmg.2021.03.006

²⁰ Chow BJW, Yam Y, Alenazy A, et al. Are Training Programs Ready for the Rapid Adoption of CCTA?: CBME in CCTA. JACC Cardiovasc Imaging 2021; 14: 1584–93. doi:10.1016/j.jcmg.2021.01.040

²¹ Bernardo R, Nurmohamed N, Bom M, et al. Diagnostic Accuracy In Coronary CT Angiography Analysis: Artificial Intelligence Versus Visual Assessment. J Cardiovasc Comput Tomogr 2024; 18: S14. doi:10.1016/j.jcct.2024.05.032