An automated quality assurance system with deep learning for small cube-ball phantom localization in noisy megavoltage images

Abstract

To enhance efficiency and minimize errors, we automated the quality assurance (QA) process in radiation oncology, specifically laser localization. Additionally, we explored the use of a convolutional neural network (CNN) to enhance the detection of small cube-ball phantoms in noisy images. Laser localizations were measured manually on the acquired images. To automate the QA workflow, we developed a Linux server equipped with database and web servers. Digital Imaging and Communications in Medicine (DICOM) files were retrieved 40 times for 10 linear accelerators (LINACs). The center of the cube-ball phantoms was estimated through Gaussian fitting. We applied CNN using 6,968 stored results to improve the estimation performance in noisy megavoltage (MV) images. Subsequently, both analysis time and accuracy were compared. Our hospital has been employing the automated QA system since 2018, notably reducing the time for laser localization from 30 min to just 1 min. The average and standard deviation (SD) of inter-observer variability in the X- and Y-axes were 0.06 ± 0.01 mm and 0.05 ± 0.01 mm, respectively. Absolute differences between manual assessment and Gaussian fitting presented average and SD values of 0.40 ± 0.51 mm and 0.23 ± 0.24 mm, respectively. In contrast, absolute differences between manual assessment and CNN presented average and SD values of 0.12 ± 0.10 mm and 0.11 ± 0.09 mm, respectively. Overall, the automated QA system significantly hastened procedures in our large hospital and improved the estimation of the cube-ball phantom’s position in noisy images through deep learning.

Introduction

Quality assurance (QA) is pivotal in ensuring precise treatment delivery and patient safety in radiation oncology. It encompasses various processes and techniques employed to verify and validate the accuracy and effectiveness of radiation therapy procedures. In recent years, the utilization of in-room imaging systems has become prevalent for patient-specific intensity-modulated radiation therapy (IMRT) QA and for machine QA to ensure mechanical stability [1, 2, 3–4]. Kilovoltage (kV) X-ray imaging, known for its relatively low energy, is commonly utilized in patient setup. This imaging technique provides valuable information about the position and localization of the patient’s anatomy concerning the planned treatment fields. Radiographic images and cone-beam computed tomography (CBCT) scans are used to precisely localize tumors, facilitating accurate treatment planning and delivery. Fluoroscopy, which provides real-time imaging, is utilized for continuous monitoring of tumor positions. Megavoltage (MV) images obtained using an electronic portal imaging device (EPID) are used for verifying patient positioning, ensuring accurate field placement, and confirming proper tumor localization. Furthermore, MV images are used in the pretreatment QA process for IMRT by comparing the delivered dose to the intended plan. Beyond patient-specific QA, in-room imaging systems are crucial in machine QA to ensure mechanical stability and dosimetric accuracy. Various evaluations, including the Winston–Lutz test, multileaf collimators (MLC) and jaw position, as well as measurements of output, symmetry, and beam quality, are performed using these imaging systems.

To expedite time-consuming QA procedures, automated QA analysis systems, such as DoseLab (Varian Medical Systems), RIT (Radiological Imaging Technology), and SafetyNet (University of Michigan), have been developed. DoseLab and RIT analyze Digital Imaging and Communications in Medicine (DICOM) files acquired from in-room imaging systems for machine and patient QA, following standards like TG-40/TG-142 and TG-51 [5]. SafetyNet streamlines QA workflows by automating tasks such as plan checks, analysis of treatment history parameters for weekly chart checks, and analysis of Winston–Lutz test with DoseLab [6].

We use the metal ball detection technique of the Winston–Lutz test for the daily QA practice of laser localization. kV and MV images are acquired from a cube-ball phantom, which is placed using the in-room lasers [7]. Acquired images are automatically retrieved and the ball position is estimated through image processing. Varian cubic phantoms are used for daily QA practices in our institution as the quantity of Winston–Lutz phantom is insufficient for ten LINACs. The diameter of the metal ball in the Varian cubic phantom is 2 mm, in contrast to the 5–6.25 mm metal ball in the Winston–Lutz phantom. However, conventional automated QA systems are tailored for analyzing cube phantoms with larger ball diameters (> 5 mm), resulting in limited performance when applied to cube phantoms with smaller ball diameters (2 mm) in noisy MV images. To address the issue of noisy MV image quality, modern treatment machines such as TrueBeam and VitalBeam facilitate 2.5 MV imaging, providing an enhanced contrast-to-noise ratio (CNR) [8, 9–10]. However, older machines, such as Trilogy and Clinac, lack this feature.

This study presents an automated QA system for laser localization using kV/MV images of a cube phantom with a small ball. We developed this automated QA system to reduce the time required to analyze many different linear accelerators (LINACs) in a large hospital. Additionally, we employed deep learning using a convolutional neural network (CNN) to improve estimation performance, especially in noisy images.

Methods

Conventional QA analysis of laser localization

The laser localization of in-room imaging systems is determined by images of a cube-ball phantom [11]. When the cube-ball phantom is aligned with in-room lasers, laser localization is accessed based on the center positioning of the metal sphere within the phantom images, as shown in Fig. 1. Both kV (70 kVp) and MV (6 MV) images are concurrently acquired at gantry angles of 0° and 90°. The localizations of the horizontal and vertical axes of both lateral lasers are estimated as the averaged centers of the kV image at a 270° gantry angle and that of the MV image at a 90° gantry angle. Localization of the vertical axes of the sagittal and lateral lasers is estimated as the averaged centers of kV and MV images at a 0° gantry angle.

[See PDF for image]

Fig. 1

Cube-ball phantom aligned with in-room lasers

In our conventional laser localization evaluation process, images are manually searched and analyzed through an “offline review” application (Varian ARIA 13.6), requiring the repetition of this procedure 40 times for ten LINACs, including TrueBeam, VitalBeam, Trilogy, and Clinac. MV images are acquired using various EPID panels, such as Varian aS1000 and aS1200, and at varying energies, such as 2.5 MV, 4 MV, 6 MV, and 15 MV [12]. The number of MV images acquired from each EPID panel and at each imaging energy is summarized in Tables 1 and 2.

Table 1. Specifications of the electronic portal imaging device (EPID) panel and the number of megavoltage (MV) images captured using different modes and panels

	aS1000		aS1200
Active area (cm2)	30 × 40		40 × 40
Total pixel matrix	384 × 512	768 × 1,024	640 × 640	1,280 × 1,280
Pixel size (mm)	0.784	0.392	0.672	0.336
Total images	438	2,583	3,030	917
Training set	227	1,314	1,511	432
Test set	211	1,269	1,519	485

Table 2. The number of MV images captured using different imaging energies

	2.5 MV	4 MV	6 MV	15 MV
Total images	1,048	194	5,642	84
Training set	541	93	2,805	45
Test set	507	101	2,837	39

Automated analysis using image processing

We developed an automated QA system named “MP.Lab” using in-house programs written in MATLAB (MathWorks Inc.) and deployed on a Linux server (Ubuntu 18.04 LTS, Intel i7-8700 3.20 GHz, 32G RAM). This system comprises an MS-SQL database server (Microsoft, 15.0.4178.1–3, Express), a modified DICOM toolkit (DCMTK, 3.6.4) server, and an Apache2 web server with PHP and JavaScript. This automated system can connect to the ARIA oncology information system, which stores information on radiotherapy plans, treatment records, and acquired images. DICOM unique identifier (UID) is retrieved using pre-defined structured query language (SQL) programs. This system automatically downloads DICOM files using retrieved UID and analyzes them using image processing programs. This analysis is performed automatically every morning using a job scheduler and can be performed upon request, as shown in Fig. 2.

[See PDF for image]

Fig. 2

Schematic representation of the automation system

The center of the downloaded DICOM images is cropped to 3 × 3 mm and low-pass filtered using a moving average method to remove noise. Subsequently, these cropped images are spline-interpolated to a pixel size of 0.1 mm to normalize the spatial resolutions of different EPIDs. The center of the cube phantom is estimated as the mean of the Gaussian fitting of projections in both X- and Y-axes [13]. The daily estimations of laser localizations are stored in the database and can be accessed and reviewed on the website, as shown in Fig. 3.

[See PDF for image]

Fig. 3

Estimations using Gaussian fitting on kilovoltage (kV) images (left) and using convolutional neural network (CNN) on megavoltage (MV) images (right). Images were cropped to a 4 mm × 4 mm size for review

Experienced medical physicists conduct manual assessments via the web interface. A total of 6,968 MV images and their manual assessments are stored on the server, with 3,023 images measured manually more than two times. Averaged manual assessments serve as the standard for the analysis. To assess inter-observer variability, we calculate the average of individual absolute differences from the average of manual assessments using the following formula:$$\frac{{\sum }_{i=1}^k\left|M_{i,j},-,\frac{{\sum }_{j=1}^n{M}_{i,j}}{n}\right|}{k},$$where M_i,j indicates i^th manual assessment of j^th image, n represents the number of manual assessments of an image, and k represents the number of test images.

Deep learning applied to MV images

Varian cubic phantoms are used for daily laser localization QA at our hospital. The diameter of the metal sphere at the center of the Varian cubic phantom is 2 mm, which is smaller than the Winston–Lutz phantom (typically 5–6 mm diameter). Consequently, the visibility of the metal ball in the Varian cubic phantom is inferior to the Winston–Lutz phantom, as shown in Fig. 4. Since conventional analysis programs such as DoseLab and RIT are designed to use the Winston–Lutz phantom, the parameters for detecting the Varian cubic phantom require manual adjustment.

[See PDF for image]

Fig. 4

The 6 MV images of a Varian cubic phantom (upper) and Winston–Lutz phantom (lower)

To enhance the estimation performance in noisy MV images, we employed CNNs for estimating the position of the cube phantom. The cropped images were trained with a modified LeNet architecture with convolution layers, normalization layers, rectified linear unit (ReLU) layers, and fully connected layers. Dropout was used to regularize the deep neural networks. Training using a stochastic gradient descent with momentum (SGDM) optimizer was continued until saturation (min batch: 100, max epoch: 200).

Half of the stored manual assessments is randomly selected for training the CNNs in X- and Y-axes, and the other half is used to test the performance of the trained neural networks. The parameters of the training and test data set are summarized in Table 1 and Table 2. To assess the performance of the neural networks, we calculate the absolute difference between the predictions of the neural network and the average of the manual assessments using the following formula:$$\frac{{\sum }_{i=1}^k\left|P_i,-,\frac{{\sum }_{j=1}^n{M}_{i,j}}{n}\right|}{k},$$where P_i indicates the prediction of the neural network from i^th image, M_i,j indicates j^th manual assessment of i^th image, n represents the number of manual assessments of an image, and k represents the number of test images.

Results

Manual assessments took approximately 30 min for ten LINACs (40 images). In contrast, the developed QA system automatically downloaded and analyzed the images within 1 min. Furthermore, the results were stored in the database and could be accessed and reviewed on the website from anywhere in the hospital.

Regarding inter-observer variability, the average differences between the manual assessments and their means were 0.03 mm and 0.02 mm for the X- and Y-axes, respectively, based on 3,023 multiple manual assessments. In the test set of 3,484 MV images, using the Gaussian fitting, the average of the absolute differences between the Gaussian fitting and the averaged manual assessment were 0.40 ± 0.51 mm and 0.23 ± 0.24 mm in the X- and Y-axes, respectively. Using CNN, the average of the absolute differences between CNN and the averaged manual assessment were 0.12 ± 0.10 mm and 0.11 ± 0.09 mm in the X- and Y-axes, respectively. To compare the estimation performance, two-sample t tests between Gaussian fitting and CNN were conducted. CNN demonstrated a significantly lower average of the absolute differences compared to Gaussian fitting (p < 0.001). Figure 5 represents normalized histograms of the absolute differences between CNN and Gaussian fitting compared to the average of manual assessments in the X- and Y-axes. The results are summarized in Table 3.

[See PDF for image]

Fig. 5

Absolute differences between manual assessments and automated analyses in the X (upper) and Y (lower) axes

Table 3. Differences in the estimation methods compared to the average of manual assessments

	X-axis	Y-axis
Manual assessment	0.06 ± 0.01 mm	0.05 ± 0.01 mm
Gaussian fitting	0.40 ± 0.51 mm	0.23 ± 0.24 mm	p < 0.001
Convolutional neural network	0.12 ± 0.10 mm	0.11 ± 0.09 mm

Discussion

The automated system, developed and operational at our hospital since July 2018, has considerably enhanced operational efficiency. This improvement stems from its robust automation features and web accessibility. The system automatically collects QA data, such as DICOM images stored within ARIA (Varian), autonomously processes it using image processing algorithms, and stores the results in the SQL database. Physicists can access archived QA results via the web, enabling manual review and adjustments when necessary. Furthermore, these archived QA results are subsequently employed as the labeled dataset for the deep learning analysis.

The application of deep learning analysis has substantially improved the accuracy of automated estimation, even in noisy MV images of cube-ball phantoms containing a small ball. While the analytical method, which uses Gaussian fitting on kV images with a superior CNR compared to MV images, did not encounter any issues, its application to noisy MV images decreased the accuracy of estimation. To reduce noise, the Gaussian fitting of the X-axis was computed through its projection onto the Y-axis. This pre-processing step successfully mitigates noise; however, it unintentionally amplifies the background pattern along the Y-axis. Therefore, imaging pattern of EPID, which can be shown in Fig. 3, makes the performance of Gaussian fitting worse. Furthermore, while other studies designed to analyze the Winston–Lutz test demonstrated satisfactory analysis speed and detection performance, they encountered challenges in analyzing a Varian cube-ball phantom due to the low CNR. To address this issue, we employed the deep learning method using CNN. The deep learning approach demonstrated more robust estimation and fewer discrepancies from manual assessments compared to Gaussian fitting, regardless of EPID panel or imaging parameters. Importantly, this method can be applied to old treatment machines without adding additional energy for imaging.

However, this system relies on images stored on the server, making it unsuitable for real-time processing of non-stored images. No optimization for deep learning techniques was employed in this study; therefore, further improvements could be achieved by optimizing factors such as the training set, neural network structure, and training parameters.

Conclusion

This study presents the development of a fully automated QA system that has been in clinical and research use in our hospital since July 2018. This system has enhanced operational efficiency by seamlessly processing data for the machine learning study. Notably, the incorporation of deep learning has improved accuracy compared to the analytic method when estimating laser localization, especially for noisy MV images.

Acknowledgements

The authors are grateful to the medical physicists who participated in manual assessments.

Declarations

Conflict of interest

The authors report no conflicts of interest related to this work.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

1. F.-F. Yin et al., “The role of in-room kV X-ray imaging for patient setup and target localization report of AAPM task group 104,” Data Manag., no. 104, 2009.

2. Kanakavelu, N; Samuel, EJJ. Assessment and evaluation of MV image guidance system performance in radiotherapy. Reports Pract. Oncol. Radiother. J. Gt. Cancer Cent. Pozn. Polish Soc. Radiat. Oncol.; 2015; 20, 3 pp. 188-197. [DOI: https://dx.doi.org/10.1016/j.rpor.2015.01.002]

3. Klein, EE et al. Task group 142 report: quality assurance of medical acceleratorsa. Med. Phys.; 2009; 36, 9 pp. 4197-4212. [DOI: https://dx.doi.org/10.1118/1.3190392]

4. Baker, SJK; Budgell, GJ; MacKay, RI. Use of an amorphous silicon electronic portal imaging device for multileaf collimator quality control and calibration. Phys. Med. Biol.; 2005; 50, 7 pp. 1377-1392. [DOI: https://dx.doi.org/10.1088/0031-9155/50/7/003]

5. Bonanno, E et al. Use of an automated software module for monthly routine machine QA tests. J. Instrum.; 2023; [DOI: https://dx.doi.org/10.1088/1748-0221/18/07/T07010]

6. Hadley, SW et al. SafetyNet: streamlining and automating QA in radiotherapy. J. Appl. Clin. Med. Phys.; 2016; 17, 1 pp. 387-395. [DOI: https://dx.doi.org/10.1120/jacmp.v17i1.5920]

7. Szweda, H; Graczyk, K; Radomiak, D; Matuszewski, K; Pawałowski, B. Comparison of three different phantoms used for Winston-Lutz test with Artiscan software. Reports Pract. Oncol. Radiother.; 2020; 25, 3 pp. 351-354. [DOI: https://dx.doi.org/10.1016/J.RPOR.2020.03.003]

8. Borsavage, JM; Cherpak, A; Robar, JL. Investigation of planar image quality for a novel 2.5 MV diamond target beam from a radiotherapy linear accelerator. Phys. Imaging Radiat. Oncol.; 2020; 16, pp. 103-108. [DOI: https://dx.doi.org/10.1016/J.PHRO.2020.10.007]

9. Song, KH et al. Characterization and evaluation of 2.5 MV electronic portal imaging for accurate localization of intra- and extracranial stereotactic radiosurgery. J. Appl. Clin. Med. Phys.; 2016; 17, 4 pp. 268-284. [DOI: https://dx.doi.org/10.1120/jacmp.v17i4.6247]

10. Gräfe, JL; Owen, J; Villarreal-Barajas, JE; Khan, RFH. Characterization of a 2.5 MV inline portal imaging beam. J. Appl. Clin. Med. Phys.; 2016; 17, 5 222. [DOI: https://dx.doi.org/10.1120/JACMP.V17I5.6323]

11. Guan, H; Hammoud, R; Yin, FF. A positioning QA procedure for 2D/2D (kV/MV) and 3D/3D (CT/CBCT) image matching for radiotherapy patient setup. J. Appl. Clin. Med. Phys.; 2009; 10, 4 273. [DOI: https://dx.doi.org/10.1120/JACMP.V10I4.2954]

12. Mhatre, V; Pilakkal, S; Chadha, P; Talpatra, K. Dosimetric comparison of a-Si 1200 and a-Si 1000 electronic portal imager for intensity modulated radiation therapy (IMRT). J. Nucl. Med. Radiat. Ther.; 2018; 09, 01 pp. 1-6. [DOI: https://dx.doi.org/10.4172/2155-9619.1000354]

13. Schreibmann, E; Elder, E; Fox, T. Automated quality assurance for image-guided radiation therapy. J. Appl. Clin. Med. Phys.; 2009; 10, 1 pp. 71-79. [DOI: https://dx.doi.org/10.1120/JACMP.V10I1.2919]