Research Highlight: Use of Generative Images Created with Artificial Intelligence for Brain Tumor Imaging

Take-home points A generative adversarial network (GAN) is a deep learning technique that enables the generation of new images from unlabeled original images. Brain tumor imaging using MRI is an area of particular interest for clinical application of image generation using GAN. A recent study demonstrated that the time to progression of glioblastoma can be accurately predicted using synthetic post-contrast MR images. GANs can be clinically validated when used as an adjunct to imaging-based deep learning segmentation or classification tasks. Further investigations are warranted to determine whether GAN-based synthetic post-contrast MR images may substitute real images.

A GAN is a deep learning technique that enables the generation of new images from unlabeled original images [1]. GANs can learn the data distribution from training samples and generate realistic imaging data that have a similar distribution to the original data but are otherwise different [2, 3, 4, 5]. Image generation using a GAN is an attractive solution to overcome the limitations of small datasets [6, 7], and the generated images eventually become data inputs and expand the use of deep learning algorithms. Brain tumor imaging using MRI is of particular interest for the clinical application of image generation using GAN, because rare tumor types and the use of multiparametric imaging sequences tend to result in insufficient or incomplete imaging datasets. Although studies have demonstrated the technical feasibility of GANs for creating synthetic images for various purposes, such as filling in missing images [8], cross-modality transfer [9], improving image quality by denoising or creating super-resolution for CT [10], MRI [11, 12], and PET [13], or segmentation tasks in brain tumors [14], few studies have targeted the clinical implications and evaluated the real-world clinical utility of generative imaging.

A potential clinical use case of image generation using GAN is enabling reduced or no use of gadolinium-based contrast agents (GBCAs) by generating virtual contrast-enhanced T1-weighted images (vc-T1WI) from non-enhanced sequences. Although injection of GBCAs is generally considered a safe procedure, 1.5% of patients have mild adverse reactions [15], and there is general agreement that the safety concerns associated with GBCA should be minimized. The feasibility of creating synthetic vc-T1WI for brain MRI was demonstrated using multi-parametric non-contrast T1WI, T2-weighted images (T2WI), fluid-attenuated inversion recovery (FLAIR) images, diffusion-weighted images, susceptibility-weighted images; diagnostic quality tests and quantitative evaluations were performed [16]. The metrics used to evaluate the quality of GAN-generated images are generally either qualitative (Turing test) or structural similarity indices and the peak signal-to-noise ratio. In addition to the limited number of clinical implementations, there is a lack of evaluation metrics for determining the clinical performance of GANs applied to patient data; thus, they are undertested and not widely applied.

Recently, Jayachandran Preetha et al. [17] investigated the synthesis of post-contrast MRI sequences using pre-contrast MRI sequences, filling in the absence of imaging data for imaging evaluation of glioblastoma (Fig. 1). Their work was unique in that it encompassed both an image-to-image-based task using a GAN and an image-based task using UNet to assess tumor responses in neuro-oncology. Clinical utility was demonstrated by incorporating MRI data from three phase 2 and 3 clinical trials with over 2000 patients. The authors evaluated the clinical performance using an image-based artificial intelligence method for tumor volumetry. Their results showed that prediction of volumetrically-defined time-to-progression was possible with synthetic post-contrast MRI images, and automatic volumetry revealed—on average—no significant difference (0.1 months) between synthetic and true post-contrast MRI sequences. Therefore, very similar performances between synthetic and true post-contrast MRI data in predicting the overall survival were demonstrated.

Fig. 1 Examples of virtual contrast-enhanced T1-weighted images using generative model for glioblastoma, IDH-wild type (left), as compared with real images of glioblastoma, IDH-wild type (right). IDH = isocitrate dehydrogenase

From this hypothesis-generating study [17], we can obtain ideas about how to apply and validate GANs in clinical cases. First, synthetic images can be used as direct substitutes for real images and can make the use contrast media for MRI or CT imaging or an additional radiation exposure for X-ray, CT, or PET imaging optional, thereby reducing the harm or cost associated with the extra imaging procedures. The utility of a GAN can be demonstrated by generating synthetic data to fill in the absent or insufficient data in a multicenter trial [18]. Second, because evaluation metrics applied to images are often based on technical similarity or image quality itself, the clinical performance of synthetic images can be measured to fulfill further image-based tasks of detection, segmentation, and classification, which are often used as evaluation metrics to determine the clinical utility of imaging (Fig. 2). Currently, the evaluation metrics applied to GANs largely focus on the image quality and diversity of the generated images [19], often without a clear reference standard [20]. The diversity of the generated images was clinically shown in a study on the radiologic features of molecular subtypes of gliomas on MRI [21]. Likewise, when used as an adjunct to imaging-based tasks of segmentation or classification, the clinical performance of generative images and models can be measured, and it can be determined whether they would have a clinical impact by enhancing datasets, reducing harm, or increasing benefit.

Fig. 2 Diagram demonstrating how generative imaging can be used and validated in a clinical workflow. Generative images can be applied during the data input stage and may improve prediction performance during every process of artificial intelligence in neuro-oncologic imaging, including detection, segmentation, and subsequent classification. FLAIR = fluid-attenuated inversion recovery

However, whether virtual GBCA enhancement can replace contrast-enhanced T1WI in neuro-oncology remains to be addressed. The imaging requisites for synthetic vc-T1WI were multi-parametric imaging, including T1WI, T2WI, FLAIR [17], and additional diffusion-weighted and susceptibility-weighted sequences [16]. A feasibility study showed that diffusion-weighted imaging and T2WI contributed to demonstrating peritumoral edema, cellularity, and necrosis [16], while Jayachandran Preetha et al. [17] found that T2WI and FLAIR contributions were larger than diffusion-weighted imaging because the contrast enhancement of brain tumors is based on disruption of the blood-brain barrier, not on cellularity. The different prerequisites for imaging sequences show that the technique is yet to be optimized and further studies are warranted. As a clinical task, glioblastoma is a large tumor for which tumor segmentation produces a reliable measurement, but the identification of comparatively subtle and small contrast enhancements, for example, in brain metastasis and lower-grade gliomas (or active lesions in multiple sclerosis), is far more challenging because the available information in pre-contrast MRI sequences for this task might be insufficient and because synthetic images are limited (distorted) with respect to small vessel structures and image smoothness, rather than the generation of large enhancing (tumor) regions. One potential strategy to—at least partially—address these limitations could be the use of low-dose GBCA administration schemes (e.g., with 10% of the full-dose) and use GAN-based approaches to synthesize (virtual) full-dose contrast-enhanced T1WI [22]. Finally, the benefit of GBCA excels the potential adverse effects in glioblastoma, especially in the differentiation of pseudoprogression from true progression when brain MRI is crucial for an early diagnosis and accurate diagnosis needs to be pursued rather than reducing GBCA because the diagnosis substantially impacts patient treatment.

In summary, by generating post-contrast enhancement images as an adjunct to deep learning segmentation, GANs can be useful for the quantitative measurement of surrogate endpoints of tumor progression. As there is no clear reference standard for measuring the clinical performance of GAN, adjunctive imaging-based tasks of deep learning segmentation or classification will help measure the clinical performance of GAN. Image generation using GAN may potentially add substantial value by reducing the risks associated with imaging, including the use of contrast agents or radiation exposure.

Notes

Conflicts of Interest: Ji Eun Park, Namkug Kim and Ho Sung Kim who is on the editorial board of the Korean Journal of Radiology was not involved in the editorial evaluation or decision to publish this article. All remaining authors have declared no conflicts of interest.

Funding Statement: This research was supported by a National Research Foundation of Korea (NRF) grant, funded by the Korean government (MSIP) (grant number: NRF-2020R1A2B5B01001707) and supported by Ministry of Health and Welfare, South Korea (HI21C1161).

References

1. Engstrom L, Tran B, Tsipras D, Schmidt L, Madry A. A rotation and atranslation suffice: fooling cnns with simple transformations. OpenReview [Preprint]. 2018 [cited 2022 February 10]. Available at: https://openreview.net/forum?id=BJfvknCqFQ.

2. Dar SU, Yurt M, Karacan L, Erdem A, Erdem E, Cukur T. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans Med Imaging 2019;38:2375–2388.

3. Yurt M, Dar SU, Erdem A, Erdem E, Oguz KK, Çukur T. mustGAN: multi-stream generative adversarial networks for MR image synthesis. Med Image Anal 2021;70:101944

4. Dar SU, Yurt M, Shahdloo M, Ildız ME, Tınaz B, Çukur T. Prior-guided image reconstruction for accelerated multi-contrast MRI via generative adversarial networks. IEEE J Sel Top Signal Process 2020;14:1072–1087.

5. Benzakoun J, Deslys MA, Legrand L, Hmeydia G, Turc G, Hassen WB, et al. Synthetic FLAIR as a substitute for FLAIR sequence in acute ischemic stroke. Radiology 2022;303:153–159.

6. Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J Big Data 2019;6:60

7. Moreno-Barea FJ, Jerez JM, Franco L. Improving classification accuracy using data augmentation on small data sets. Expert Syst Appl 2020;161:113696

8. Hiasa Y, Otake Y, Takao M, Matsuoka T, Takashima K, Carass A, et al. Cross-modality image synthesis from unpaired data using CycleGAN; Proceedings of SASHIMI: International Workshop on Simulation and Synthesis in Medical Imaging (Third International Workshop, SASHIMI 2018); 2018 Sep 16; Granada: Spain. Springer; 2018. pp. 31-41.

9. Wolterink JM, Dinkla AM, Savenije MHF, Seevinck PR, van den Berg CAT, Išgum I. Deep MR to CT synthesis using unpaired data; Proceedings of SASHIMI: International Workshop on Simulation and Synthesis in Medical Imaging (Second International Workshop, SASHIMI 2017); 2017 Sep 10; Québec: Canada. Springer; 2017. pp. 14-23.

10. Wolterink JM, Leiner T, Viergever MA, Isgum I. Generative adversarial networks for noise reduction in low-dose CT. IEEE Trans Med Imaging 2017;36:2536–2545.

11. Dar SUH, Yurt M, Shahdloo M, Ildız ME, Çukur T. Synergistic reconstruction and synthesis via generative adversarial networks for accelerated multi-contrast MRI. arXiv [Preprint]. 2018 [cited 2022 February 10]. Available at: https://doi.org/10.48550/arXiv.1805.10704.

12. Kim KH, Do WJ, Park SH. Improving resolution of MR images with an adversarial network incorporating images with different contrast. Med Phys 2018;45:3120–3131.

13. Wang Y, Yu B, Wang L, Zu C, Lalush DS, Lin W, et al. 3D conditional generative adversarial networks for high-quality PET image estimation at low dose. Neuroimage 2018;174:550–562.

14. Rezaei M, Harmuth K, Gierke W, Kellermeier T, Fischer M, Yang H, et al. A conditional adversarial network for semantic segmentation of brain tumor; Proceedings of the International MICCAI Brainlesion Workshop; 2017 Sep 14; Quebec, Canada. Springer; 2017. pp. 241-252.

15. Runge VM. Safety of the gadolinium-based contrast agents for magnetic resonance imaging, focusing in part on their accumulation in the brain and especially the dentate nucleus. Invest Radiol 2016;51:273–279.

16. Kleesiek J, Morshuis JN, Isensee F, Deike-Hofmann K, Paech D, Kickingereder P, et al. Can virtual contrast enhancement in brain MRI replace gadolinium?: a feasibility study. Invest Radiol 2019;54:653–660.

17. Jayachandran Preetha C, Meredig H, Brugnara G, Mahmutoglu MA, Foltyn M, Isensee F, et al. Deep-learning-based synthesis of post-contrast T1-weighted MRI for tumour response assessment in neuro-oncology: a multicentre, retrospective cohort study. Lancet Digit Health 2021;3:e784–e794.

18. Conte GM, Weston AD, Vogelsang DC, Philbrick KA, Cai JC, Barbera M, et al. Generative adversarial networks to synthesize missing T1 and FLAIR MRI sequences for use in a multisequence brain tumor segmentation model. Radiology 2021;299:313–323.

19. Borji A. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding 2019;179:41–65.

20. Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: a review. Med Image Anal 2019;58:101552

21. Park JE, Eun D, Kim HS, Lee DH, Jang RW, Kim N. Generative adversarial network for glioblastoma ensures morphologic variations and improves diagnostic model for isocitrate dehydrogenase mutant type. Sci Rep 2021;11:9912

22. Gong E, Pauly JM, Wintermark M, Zaharchuk G. Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI. J Magn Reson Imaging 2018;48:330–340.