Progress in Serial Imaging for Prognostic Stratification of Lung Cancer Patients Receiving Immunotherapy: A Systematic Review and Meta-Analysis

Abstract

Simple Summary

Immunotherapy with checkpoint inhibitors is a promising treatment for lung cancer patients. However, not all patients respond well to immunotherapy, and researchers are seeking new predictive biomarkers for immunotherapy. Radiomics and its derivative, delta radiomics, are potential candidates for use as predictive biomarkers for use in immunotherapy. In this meta-analysis, we performed qualitative and quantitative assessments and confirmed the effectiveness of delta radiomics in predicting the treatment responses and clinical outcomes of immunotherapy. Further studies are warranted to compare the performance of traditional radiomics and deep-learning radiomics.

Abstract

Immunotherapy, particularly with checkpoint inhibitors, has revolutionized non-small cell lung cancer treatment. Enhancing the selection of potential responders is crucial, and researchers are exploring predictive biomarkers. Delta radiomics, a derivative of radiomics, holds promise in this regard. For this study, a meta-analysis was conducted that adhered to PRISMA guidelines, searching PubMed, Embase, Web of Science, and the Cochrane Library for studies on the use of delta radiomics in stratifying lung cancer patients receiving immunotherapy. Out of 223 initially collected studies, 10 were included for qualitative synthesis. Stratifying patients using radiomic models, the pooled analysis reveals a predictive power with an area under the curve of 0.81 (95% CI 0.76–0.86, p < 0.001) for 6-month response, a pooled hazard ratio of 4.77 (95% CI 2.70–8.43, p < 0.001) for progression-free survival, and 2.15 (95% CI 1.73–2.66, p < 0.001) for overall survival at 6 months. Radiomics emerges as a potential prognostic predictor for lung cancer, but further research is needed to compare traditional radiomics and deep-learning radiomics.

1. Introduction

1.1. Overview of Lung Cancer and Its Global Burden

Lung cancer, a complex and heterogeneous disease, has long been recognized as a major global health issue owing to its high incidence and mortality rate [1]. Lung cancer is the leading cause of cancer-related deaths and is responsible for over 1.8 million fatalities annually, accounting for 18% of all cancer-related deaths worldwide [2]. The disease is primarily classified into two histological subtypes: non-small cell lung cancer (NSCLC), which represents approximately 85% of all cases, and small cell lung cancer (SCLC), which constitutes the remaining 15% [3]. The primary risk factors for lung cancer are tobacco smoking, exposure to environmental carcinogens such as radon gas and asbestos, air pollution, and genetic predispositions [4,5]. Although recent advancements in early detection and therapeutic interventions have led to improved clinical outcomes for some patients, the overall five-year survival rate for lung cancer remains low, at approximately 18% [6,7]. This highlights the urgent need for innovative diagnostic and prognostic tools to optimize treatment outcomes and tailor personalized therapy strategies for patients receiving advanced treatments, such as immunotherapy [8,9].

1.2. Role of Immunotherapy in Lung Cancer Treatment

The advent of immunotherapy has revolutionized the landscape of lung cancer treatment, offering new therapeutic options for patients and significantly affecting clinical outcomes [10]. Immunotherapy primarily focuses on recovering the power of the immune system to recognize and eliminate cancer cells, thereby providing a targeted and personalized approach to treatment [11]. Recently, immune checkpoint inhibitors (ICIs), which modulate the immune system by targeting immune checkpoints, such as programmed cell death protein 1 (PD-1) [12,13], programmed cell death ligand 1 (PD-L1) [14], and cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) [15], have been enrolled in the standard of care for both NSCLC and SCLC [14,16]. These novel agents have demonstrated improved response rates, prolonged progression-free survival (PFS), and enhanced overall survival (OS) compared with conventional chemotherapy in several clinical trials [17,18]. Despite these promising results, not all patients respond to immunotherapy, and some may experience immune-related adverse events (irAEs) [19]. Consequently, ongoing research seeks to identify predictive biomarkers and develop combinatorial strategies to maximize treatment efficacy while minimizing adverse effects [20].

1.3. Importance of Delta Radiomics in Predicting Treatment Outcomes

In recent years, radiomics has emerged as a promising tool for predicting treatment outcomes in cancer patients, including those with lung cancer [21]. Radiomics is a quantitative image analysis approach that extracts high-dimensional, mineable data from medical images, such as computed tomography (CT) scans, magnetic resonance imaging (MRI), and positron emission tomography (PET) scans [22]. Delta radiomics, a subtype of radiomics, specifically focuses on changes in radiomics features over time, capturing the temporal evolution of the tumor phenotype and treatment response [23]. By providing a noninvasive and comprehensive analysis of tumor heterogeneity and progression, delta radiomics has the potential to significantly impact clinical decision making in personalized medicine, particularly in the era of immunotherapy [8,24]. Furthermore, delta radiomics may aid in identifying patients who are likely to respond to specific treatments, monitoring treatment efficacy, and predicting the risk of disease recurrence and metastasis [25]. Ongoing research aims to optimize the clinical application of delta radiomics by validating its predictive accuracy and addressing the methodological challenges related to reproducibility and standardization proposed by the image biomarker standardization initiative (IBSI) [26].

1.4. Objectives and Hypothesis of the Meta-Analysis

In recent years, there has been a significant surge in research aimed at elucidating the clinical implications of delta radiomic features obtained from computed tomography (CT) images in patients with NSCLC. These studies highlight the association between delta radiomic features and treatment response or clinical outcomes following immunotherapy. The primary objective of this investigation was to conduct an exhaustive systematic review of the existing body of delta radiomic research, particularly focusing on its potential to predict treatment response or outcomes in patients with NSCLC undergoing immunotherapy. This review entailed an assessment of the methodological quality of delta radiomic studies, utilizing the Quality in Prognosis Studies (QUIPS [27]) tool for image-mining research and the radiomics quality scoring (RQS [28]) tool as reference standards. Moreover, quantitative analysis was performed to determine the effectiveness of delta radiomics in predicting treatment response and clinical outcomes of immunotherapy in this cohort of patients.

2. Materials and Methods

2.1. Search Strategy and Selection Criteria

2.1.1. Databases and Search Terms

This systematic review and meta-analysis were conducted according to the PRISMA guidelines [29]. The study has not been registered. To ensure meticulous adherence to these guidelines, checklists corresponding to PRISMA were utilized and are presented in Supplementary Table S1. A comprehensive literature search was performed using the following electronic databases: PubMed, Embase, Web of Science, and the Cochrane Library. Articles from 2017 January 1st to 2023 April 4th were searched. The search strategy comprised a combination of MeSH terms and keywords pertinent to lung cancer, immunotherapy, and delta radiomics. Example search terms include “lung cancer”, “non-small cell lung cancer”, “NSCLC”, “immunotherapy”, “PD-1”, “PD-L1”, “nivolumab”, “pembrolizumab”, “immunomodulating agent”, “radiomics”, “radiomic features”, “radiomic signature”, “texture analysis”, “delta radiomics”, “follow-up CT”, “serial imaging”, “early response”, “treatment monitoring”, “longitudinal monitoring”, “machine learning”, “deep learning”, “PFS”, “progression-free survival”, “survival”, “overall survival”, “OS”, “treatment outcome”, “response”, and “prediction”. A tailored search strategy was devised for each database to ensure an exhaustive and systematic review of the available literature.

2.1.2. Inclusion and Exclusion Criteria

Studies were considered for inclusion if they fulfill the following criteria: (1) original research articles encompassing retrospective and prospective studies; (2) focus on patients with lung cancer treated with immunotherapy; (3) evaluate the prognostic significance of delta radiomics using CT scans to predict treatment outcomes; (4) report pertinent statistical measures, such as hazard ratios or odds ratios, for survival or other clinical endpoints; and (5) are published in English. The exclusion criteria comprise the following: (1) review articles, conference abstracts, case reports, editorials, and supplementary materials; (2) studies with insufficient data or ambiguous methodology; and (3) studies not centered on the application of delta radiomics in patients with lung cancer undergoing immunotherapy. Two independent reviewers (T.-W.W. and M.-S.H.) screened the titles. After reading the abstracts, a full-text evaluation was carried out to determine the eligibility for inclusion. Discrepancies between the reviewers were sent to a third reviewer (H.-Y.C.).

2.2. Data Extraction and Quality Assessment

2.2.1. Data Extraction Process

Two independent reviewers (T.-W.W. and M.-S.H.) extracted data from the included studies using a standardized data extraction form (Supplementary Table S2), ensuring an accurate and comprehensive collection of information. The extracted information includes the following: (a) study characteristics such as authors, year of publication, study duration, the country where the study was conducted, study design, and sample size; (b) patient demographics, including age, sex, and stage of lung cancer; (c) delta radiomic features, encompassing preprocessing, imaging modality, software, segmentations, feature extraction methodologies, radiomic signature, and formula; (d) clinical and molecular data, including smoking history, type of immunotherapy, biomarkers, and other relevant clinical factors; and (e) outcome measures, namely area under the curve (AUC) of 6-month response, the hazard ratio of PFS, and OS. Discrepancies between reviewers were resolved through discussion or, if required, consultation with a third reviewer to ensure a consistent and rigorous analysis.

2.2.2. Quality Assessment

The methodological quality of the studies included in this review was meticulously evaluated using two assessment tools: the QUIPS [27] tool and the RQS [28] tool. The QUIPS tool, specifically designed for prognostic studies, examines the risk of bias and applicability concerns across six domains: participant selection, study attrition, measurement of prognostic factors, assessment of outcomes, control of confounding factors, and statistical analysis, and reporting procedures [27]. Bias risk was assessed for all six domains, while applicability concerns were addressed for the initial three domains. In contrast, the RQS tool, developed to assess the validity and potential bias of radiomics studies, consists of 16 components [8]. Each study was assigned a score for every RQS component, and these scores were then aggregated to generate a total score. To ensure consistency and accuracy in the quality assessment process, two independent reviewers performed evaluations. If any discrepancies arose between the reviewers, they engaged in discussion to reach a consensus or, if necessary, seek input from a third reviewer.

2.3. Definitions of 6-Month Progression-Free Survival and Overall Survival

The definitions of 6-month PFS and treatment response at the 6th month were interchangeable among studies; thus, we collectively defined them as 6-month response: whether the disease progressed within 6 months after treatment. The 6-month OS was defined as whether the patient was alive within 6 months after treatment.

2.4. Meta-Analysis

We conducted 3 distinct meta-analyses using the studies we included. (1) The first was a meta-analysis evaluating the predictive performance of delta radiomics models for 6-month response to immunotherapy using a fixed-effects model. We used AUC as the performance evaluation metric. The AUC, derived from the receiver operating characteristic (ROC) curve, represents a graphical depiction of a model’s sensitivity against its false-positive rate (1–specificity) at different threshold settings. When the AUC has a value of 1, it indicates that the classification model performed perfectly, and a value of 0.5 indicates that the model is no better than random chance. When multiple AUC values were reported in a single study, we chose the best-performing model incorporating delta radiomics features for further inclusion in the meta-analysis. The AUC values were treated as expected values for further analysis. Additional subgroup analyses were performed according to traditional radiomics or deep-learning radiomics. (2) The other 2 meta-analyses compared the PFS and OS of immunotherapy between high- and low-risk groups in the validation datasets. We measured the performance by using the pooled hazard ratio (HR) and a 95% confidence interval (CI) using a fixed-effects model. The hazard ratio was transformed to a logarithmic scale. The 95% CI was used to back-calculate the standard deviation (SD) with the corresponding T-score from a Student’s T-distribution with n−1 degrees of freedom. In instances where a single study did not provide a 95% CI or SD but instead reported a standard error of the mean (SE), the SD was calculated by multiplying the square root of the sample size by the SE.

2.5. Statistical Analysis

Heterogeneity assessment among studies was conducted using Cochran’s Q test, and the I² statistic was employed for quantification purposes. The I² statistic gauges the proportion of variability in effect estimates resulting from heterogeneity as opposed to sampling error. I² values of 25%, 50%, and 75% corresponded to low, moderate, and high heterogeneity, respectively. A random-effects model (DerSimonian–Laird method) was applied for the meta-analysis in the presence of significant heterogeneity (I² > 50%). In contrast, a fixed-effects model (the Mantel–Haenszel method) was adopted when no significant heterogeneity was observed. Combined effects were calculated, and a two-sided p value of 0.05 was considered indicative of statistical significance [30]. Publication bias assessment was performed when more than 10 studies were included, as detecting funnel plot asymmetry requires a minimum of 10 studies [31]. All analyses were carried out using Review Manager (RevMan) [a computer program], version 5.4, Cochrane Collaboration, Paris, France, 2020.

3. Results

3.1. Study Selection and Characteristics

3.1.1. Flow Diagram of Study Selection

Figure 1 depicts the flow diagram that outlines the study selection process utilized in this systematic review and meta-analysis. The initial search across PubMed (n = 48), Embase (n = 108), and Web of Science (n = 67) generated a total of 223 studies. Following the removal of duplicates, 176 studies remained for further evaluation. The titles and abstracts of the mentioned articles underwent screening, leading to the exclusion of 151 articles. Subsequently, 25 studies were evaluated for eligibility, resulting in the exclusion of 15 articles. Seven of these articles were excluded due to their study designs not being related to the research interest. Two articles were excluded as their outcomes were not pertinent to the focus of this review. Additionally, one article was a conference paper, one was a meeting abstract, and four articles were supplementary materials, all of which were not considered for inclusion in this systematic review and meta-analysis. In the end, 10 studies met the inclusion criteria and were incorporated into this systematic review and meta-analysis [32,33,34,35,36,37,38,39,40,41].

3.1.2. Characteristics of Included Studies

The detailed characteristics of the 10 eligible studies are summarized in Table 1. The 10 studies enrolled a total of 1513 patients with advanced NSCLC treated with immunotherapy [32,33,34,35,36,37,38,39,40,41]. The basic characteristics of the studies are summarized in Table 1 and Table 2. Two studies were conducted in Spain [32,37], one in Belgium [33], four in China [33,35,36,38], one in the Netherlands [37] and two in the US [40,41]. All of the 10 studies were retrospective. The size of the study cohort ranges from 88 to 224. The median patient age ranged from 61 to 65 years, and the proportion of patients who were female ranged from 9% to 67%. CT was performed before and after immunotherapy treatment. The immunotherapy agent included anti-PD-1 and anti-PD-L1 as monotherapy or combination therapy with the other regimen.

3.1.3. Radiomics and Image Analysis

The details of the radiomics, type of classifier, and included clinical features are presented in Table 2. To obtain radiomics, tumor segmentation was performed manually in nine studies [32,33,34,35,37,38,39,40,41]. One study calculated radiomics not from segmented tumors but from a whole-lung image [37]. For the studies with segmented tumors, primary tumors alone were segmented in seven studies [32,33,35,36,38,39,41], while the others defined regions of interest [34,40]. The included radiomic parameters are listed in Supplementary Table S3. Six studies disclosed the predictive radiomic parameters [33,34,35,38,40,41], while the other four studies did not describe the details of the radiomic parameters [32,36,37,39]. Although the names of the radiomic parameters and their biological interpretations differed in each study, both size-based radiomic features [33,38] and texture-based radiomic features [33,34,35,38,40,41] are significant predictive parameters. Clinical features were incorporated into five studies [32,33,34,36,38]. A variety of formulas were used to calculate the difference in radiomic features across studies, incorporating both follow-up and pretreatment information. The validation methods varied, with four studies using external testing [28,30,34,35] and others employing cross-validation or split-sample techniques. The classifier methods included random forest (RF) [32,33,36,40], LASSO-Cox [34,38,39], SVM [35], and LDA [40]. The endpoints assessed in these studies were predominantly PFS, OS, and treatment response [32,33,34,35,36,37,38,39,40,41].

3.2. Quality Assessment Results

The Quality in Prognosis Studies (QUIPS [27]) quality assessment of the 10 studies is shown in Figure 2 [32,33,34,35,36,37,38,39,40,41]. The assessment of the individuals showed that there was a low risk of bias and fair application concerns for most of the assessed criteria, except for a higher risk of study participants in two studies and confounding measurement in four studies (Figure 2a). A summary of the risk of bias for all studies is shown in Figure 2b.

Table 3 presents the details of radiomic quality assessment in six domains. The mean RQS [28] for the 10 studies was 13.5 (range: 10–26). A majority of the studies provided well-documented image acquisition protocols. Four studies fulfilled the multiple segmentation criteria with different methods. Dong X. et al. segmented both the tumor and region of interest [34]. Yi Y. et al. segmented the tumor with two readers [36]. Benito F. et al. segmented the tumor using an algorithm under the supervision of experienced radiologists [39]. Laurent D. et al. segmented the tumor using an algorithm, and the results were checked by a trained chest radiologist [41]. A phantom study was not used in all studies. Three studies acquired imaging at multiple time points [32,33,34]. All studies used feature reduction methods and validation datasets. Eight of the ten studies integrated non-radiomic features into the prediction models. Only Laurent D. et al. discussed the biological correlates of phenotype difference in terms of radiomics [41]. A comparison to the ‘gold standard’ was not performed in all studies. Most of the studies demonstrated the potential clinical utility of radiomic models. Six out of ten studies performed cut-off analysis. All studies reported discrimination statistics for further analyses. Only three studies reported calibration statistics for further resampling. Only Laurent et al. prospectively collected the retrospective data with registration in a trial database [41]. Stefano T. et al. performed a cost-effectiveness analysis and featured open science and data [37].

3.3. Delta Radiomic Features and Prognostic Performance

The patients could be stratified into low- and high-risk groups through the use of radiomic models. The first meta-analysis evaluates the predictive power of the prognosis model at 6-month response with AUC, showing that the pooled AUC was 0.81 (95% CI 0.76–0.86, p < 0.001) for 6-month response (six studies, Figure 3a). The I² statistic implied low heterogeneity among the studies (I² = 15.1%, p = 0.58). The traditional radiomics subgroup showed a higher pooled AUC of 0.84 (95% CI 0.77–0.92, p < 0.001) with lower heterogeneity among the studies (I² = 0%); however, the deep-learning radiomics subgroup showed a lower pooled AUC of 0.79 (95% CI 0.72–0.86, p < 0.001) with higher heterogeneity among studies (I² = 33%, p = 0.22).

The second and third meta-analyses of comparisons of the immunotherapy outcomes between the two groups showed that the pooled HR was 4.77 (95% CI 2.70–8.43, p < 0.001) for PFS (four studies; Figure 3b) and 2.15 (95% CI 1.73–2.66, p < 0.001) for OS (five studies; Figure 3c). The I² statistic implied moderate and low heterogeneity among the studies (I² = 58%, p = 0.07; I² = 48%, p = 0.10). Funnel plots of PFS AUC, PFS HR, and OS HR are presented in Figure 4. The Egger’s tests for all endpoints showed a low risk of publication bias (p = 0.1148 for PFS AUC, p = 0.1198 for PFS HR, p = 0.0885 for OS HR).

4. Discussion

4.1. Summary of Key Findings

We conducted this meta-analysis to assess the prognostic ability of delta radiomics in immunotherapy in NSCLCs. In this meta-analysis encompassing 10 studies, incorporated into a qualitative synthesis, we integrated 6 studies to assess AI performance in predicting the 6-month response to immunotherapy through serial imaging, 4 studies to forecast the hazard ratio for PFS, and 5 studies for the hazard ratio for OS. All studies were retrospective in nature. Some studies showed a high risk of bias in study participation and study confounding. The RQS was 13.5 (range: 10–26) out of 36, indicating a moderate level of quality. The predictive parameters varied in terms of both radiomic formulas and the inclusion of clinical features. Despite variations in materials and AI models among the studies, all AI models effectively categorized patients undergoing immunotherapy into low- and high-risk groups across all three aspects: 6-month response, PFS hazard ratio, and OS hazard ratio.

4.2. Comparison with Previous Studies and Literature

To the best of our knowledge, this is the first meta-analysis evaluating the application of delta radiomics in prognosis stratification in lung cancer treated with ICIs. Two previous meta-analyses [43,44] have demonstrated the role of radiomics in predicting lung cancer prognosis. Our meta-analysis further extends the landscape to delta radiomics and multi-omic models. Compared to our previous meta-analysis about the application of radiomics in prognosing lung cancer treated with epidermal growth factor receptor tyrosine kinase inhibitors (EGFR-TKI) [45], the quality of the studies about immunotherapy was better, except in the study-confounding domain. The mean RQS was 13.5 in studies about immunotherapy and 11.67 in studies about EGFR-TKI, both indicating a moderate level of quality. In another systemic review about tracking tumor biology with radiomics [28], most of the studies (30 out of 41) included performed at less than 12 out of 36 (<30%) in terms of RQS. The mean RQS was 10.66 (29.6%) in studies about predicting immunotherapy response in NSCLC [43]. The RQS in all meta-analyses mentioned previously indicated a moderate level of quality. Most studies missed the quality in “Phantom Study on All Scanner”, “Detect and Discuss Biological Correlates”, “Comparison to ‘Gold Standard’”, “Prospective Study Registered in a Trial Database”, “Cost-Effectiveness Analysis”, and “Open Science and Data”. With the goal of discovering new biomarkers and developing a prediction model, some RQS domains may not be applicable. However, it would be important to discuss the biological correlates to make biomarkers interpretable. Open science and data is another important domain, and by combining discrimination statistics and calibration statistics, researchers could pile all the data together to train better models and perform more analyses.

4.3. Comparison of Radiomic Features in the Enrolled Studies

While the 10 enrolled studies utilized radiomics/delta radiomics as predictive parameters, the radiomic features employed in the predictive models differ across these studies. To better understand the biological interpretation of radiomic features, we categorized radiomics into several groups based on IBSI [26]. Among the six studies that disclosed their predictive radiomic features, a majority of these features are texture-related, such as entropy [33,35,41], contrast [41], and grey-level uniformity [34]. The findings regarding the importance of texture-based radiomics align with the heterogeneity observed in tumors, a factor proven to be associated with prognosis in solid tumors [46,47]. Only in two studies did size-based radiomic features, such as tumor volume [33] and axis length [38], enter the final prediction model. These findings suggest that changes in texture may provide more information than the traditional volume change proposed by the Response Evaluation Criteria in Solid Tumors (RECIST) working group or its modified version for immunotherapy, iRECIST [48]. Furthermore, radiomics that captured both size and heterogeneity were also deemed significant. For instance, metrics such as large/small area and low/high grey-level emphasis [34,35,38], which address both size and heterogeneity, were employed in three studies included in our analysis.

4.4. Strength and Limitations

The most significant strength of this study is its status as the first meta-analysis discussing the predictive power of delta radiomics. Both QUIPS and RQS assessments were conducted in this study. However, several limitations were encountered. Firstly, out of the 223 studies/articles searched, only 10 studies could be included in the final analysis, indicating that delta radiomics is a relatively novel predictive parameter. Secondly, limited data hindered our ability to perform pooled sensitivity, pooled specificity, and heterogeneity assessments as well as threshold effect assessment (HSROC). Therefore, we present the funnel plots in Figure 4, which demonstrate a low risk of publication bias. Thirdly, the varying definitions of radiomics and the inclusion of different radiomic features in various publications made it challenging for us to conduct further analysis. Finally, the immunotherapy agents varied across different studies, and detailed information could not be obtained from some studies. This limitation prevents us from evaluating the individual effects of immunotherapy.

4.5. Future Directions and Research Opportunities

To extend the application of radiomics and delta radiomics for predicting prognoses of NSCLC patients receiving immunotherapy with checkpoint inhibitors, future studies should focus on the subsections below.

4.5.1. External Validation

As shown in our PRISMA evaluation, nearly half of the studies are at high risk of bias in study confounding. Using external validation and standardized protocols could ensure the reliability and reproducibility of predicting models [49]. The validation of the model with either data from other hospitals or temporal data is applicable [50].

4.5.2. Biomarker Interpretation

In our systemic review, a limited number of studies have interpreted the selected radiomic parameters in prediction models, echoing similar findings in other meta-analyses [43,44,45] focusing on radiomics predicting lung cancer prognosis. Despite radiomics being derived from images, understanding the significance of calculations is crucial. As the field progresses into deep-learning parameters, the challenge shifts to providing interpretative tools like heat maps [51,52] for a comprehensive understanding.

4.5.3. Trial Registration and Quality Insurance

As the convenience of employing supercomputers for model development with ample data increases, guarding against overfitting becomes pivotal. A registered study design prior to analysis serves as a preventive measure against overfitting. Additionally, the utilization of tools like the QUIPS tool [27], the QUADUS tool [53], and the RQS tool [28] is indispensable, serving as reminders to authors about potential pitfalls and safeguarding against the risk of bias.

4.5.4. Data Sharing

A notable limitation in our systematic review and meta-analysis is that, although 10 studies entered the analysis phase, only 3 to 5 studies could be included in each meta-analysis due to a lack of discrimination/calibration statistics or missing study endpoints. This limitation could potentially be addressed with a standardized data-sharing format. As more parameters and drugs are continually developed, leveraging previous knowledge becomes crucial for scientific progress. Encouragingly, both geographic and temporal dimensions of data sharing are vital. We advocate for future authors to engage in open databases such as The Cancer Imaging Archive [54], ensuring compliance with institute review board standards and adherence to Findable–Accessible–Interoperable–Reusable (FAIR) data principles. For model sharing, federated learning emerges as a potential breakthrough in the future. Compliance to the institute review board and Findable–Accessible–Interoperable–Reusable (FAIR) data principles [55] could be a solution. For model sharing, federated learning could be a future breakpoint.

5. Conclusions

Our systemic review and meta-analysis demonstrated the usefulness of delta radiomics as a biomarker to predict the prognosis of NSCLC patients receiving immunotherapy with checkpoint inhibitors. With models containing delta radiomics, the performance was 0.81 in terms of the AUC when predicting 6-month response, and the stratified hazard ratio of the high-risk/low-risk groups was 4.77 for PFS and 2.15 for OS. To further optimize the ability of the models in clinical settings, future authors could register the trial before starting, focus on external validation and biomarker interpretation in study design, and share the data in a standard format for further analysis.

Author Contributions

Conceptualization, H.-Y.C. and T.-W.W.; methodology, T.-W.W. and M.-S.H.; software, T.-W.W.; validation, T.-W.W., M.-S.H. and H.-Y.C.; formal analysis, H.-Y.C. and T.-W.W.; investigation, H.-Y.C. and T.-W.W.; resources, C.-Y.L., C.-F.L., H.-S.C., H.-Y.C., Y.-T.W. and Y.-M.C.; data curation, H.-Y.C. and T.-W.W.; writing—original draft preparation, H.-Y.C. and T.-W.W.; writing—review and editing, M.-S.H., H.-S.C., C.-Y.L., C.-F.L., Y.-T.W. and Y.-M.C.; visualization, H.-Y.C. and T.-W.W.; supervision, C.-F.L., H.-S.C., Y.-T.W. and Y.-M.C.; project administration, H.-S.C., Y.-T.W. and Y.-M.C.; funding acquisition, H.-Y.C., H.-S.C., Y.-T.W. and Y.-M.C. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data presented in this study are available upon request from the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. PRISMA flowchart of the included studies.

Figure 2. Quality assessment. (a) Risk of bias for individual studies. (b) Summary of risk of biases [32,33,34,35,36,37,38,39,40,41].

Figure 3. Forest plots of the predictive performance of radiomics models in progression-free survival and overall survival of NSCLC patients treated with immunotherapy [32,33,34,35,36,37,38,39,41]. (a) The 6-month response, (b) hazard ratio for progression-free survival, (c) hazard ratio for overall survival.

Figure 4. Funnel plots of the pooled predictive performance of radiomics models in progression-free survival and overall survival of NSCLC patients treated with immunotherapy. (a) The 6-month response AUC, (b) hazard ratio for progression-free survival, (c) hazard ratio for overall survival.

Table 1

Basic characteristics of studies included in systematic review and the meta-analysis.

Author	Dataset	StudyDuration	Country	Study Design	Patients	Age	Sex(Female)	Smoker	Stage	Adeno	Immunotherapy Agent	Immunotherapy Regimen
Benito, F. (2023) [32]	D	2013–2021	Spain	Retrospective	200	65 (58–70)	58 (29)	174 (87.9)	IV	151 (75.5)	Immunotherapy	Monotherapy or combination
François, C. (2022) [33]	D	2015–2020	Belgium	Retrospective	121	65 (41– 85)	45 (37)	119 (98)	III–IV	71 (59)	Pembrolizumab, nivolumab, atezolizumab	Monotherapy
	E	2015–2018	Belgium	Retrospective	39	64 (44– 95)	19 (49)	32 (82)	III–IV	29 (74)	Pembrolizumab, nivolumab, atezolizumab	Monotherapy
Dong, X. (2022) [34]	D	2016–2021	China	Retrospective	68	NA	6	35	III–IV	23	Camrelizumab, sintilimab, tislelizumab, nivolumab, atezolizumab	Monotherapy or combination
	V	2016–2021	China	Retrospective	29	NA	3	15	III–IV	6	Camrelizumab, sintilimab, tislelizumab, nivolumab, atezolizumab	Monotherapy or combination
Jing, G. (2022) [35]	D	2015–2018	China	Retrospective	93	67 (31–85)	13 (14)	42 (45.2)	III–IV	57 (61.3)	Immunotherapy	Monotherapy
	E	2016–2020	China	Retrospective	68	61 (27–76)	16 (23.5)	46 (67.6)	III–IV	54 (79.4)	Immunotherapy	Monotherapy
	E	2018–2020	China	Retrospective	63	66 (29–86)	11 (17.5)	19 (30.2)	III–IV	38 (60.3)	Immunotherapy	Monotherapy
Yi, Y. (2021) [36]	D	2016–2019	China	Retrospective	200	NA	35	119	IIIB–IV	132 (66)	Nivolumab, pembrolizumab, atezolizumab	Monotherapy
Stefano, T. (2021) [37]	D	2014–2016	Netherlands	Retrospective	152	64.4 (57.8–68.9)	64	x	IV	92 (60.5)	Anti-PD1 immunotherapy	Monotherapy
Ying, L. (2021) [38]	D	2018–2019	China	Retrospective	112	NA	14	81	NA	61	Immunotherapy	Monotherapy or combination
	V	2018–2019	China	Retrospective	49	NA	13	35	NA	23	Immunotherapy	Monotherapy or combination
Benito, F. (2021) [39]	D	2013–2019	Spain	Retrospective	88	NA	NA	NA	NA	NA	Immunotherapy	Monotherapy or combination
Mohammadhadi, K. (2020) [40]	D	2012–2017	America	Retrospective	112	65 (42–83)	54 (48)	96 (86)	NA	80 (71)	Nivolumab, pembrolizumab, atezolizumab	Monotherapy
	E	2012–2017	America	Retrospective	27	63 (42–83)	18 (67)	21 (78)	NA	21 (78)	Nivolumab, pembrolizumab, atezolizumab	Monotherapy
Laurent, D. (2020) [41]	D	NA	America	Retrospective	92	NA	NA	NA	III–IV	0	Nivolumab	Monotherapy

D, development dataset; V, validation dataset; E, external validation dataset; NA, not applicable; Adeno, adenocarcinoma.

Table 2

Summary of details of radiomic and image analyses.

Author	Segmentation	VOI	Clinical Feature	Radiomics	Formula	Software	Validation	Classifier	EndPoints
Benito, F. (2023) [32]	Manual	primary tumor	NLR, SII, Hb, MLR, neutrophil, liver metastasis, histology, platelet, smoking, PLR, BMI, age	longitudinal radiomics	concatenate pretreat and follow	NoduleX (deep learning)	Cross validation	Random forest	PFS, OS
François, C. (2022) [33]	Manual	primary tumor	sex, clinical stage, ANC, eosinophil, and NLR	delta radiomics	follow-pretreat	Radiomics (Oncoradiomics SA, Belgium)	External testing	RF, CoxPH	Response, OS
Dong, X. (2022) [34]	Manual	tumor	tumor anatomical classification and brain metastasis	delta radiomics	follow-pretreat	Pyradiomics	Split sample	LASSO-Cox	PFS
Jing, G. (2022) [35]	Manual	primary tumor	NA	delta radiomics	follow-pretreat	Pyradiomics	External testing	SVM	Response, PFS, OS
Yi, Y. (2021) [36]	Manual	primary tumor	clinical + blood test	longitudinal radiomics	SimTa module	Pyradiomics	Cross validation	SimTA	Response, PFS, OS
Stefano, T. (2021) [37]	NA	whole lung	NA	longitudinal radiomics	deep feature	VGG-like network	Split sample	RF	PFS, OS
Ying, L. (2021) [38]	Manual	primary tumor	distant metastasis	delta radiomics	(follow-pretreat)/pretreat	Analysis Kit, version 3.2.5, GE Healthcare	Split sample	LASSO-Cox	Response
Benito, F. (2021) [39]	Manual	primary tumor	NA	delta radiomics	follow-pretreat	Pyradiomics	External testing	LASSO-Cox	OS
Mohammadhadi, K. (2020) [40]	Manual	tumor	NA	delta radiomics	follow-pretreat	In-house developed toolbox with MATLAB 2018b	External testing	LDA	Response, OS
Laurent, D. (2020) [41]	Manual	primary tumor	NA	delta radiomics	size: (follow-pretreat)/pretreat, other: follow-pretreat	In-house developed toolbox	Split sample	RF	Response

ANC, absolute neutrophil count; BMI, body mass index; LASSO-Cox, least absolute shrinkage and selection operator–Cox proportional hazards; LDA, linear discriminant analysis; MLR, monocyte to lymphocyte ratio; NA, not applicable; NLR, neutrophil to lymphocyte ratio; OS, overall survival; PFS, progression-free survival; PLR, platelet to lymphocyte ratio; RF, random forest; SII, systemic inflammation index; SVM, support vector machine; VOI, volume of interest.

Table 3

Details of radiomics quality scores.

	Domain 1				Domain 2		Domain 3				Domain 4			Domain 5		Domain 6
Author	Image Protocol Quality	Multiple Segmentation	Phantom Study on All Scanner	Imaging at Multiple Time Points	Feature Reduction or Adjustment for Multiple Testing	Validation	Multivariable Analysis with Non -Radiomic Features	Detect and Discuss Biological Correlates	Comparison to ’Gold Standard’	Potential Clinical Utility	Cut-Off Analyses	Discrimination Statistics	Calibration Statistics	Prospective Study Registered in a Trial Database	Cost-Effectiveness Analysis	Open Science and Data	Total
Benito, F. (2023) [32]	1	0	0	1	3	3	1	0	0	1	0	2	0	0	0	0	12
François, C. (2022) [33]	1	0	0	1	3	3	1	0	0	1	0	2	0	0	0	0	12
Dong, X. (2022) [34]	1	1	0	1	3	2	1	0	0	1	1	2	1	0	0	0	14
Jing, G. (2022) [35]	0	0	0	0	3	3	0	0	0	1	1	2	0	0	0	0	10
Yi, Y. (2021) [42]	1	1	0	0	3	3	1	0	0	1	1	2	0	0	0	0	13
Stefano, T. (2021) [37]	1	0	0	0	3	3	1	0	0	1	0	2	0	0	1	1	12
Ying, L. (2021) [38]	1	0	0	0	3	3	1	0	0	1	0	2	1	0	0	0	12
Benito, F. (2021) [39]	1	1	0	0	3	3	0	0	0	0	1	2	0	0	0	0	11
Mohammadhadi, K. (2020) [40]	1	0	0	0	3	4	1	0	0	1	1	2	0	0	0	0	13
Laurent, D. (2020) [41]	1	1	0	0	3	2	1	1	0	2	1	2	2	7	0	0	26

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/cancers16030615/s1, Table S1: PRISMA Checklist; Ref. [56] is cited in Supplementary Materials. Table S2: Data extraction form. Table S3: Delta radiomics parameters of studies included in systematic review and the meta-analysis.

References

1. Sung, H.; Ferlay, J.; Siegel, R.L.; Laversanne, M.; Soerjomataram, I.; Jemal, A.; Bray, F. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J. Clin.; 2021; 71, pp. 209-249. [DOI: https://dx.doi.org/10.3322/caac.21660]

2. World Health Organization. Cancer: Key Facts. Available online: https://www.who.int/news-room/fact-sheets/detail/cancer (accessed on 1 October 2023).

3. Herbst, R.S.; Heymach, J.V.; Lippman, S.M. Lung cancer. N. Engl. J. Med.; 2008; 359, pp. 1367-1380. [DOI: https://dx.doi.org/10.1056/NEJMra0802714]

4. Wakelee, H.A.; Chang, E.T.; Gomez, S.L.; Keegan, T.H.; Feskanich, D.; Clarke, C.A.; Holmberg, L.; Yong, L.C.; Kolonel, L.N.; Gould, M.K. et al. Lung cancer incidence in never smokers. J. Clin. Oncol.; 2007; 25, pp. 472-478. [DOI: https://dx.doi.org/10.1200/JCO.2006.07.2983]

5. Alberg, A.J.; Samet, J.M. Epidemiology of lung cancer. Chest; 2003; 123, pp. 21S-49S. [DOI: https://dx.doi.org/10.1378/chest.123.1_suppl.21S]

6. Siegel, R.L.; Miller, K.D.; Fuchs, H.E.; Jemal, A. Cancer Statistics. CA Cancer J. Clin.; 2021; 71, pp. 7-33. [DOI: https://dx.doi.org/10.3322/caac.21654]

7. Doroshow, D.B.; Herbst, R.S. Treatment of Advanced Non-Small Cell Lung Cancer in 2018. JAMA Oncol.; 2018; 4, pp. 569-570. [DOI: https://dx.doi.org/10.1001/jamaoncol.2017.5190]

8. Lambin, P.; Leijenaar, R.T.H.; Deist, T.M.; Peerlings, J.; de Jong, E.E.C.; van Timmeren, J.; Sanduleanu, S.; Larue, R.; Even, A.J.G.; Jochems, A. et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol.; 2017; 14, pp. 749-762. [DOI: https://dx.doi.org/10.1038/nrclinonc.2017.141]

9. Ribas, A.; Wolchok, J.D. Cancer immunotherapy using checkpoint blockade. Science; 2018; 359, pp. 1350-1355. [DOI: https://dx.doi.org/10.1126/science.aar4060]

10. Doroshow, D.B.; Sanmamed, M.F.; Hastings, K.; Politi, K.; Rimm, D.L.; Chen, L.; Melero, I.; Schalper, K.A.; Herbst, R.S. Immunotherapy in Non-Small Cell Lung Cancer: Facts and Hopes. Clin. Cancer Res.; 2019; 25, pp. 4592-4602. [DOI: https://dx.doi.org/10.1158/1078-0432.CCR-18-1538]

11. Chen, D.S.; Mellman, I. Oncology meets immunology: The cancer-immunity cycle. Immunity; 2013; 39, pp. 1-10. [DOI: https://dx.doi.org/10.1016/j.immuni.2013.07.012]

12. Reck, M.; Rodriguez-Abreu, D.; Robinson, A.G.; Hui, R.; Csoszi, T.; Fulop, A.; Gottfried, M.; Peled, N.; Tafreshi, A.; Cuffe, S. et al. Pembrolizumab versus Chemotherapy for PD-L1-Positive Non-Small-Cell Lung Cancer. N. Engl. J. Med.; 2016; 375, pp. 1823-1833. [DOI: https://dx.doi.org/10.1056/NEJMoa1606774] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27718847]

13. Horn, L.; Spigel, D.R.; Vokes, E.E.; Holgado, E.; Ready, N.; Steins, M.; Poddubskaya, E.; Borghaei, H.; Felip, E.; Paz-Ares, L. et al. Nivolumab Versus Docetaxel in Previously Treated Patients With Advanced Non-Small-Cell Lung Cancer: Two-Year Outcomes From Two Randomized, Open-Label, Phase III Trials (CheckMate 017 and CheckMate 057). J. Clin. Oncol.; 2017; 35, pp. 3924-3933. [DOI: https://dx.doi.org/10.1200/JCO.2017.74.3062] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29023213]

14. Horn, L.; Mansfield, A.S.; Szczesna, A.; Havel, L.; Krzakowski, M.; Hochmair, M.J.; Huemer, F.; Losonczy, G.; Johnson, M.L.; Nishio, M. et al. First-Line Atezolizumab plus Chemotherapy in Extensive-Stage Small-Cell Lung Cancer. N. Engl. J. Med.; 2018; 379, pp. 2220-2229. [DOI: https://dx.doi.org/10.1056/NEJMoa1809064] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30280641]

15. Paz-Ares, L.; Ciuleanu, T.E.; Cobo, M.; Schenker, M.; Zurawski, B.; Menezes, J.; Richardet, E.; Bennouna, J.; Felip, E.; Juan-Vidal, O. et al. First-line nivolumab plus ipilimumab combined with two cycles of chemotherapy in patients with non-small-cell lung cancer (CheckMate 9LA): An international, randomised, open-label, phase 3 trial. Lancet Oncol.; 2021; 22, pp. 198-211. [DOI: https://dx.doi.org/10.1016/S1470-2045(20)30641-0] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33476593]

16. Paz-Ares, L.; Dvorkin, M.; Chen, Y.; Reinmuth, N.; Hotta, K.; Trukhin, D.; Statsenko, G.; Hochmair, M.J.; Ozguroglu, M.; Ji, J.H. et al. Durvalumab plus platinum-etoposide versus platinum-etoposide in first-line treatment of extensive-stage small-cell lung cancer (CASPIAN): A randomised, controlled, open-label, phase 3 trial. Lancet; 2019; 394, pp. 1929-1939. [DOI: https://dx.doi.org/10.1016/S0140-6736(19)32222-6]

17. Borghaei, H.; Paz-Ares, L.; Horn, L.; Spigel, D.R.; Steins, M.; Ready, N.E.; Chow, L.Q.; Vokes, E.E.; Felip, E.; Holgado, E. et al. Nivolumab versus Docetaxel in Advanced Nonsquamous Non-Small-Cell Lung Cancer. N. Engl. J. Med.; 2015; 373, pp. 1627-1639. [DOI: https://dx.doi.org/10.1056/NEJMoa1507643]

18. Herbst, R.S.; Baas, P.; Kim, D.W.; Felip, E.; Perez-Gracia, J.L.; Han, J.Y.; Molina, J.; Kim, J.H.; Arvis, C.D.; Ahn, M.J. et al. Pembrolizumab versus docetaxel for previously treated, PD-L1-positive, advanced non-small-cell lung cancer (KEYNOTE-010): A randomised controlled trial. Lancet; 2016; 387, pp. 1540-1550. [DOI: https://dx.doi.org/10.1016/S0140-6736(15)01281-7]

19. Postow, M.A.; Sidlow, R.; Hellmann, M.D. Immune-Related Adverse Events Associated with Immune Checkpoint Blockade. N. Engl. J. Med.; 2018; 378, pp. 158-168. [DOI: https://dx.doi.org/10.1056/NEJMra1703481]

20. Rizvi, N.A.; Hellmann, M.D.; Snyder, A.; Kvistborg, P.; Makarov, V.; Havel, J.J.; Lee, W.; Yuan, J.; Wong, P.; Ho, T.S. et al. Cancer immunology. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer. Science; 2015; 348, pp. 124-128. [DOI: https://dx.doi.org/10.1126/science.aaa1348]

21. Aerts, H.J.; Velazquez, E.R.; Leijenaar, R.T.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun.; 2014; 5, 4006. [DOI: https://dx.doi.org/10.1038/ncomms5006]

22. Lambin, P.; Rios-Velazquez, E.; Leijenaar, R.; Carvalho, S.; van Stiphout, R.G.; Granton, P.; Zegers, C.M.; Gillies, R.; Boellard, R.; Dekker, A. et al. Radiomics: Extracting more information from medical images using advanced feature analysis. Eur. J. Cancer; 2012; 48, pp. 441-446. [DOI: https://dx.doi.org/10.1016/j.ejca.2011.11.036]

23. Fave, X.; Zhang, L.; Yang, J.; Mackin, D.; Balter, P.; Gomez, D.; Followill, D.; Jones, A.K.; Stingo, F.; Liao, Z. et al. Delta-radiomics features for the prediction of patient outcomes in non-small cell lung cancer. Sci. Rep.; 2017; 7, 588. [DOI: https://dx.doi.org/10.1038/s41598-017-00665-z]

24. Tunali, I.; Stringfield, O.; Guvenis, A.; Wang, H.; Liu, Y.; Balagurunathan, Y.; Lambin, P.; Gillies, R.J.; Schabath, M.B. Radial gradient and radial deviation radiomic features from pre-surgical CT scans are associated with survival among lung adenocarcinoma patients. Oncotarget; 2017; 8, pp. 96013-96026. [DOI: https://dx.doi.org/10.18632/oncotarget.21629] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29221183]

25. Zhang, L.; Fried, D.V.; Fave, X.J.; Hunter, L.A.; Yang, J.; Court, L.E. IBEX: An open infrastructure software platform to facilitate collaborative work in radiomics. Med. Phys.; 2015; 42, pp. 1341-1353. [DOI: https://dx.doi.org/10.1118/1.4908210] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25735289]

26. Zwanenburg, A.; Vallieres, M.; Abdalah, M.A.; Aerts, H.; Andrearczyk, V.; Apte, A.; Ashrafinia, S.; Bakas, S.; Beukinga, R.J.; Boellaard, R. et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology; 2020; 295, pp. 328-338. [DOI: https://dx.doi.org/10.1148/radiol.2020191145] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32154773]

27. Grooten, W.J.A.; Tseli, E.; Ang, B.O.; Boersma, K.; Stalnacke, B.M.; Gerdle, B.; Enthoven, P. Elaborating on the assessment of the risk of bias in prognostic studies in pain rehabilitation using QUIPS-aspects of interrater agreement. Diagn. Progn. Res.; 2019; 3, 5. [DOI: https://dx.doi.org/10.1186/s41512-019-0050-0] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31093575]

28. Sanduleanu, S.; Woodruff, H.C.; de Jong, E.E.C.; van Timmeren, J.E.; Jochems, A.; Dubois, L.; Lambin, P. Tracking tumor biology with radiomics: A systematic review utilizing a radiomics quality score. Radiother. Oncol.; 2018; 127, pp. 349-360. [DOI: https://dx.doi.org/10.1016/j.radonc.2018.03.033] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29779918]

29. Liberati, A.; Altman, D.G.; Tetzlaff, J.; Mulrow, C.; Gotzsche, P.C.; Ioannidis, J.P.; Clarke, M.; Devereaux, P.J.; Kleijnen, J.; Moher, D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. J. Clin. Epidemiol.; 2009; 62, pp. e1-e34. [DOI: https://dx.doi.org/10.1016/j.jclinepi.2009.06.006] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/19631507]

30. Higgins, J.P.; Thompson, S.G.; Deeks, J.J.; Altman, D.G. Measuring inconsistency in meta-analyses. BMJ; 2003; 327, pp. 557-560. [DOI: https://dx.doi.org/10.1136/bmj.327.7414.557] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/12958120]

31. Sterne, J.A.; Sutton, A.J.; Ioannidis, J.P.; Terrin, N.; Jones, D.R.; Lau, J.; Carpenter, J.; Rucker, G.; Harbord, R.M.; Schmid, C.H. et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ; 2011; 343, d4002. [DOI: https://dx.doi.org/10.1136/bmj.d4002]

32. Farina, B.; Guerra, A.D.R.; Bermejo-Pelaez, D.; Miras, C.P.; Peral, A.A.; Madueno, G.G.; Jaime, J.C.; Vilalta-Lacarra, A.; Perez, J.R.; Munoz-Barrutia, A. et al. Integration of longitudinal deep-radiomics and clinical data improves the prediction of durable benefits to anti-PD-1/PD-L1 immunotherapy in advanced NSCLC patients. J. Transl. Med.; 2023; 21, 174. [DOI: https://dx.doi.org/10.1186/s12967-023-04004-x]

33. Cousin, F.; Louis, T.; Dheur, S.; Aboubakar, F.; Ghaye, B.; Occhipinti, M.; Vos, W.; Bottari, F.; Paulus, A.; Sibille, A. et al. Radiomics and Delta-Radiomics Signatures to Predict Response and Survival in Patients with Non-Small-Cell Lung Cancer Treated with Immune Checkpoint Inhibitors. Cancers; 2023; 15, 1968. [DOI: https://dx.doi.org/10.3390/cancers15071968]

34. Xie, D.; Xu, F.; Zhu, W.; Pu, C.; Huang, S.; Lou, K.; Wu, Y.; Huang, D.; He, C.; Hu, H. Delta radiomics model for the prediction of progression-free survival time in advanced non-small-cell lung cancer patients after immunotherapy. Front. Oncol.; 2022; 12, 990608. [DOI: https://dx.doi.org/10.3389/fonc.2022.990608]

35. Gong, J.; Bao, X.; Wang, T.; Liu, J.; Peng, W.; Shi, J.; Wu, F.; Gu, Y. A short-term follow-up CT based radiomics approach to predict response to immunotherapy in advanced non-small-cell lung cancer. Oncoimmunology; 2022; 11, 2028962. [DOI: https://dx.doi.org/10.1080/2162402X.2022.2028962] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35096486]

36. Yang, Y.; Yang, J.; Shen, L.; Chen, J.; Xia, L.; Ni, B.; Ge, L.; Wang, Y.; Lu, S. A multi-omics-based serial deep learning approach to predict clinical outcomes of single-agent anti-PD-1/PD-L1 immunotherapy in advanced stage non-small-cell lung cancer. Am. J. Transl. Res.; 2021; 13, pp. 743-756. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33594323]

37. Trebeschi, S.; Bodalal, Z.; Boellaard, T.N.; Tareco Bucho, T.M.; Drago, S.G.; Kurilova, I.; Calin-Vainak, A.M.; Delli Pizzi, A.; Muller, M.; Hummelink, K. et al. Prognostic Value of Deep Learning-Mediated Treatment Monitoring in Lung Cancer Patients Receiving Immunotherapy. Front. Oncol.; 2021; 11, 609054. [DOI: https://dx.doi.org/10.3389/fonc.2021.609054] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33738253]

38. Liu, Y.; Wu, M.; Zhang, Y.; Luo, Y.; He, S.; Wang, Y.; Chen, F.; Liu, Y.; Yang, Q.; Li, Y. et al. Imaging Biomarkers to Predict and Evaluate the Effectiveness of Immunotherapy in Advanced Non-Small-Cell Lung Cancer. Front. Oncol.; 2021; 11, 657615. [DOI: https://dx.doi.org/10.3389/fonc.2021.657615] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33816314]

39. Farina, B.; Guerra, A.R.; Miras, C.P.; Madueño, G.G.; Muñoz-Barrutia, A.; Peces-Barba, G.; Maceiras, L.S.; Corral, J.; Gil-Bazo, I.; Gómez, M.D. Delta-Radiomics Signature For Prediction Of Survival In Advanced Nsclc Patients Treated With Immunotherapy. Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), IEEE; Nice, France, 3–16 April 2021; pp. 886-890.

40. Khorrami, M.; Prasanna, P.; Gupta, A.; Patil, P.; Velu, P.D.; Thawani, R.; Corredor, G.; Alilou, M.; Bera, K.; Fu, P. et al. Changes in CT Radiomic Features Associated with Lymphocyte Distribution Predict Overall Survival and Response to Immunotherapy in Non-Small Cell Lung Cancer. Cancer Immunol. Res.; 2020; 8, pp. 108-119. [DOI: https://dx.doi.org/10.1158/2326-6066.CIR-19-0476] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31719058]

41. Dercle, L.; Fronheiser, M.; Lu, L.; Du, S.; Hayes, W.; Leung, D.K.; Roy, A.; Wilkerson, J.; Guo, P.; Fojo, A.T. et al. Identification of Non-Small Cell Lung Cancer Sensitive to Systemic Cancer Therapies Using Radiomics. Clin. Cancer Res.; 2020; 26, pp. 2151-2162. [DOI: https://dx.doi.org/10.1158/1078-0432.CCR-19-2942] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32198149]

42. Xiao, Y.; Wang, X.; Li, Q.C.; Fan, R.R.; Chen, R.T.; Shao, Y.; Chen, Y.B.; Gao, Y.Z.; Liu, A.E.; Chen, L. et al. A cascade and heterogeneous neural network for CT pulmonary nodule detection and its evaluation on both phantom and patient data. Comput. Med. Imaging Graph.; 2021; 90, 101889. [DOI: https://dx.doi.org/10.1016/j.compmedimag.2021.101889] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33848755]

43. Chen, Q.; Zhang, L.; Mo, X.; You, J.; Chen, L.; Fang, J.; Wang, F.; Jin, Z.; Zhang, B.; Zhang, S. Current status and quality of radiomic studies for predicting immunotherapy response and outcome in patients with non-small cell lung cancer: A systematic review and meta-analysis. Eur. J. Nucl. Med. Mol. Imaging; 2021; 49, pp. 345-360. [DOI: https://dx.doi.org/10.1007/s00259-021-05509-7]

44. Zhang, C.; de, A.F.F.L.; Shi, Z.; Zhu, C.; Dekker, A.; Bermejo, I.; Wee, L. Systematic review of radiomic biomarkers for predicting immune checkpoint inhibitor treatment outcomes. Methods; 2021; 188, pp. 61-72. [DOI: https://dx.doi.org/10.1016/j.ymeth.2020.11.005] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33271285]

45. Wang, T.W.; Hsu, M.S.; Lin, Y.H.; Chiu, H.Y.; Chao, H.S.; Liao, C.Y.; Lu, C.F.; Wu, Y.T.; Huang, J.W.; Chen, Y.M. Application of Radiomics in Prognosing Lung Cancer Treated with Epidermal Growth Factor Receptor Tyrosine Kinase Inhibitors: A Systematic Review and Meta-Analysis. Cancers; 2023; 15, 3542. [DOI: https://dx.doi.org/10.3390/cancers15143542] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37509204]

46. Davnall, F.; Yip, C.S.; Ljungqvist, G.; Selmi, M.; Ng, F.; Sanghera, B.; Ganeshan, B.; Miles, K.A.; Cook, G.J.; Goh, V. Assessment of tumor heterogeneity: An emerging imaging tool for clinical practice?. Insights Imaging; 2012; 3, pp. 573-589. [DOI: https://dx.doi.org/10.1007/s13244-012-0196-6] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23093486]

47. Zhang, J.; Fujimoto, J.; Zhang, J.; Wedge, D.C.; Song, X.; Zhang, J.; Seth, S.; Chow, C.W.; Cao, Y.; Gumbs, C. et al. Intratumor heterogeneity in localized lung adenocarcinomas delineated by multiregion sequencing. Science; 2014; 346, pp. 256-259. [DOI: https://dx.doi.org/10.1126/science.1256930] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25301631]

48. Seymour, L.; Bogaerts, J.; Perrone, A.; Ford, R.; Schwartz, L.H.; Mandrekar, S.; Lin, N.U.; Litiere, S.; Dancey, J.; Chen, A. et al. iRECIST: Guidelines for response criteria for use in trials testing immunotherapeutics. Lancet Oncol.; 2017; 18, pp. e143-e152. [DOI: https://dx.doi.org/10.1016/S1470-2045(17)30074-8]

49. Gillies, R.J.; Kinahan, P.E.; Hricak, H. Radiomics: Images Are More than Pictures, They Are Data. Radiology; 2016; 278, pp. 563-577. [DOI: https://dx.doi.org/10.1148/radiol.2015151169]

50. Ramspek, C.L.; Jager, K.J.; Dekker, F.W.; Zoccali, C.; van Diepen, M. External validation of prognostic models: What, why, how, when and where?. Clin. Kidney J.; 2021; 14, pp. 49-58. [DOI: https://dx.doi.org/10.1093/ckj/sfaa188]

51. He, S.; Feng, Y.; Grant, P.E.; Ou, Y. Segmentation ability map: Interpret deep features for medical image segmentation. Med. Image Anal.; 2023; 84, 102726. [DOI: https://dx.doi.org/10.1016/j.media.2022.102726]

52. Lee, M.H.; Shomanov, A.; Kudaibergenova, M.; Viderman, D. Deep Learning Methods for Interpretation of Pulmonary CT and X-ray Images in Patients with COVID-19-Related Lung Involvement: A Systematic Review. J. Clin. Med.; 2023; 12, 3446. [DOI: https://dx.doi.org/10.3390/jcm12103446]

53. Whiting, P.F.; Rutjes, A.W.; Westwood, M.E.; Mallett, S.; Deeks, J.J.; Reitsma, J.B.; Leeflang, M.M.; Sterne, J.A.; Bossuyt, P.M. Group, Q QUADAS-2. A revised tool for the quality assessment of diagnostic accuracy studies. Ann. Intern. Med.; 2011; 155, pp. 529-536. [DOI: https://dx.doi.org/10.7326/0003-4819-155-8-201110180-00009] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22007046]

54. Rizzo, S.; Botta, F.; Raimondi, S.; Origgi, D.; Fanciullo, C.; Morganti, A.G.; Bellomi, M. Radiomics: The facts and the challenges of image analysis. Eur. Radiol. Exp.; 2018; 2, 36. [DOI: https://dx.doi.org/10.1186/s41747-018-0068-z]

55. Wilkinson, M.D.; Dumontier, M.; Aalbersberg, I.J.; Appleton, G.; Axton, M.; Baak, A.; Blomberg, N.; Boiten, J.W.; da Silva Santos, L.B.; Bourne, P.E. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data; 2016; 3, 160018. [DOI: https://dx.doi.org/10.1038/sdata.2016.18] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26978244]

56. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E. et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ; 2021; 372, n71. [DOI: https://dx.doi.org/10.1136/bmj.n71] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33782057]