1. Introduction
COVID-19 is a deadly pathogenic disease that results from the dissemination of coronavirus infection [1]. Acute respiratory distress syndrome, nervous system problems, organ dysfunction, or death may be caused by COVID-19 [2,3]. Thus, early recognition and prompt medical treatment of COVID-19 has become a major issue. To identify COVID-19 easily and efficiently, and to determine the prognosis, researchers have focused on researching and developing new detection methods [4]. To date, the chest X-ray is cheaper than other specialized methods. The chest X-ray is assessable from the image and has been refined during the last decade [5]. Recent evidence has indicated that the chest X-ray is a potent way of forecasting pulmonary diseases [6], respiratory diseases [7], cardiovascular diseases [8], or acute internal bleeding [9]. Cytokine storms and innate immune system overworking may trigger Acute Lung Injury (ALI) and the induction of acute respiratory distress syndrome (ARDS) related to the COVID-19 patients involved with hypertension [10]. Multiphase fibrosis, tissue stiffness, and lung function damage [11] may be caused by the hyaluronic acid (HA) molecules’ product in lung tissue, which is triggered by the cytokine storm. SARS-CoV-2 transparent cells relying on binding to the spike (S) glycoprotein of the Angiotensin-Converting Enzyme 2 (ACE2) receptor [12,13].
Therefore, a chest X-ray or computed tomography (CT) scan are recommended as first-line diagnostic tools for pulmonary involved patients [14]. Multilobar bilateral and unilateral chest X-ray, ground-glass opacity (GGO), and peripheral infiltrates on chest CT scans have been clinically proven to have a radiological role in the diagnosis of the COVID-19 disease [15,16]. For the peripheral regions of tissue, more than one lobe was found in the form of GGOs, or less nodules were found in each lobe [14,17]. The diagnosis of vascular nodules in the images of patients was difficult to attribute to the removal and large number of lung CT images and their complex and heterogeneous structures [18]. Thus, the artificial intelligence (AI) systems that assist medical imaging for screening have gained an important role in supporting decision making [19]. The AI presented an ability to change clinical decision-making; however, we should be cautious about implementing AI systems in each information system [20]. In terms of the merit of AI systems for physicians, AI systems assist as a diagnostic tool, making faster and efficient decisions. The augment ability on medical imaging in disease diagnosis is driving advances in traditional image processing and AI algorithms to retrieve diagnostic information. When diseases occur, AI provides a physician with the necessary diagnostic information required to speed up diagnosis and add precision intervention decisions. Some traditional image-assisted techniques for AI diagnosis contain contours and region progression, which provide a physician an aid with which to extract diagnostic information. Moreover, traditional models experience limited performance, customizability, and a strong reliance on in-advance computed features. AI has the ability to avoid the above limitations and to derive complex image features by importing feature semantics into classifiers [21,22,23]. In previous studies that evaluated the relationship between the use of a chest X-ray and COVID-19 incidence and death rates, it was found that the chest X-ray has not only diagnostic value but also great potential as a prediction image for clinical outcomes [24,25].
Recently, a study conducted a literature review on the diagnostic role of AI, which suggested its excellent potential and wide application relative to comparative methods, based on sufficiently sized datasets and independent testing [26]. The study found that radiographic diagnostics have sharper sensitivity than laboratory testing when compared to the numerous diagnoses of COVID-19 under development that employ AI to swiftly assess chest CT imaging [26]. Nonetheless, this study was based on a restricted review without considering a summary sensitivity and the specificity of the receiver operating characteristic curve (ROC). Hence, a more rigid multi-center study investigating the predictive role of AI-assisted chest X-ray scans for COVID-19 is guaranteed to improve our understanding of the accuracy of AI diagnostic devices.
According to the preceding study, we implemented a meta-analysis to evaluate the diagnostic value of AI-assisted chest X-ray scans to determine their accuracy in COVID-19 patients.
2. Materials and Methods
2.1. Seek Tactics and Picking Standard
We used PubMed, Cochrane Library, MedRxiv, ArXiv, and Embase to search for research published from 1 January 2020 to 30 May 2022 involving “machine learning”, “artificial intelligence”, “medical image”, “SARS-CoV-2”, and “COVID-19”, because publication related to AI was distinct from traditional therapeutic publication. Only research that considered chest X-rays to probe the usage of AI were chosen for the review. From the selected papers, the following data were excerpted: the number of datasets used for training and validation, the proportion of COVID-19 scans within the dataset, and the sensitivity, specificity, and area under curve (AUC) of the proffered manner. We also considered whether the datasets and model code were estimable. The research was then classified by imaging process: chest X-ray.
The following MeSH terms and their combinations were searched, including (Machine learning OR Artificial intelligence OR Medical image) and (SARS-CoV-2 OR COVID-19) and (AUC OR ROC OR Sensitivity OR Specificity).
2.2. Procedures
This review was performed in accordance with the preferred reporting items for systematic reviews and meta-analyses (PRISMA). Two researchers (IST and PCH) independently extracted the data from the included studies. Two researchers (IST and PCH) performed the initial screening, manually searching the results and selecting articles for full-text retrieval in the title (or abstract) review process. The opinion of the third reviewer (WLS) was considered if identification was inconsistent between IST and PCH. They determined eligibility by screening the titles and abstracts of the retrieved studies and extracted data by constructing a excel table. They validated any discrepancies and addressed concerns through discussion to achieve consensus in the extracted data. Information bias may be generated from an image data source, which also influenced the results of study. Several sources of imaging data included the type of imaging contexts (i.e., three- or two-dimensional), type of AI approach, sensitivity, and specificity extracted from recruited studies. The risk of bias was frontally evaluated by two independent reviewers (IST and PCH). Next, we reviewed each study using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) guidelines [27]. The QUADAS-2 tool was used to assess the methodological quality of the included studies [27]. The QUADAS-2 tool consisted of four key domains covering patient selection, index test, reference standard, and flow and timing. We identified the risk of bias as ’high’, ‘low’, or ‘unclear’. The result of the risk of bias for the recruited studies was presented using a plot.
2.3. Statistical Analysis
According to the patients with or without a COVID-19 diagnosis, we calculated counts of true positives (i.e., sensitivity multiply number of COVID-19 diagnoses), false positives (i.e., (1-specificity) multiply number of non-COVID-19 diagnoses), true negatives (i.e., specificity multiply number of non-COVID-19 diagnoses), and false negatives (i.e., (1-sensitivity) multiply number of COVID-19 diagnoses) of COVID-19 from the included research and calculated the pooled estimates for sensitivity and specificity and corresponding 95% CI.
The “mada” package [28] was used to assess the data to explore the pooled sensitivity and specificity and their 95% CI. The “mada” package [28] was also used to evaluate the summary receiver operating characteristic curve (SROC), which was used to calculate the AUC value. Finally, funnel plots were created to examine the publication bias of the included studies. All statistical analyses and graphical presentations mentioned above were implemented using R software (4.2.0 version, Vienna, Austria).
3. Results
3.1. Literature Selection and Quality Assessment
After evaluating and carefully screening all the studies from the databases, the literature search yielded nine eligible studies [29,30,31,32,33,34,35,36,37], including 39,603 participants (included 2976 COVID-19 cases and 36,627 non-COVID-19 individuals). The PRISMA flowchart of study selection is shown in Figure 1.
3.2. Risk of Bias Assessment
Based on the definition of the criterion standard for the detection of COVID-19 using chest X-rays assisted by AI system, data were extracted from the recruited studies for this meta-analysis. Extraction data included the first author, year of publication, the conducted study country, type of study, and number of patients. The QUADAS-2 tool was used to assess the quality and potential bias of nine studies. Four key domains covering patient selection, index test, reference standard, and flow and timing were assessed. The results of the literature quality assessment using QUADAS [27] are provided in Figure 2. Low risk bias implied confidence on the part of the literature reviewer that results represent the true diagnostic effect (such as sensitivity, specificity, and AUC). Figure 2 showed that 90% of the “overall risk of bias” item presented as having a low risk of bias.
3.3. General Characteristics
We summarize the general characteristics of all nine studies included in this meta-analysis in Table 1 and their research findings. The general characteristics of the recruited studies are shown in Table 1. We found that four studies were conducted in the United States of America and that the other five studies were conducted in different countries, respectively. Next, we included seven case-control studies and two retrospective studies. For participants in this meta-analysis, this study included 2976 COVID-19 cases and 36,627 non-COVID-19-diagnosed individuals. The sensitivity and specificity of the nine studies are presented as a forest plot in Figure 3. We also presented 95% CI of sensitivity and specificity for the nine studies in Figure 3.
3.4. Results of Pooled Estimates for Sensitivity and False Positive Rate Analysis
For the random effects model, the pooled sensitivity was 0.9472 (p = 0.0338, 95% CI 0.9009–0.9959), and the pooled specificity was 0.9610 (p < 0.0001, 95% CI 0.9428–0.9795). Two-dimensional plots were provided by the “mada” package [28]. One is the crosshair plot, and the other is the ROC ellipse plot. In Figure 4, the crosshair was conducted using an arbitrary color, which made the crosshairs wider with increased weight. Bold purple crosshairs were presented due to their biggest weight among nine studies. The SROC was generated to assess the diagnostic value, and the results revealed an AUC of 0.98 (95% CI 0.94–1.00 in Figure 5).
3.5. Deeks’ Test
We used the formulas provided by Deeks [38]. The “mada” module also performed χ2 tests to evaluate the heterogeneity of sensitivities and specificities. The null hypotheses in both cases’ heterogeneity of sensitivities and specificities were equivalent. Test results for the equality of sensitivities: X-squared = 297.3168 with p < 0.0001; test results for equality of specificities: X-squared = 581.6488 with p < 0.0001. This showed that the heterogeneity results of sensitivities and specificities were not equal in both cases.
4. Discussion
COVID-19 is a severe acute respiratory syndrome caused by the SARS-CoV-2 virus, resulting in organ exhaustion and gradual death [39]. In September 2022, COVID-19 infections and deaths worldwide reached 620 and 6.5 million, respectively, (
There were seven case-control studies and two retrospective studies recruited in this meta-analysis due to the search strategy adopted in this study. Moreover, we focused on diagnostics of the COVID-19 epidemic rather than the novelties of AI, as clinical research studies are more widespread for other AI utilizations. Finding accessible small datasets in the public domain is a general challenge for overall research. Because COVID-19 has occurred only since late 2019, there are few images of COVID-19 patients accessible at separate institutions and in public domain datasets. Some studies have adopted similar datasets (Table 1). This is a drawback, as the algorithm trained on a specific dataset may not have the ability to execute as well when lending itself to the assorted data [40]. Simultaneously, the scarcity of external checks among the investigated studies may enhance this risk of bias.
Our evaluation of the repeatability of the results of the algorithms may have been limited owing to smaller datasets. Nevertheless, some research used datasets in public domain but neglected their clarity regarding the origin and disposition of the image data. This may cause a certain level of bias. Another review utilizing AI to interpret COVID-19 diagnoses revealed that the high level of bias in most of their papers was an effect of the small number of COVID-19 images [37]. However, small datasets are not incompatible with research on AI in COVID-19. The stated influence may be controversial. A previous study focusing on pulmonary nodules assessed with AI was based on a small sample of 186 patients [41]. Images used in the study were found in numerous public storehouses and taken from publications [36]. These images probably presented extreme and provocative cases of COVID-19 that may have been easier for the algorithm to perceive. Furthermore, some datasets were branched through vision ascent and the development of repetition. Among the nine studies, only two studies presented an independent examination with an external dataset (22%), and the two studies that adopted external validation presented average sensitivity and specificity results of 97.56% and 95.15%, respectively. The remaining seven studies without external validation presented average sensitivity and specificity values of 94.18% and 96.38%, respectively. Thus, evaluating the performance of externally checked models was superior to disregarding externally checked models [42]. A previous study revealed in another current review of this objective [43] that only 21% (13/62) assessed their algorithms on separate datasets. High performance in external testing provided solid evidence that the model may be extrapolated to another patient population. Such external validation may mitigate influence and controversy. To date, the performance in external testing confirmed that the model could be converted into clinical practice.
Some studies constructed on the sampling from large COVID-19 datasets to mitigate the uncertainty of prediction [44,45,46,47]. Then, a previous systematic review described AI-based diagnostic imaging (CT and chest X-ray) tools and showed that the performance of both CT and chest X-ray diagnostic tools may be limited by the scale of the dataset [48]. In addition, the well-balanced data had the benefit of the training of neural networks. Minor differences of in the variable class of data significantly affect the results of study [49]. Equal classes of data are also found in a previous review [50]. However, a small number of chest X-rays are still to be found in some studies conducted throughout the pandemic [51,52,53].
According to the diagnostic ability of CT, another meta-analysis conducted based on CT imaging settled as the future work. This meta-analysis extracted the diagnostic performance of AI-assisted CT-Scan for COVID-19. Some studies which based their meta-analysis on AI-assisted CT-Scan diagnostic performance for COVID-19 demonstrated an AUC from 0.95 to 0.97 [54,55]. Consequently, the AI-assisted CT-Scan for COVID-19 was associated with pneumonia based on objective investigation [56,57]. A review of recent publications proved that physicians, in detecting COVID-19, used an AI supportive system [26]. Community acquired pneumonia and lung diseases were detected with an AUC value of 0.96 using a deep learning model [58]. On the contrary, some published studies showed the fair performance (AUC values ranged from 0.732 to 0.87) of DL model-assisted COVID-19 detection [59,60,61]. However, molecular diagnostic tests, such as reverse transcription–polymerase chain reaction (RT-PCR), were still the most reliable diagnostic tool, rather than the AI-assisted CT-Scan for COVID-19 detection [62]. Because lungs are infected at the later stage of infection and present a certain level of confusion in identification through medical imaging (Figure S1), the importance of RT-PCR tests cannot be underestimated. Controversially, AI, which assisted the chest CT-Scan to classify diseases’ status, has an excellent performance of diagnoses, with AUC values ranging from 0.90 to 1.00 [36,63,64,65].
This study utilized the SROC analysis of diagnostic accuracy in the adoption of AI-assisted chest X-rays for the detection of COVID-19. In this meta-analysis, a strictly screened literature search was performed for all studies published in English between 1 January 2020 and 30 May 2022. The pooled estimates for sensitivity and specificity were 0.9472 (p = 0.0338, 95% CI 0.9009–0.9959) and 0.9610 (p < 0.0001, 95% CI 0.9428–0.9795), respectively. To evaluate the diagnostic value and prediction accuracy of using AI-assisted chest X-rays for COVID-19, the SROC curve indicated that the AUC value was 0.98 (95% CI 0.94–1.00). The results of this study suggested that the use of AI in chest X-rays has a significant value for forecasting COVID-19. Soda et al. [66] determined that the AUC of AI in chest X-rays in predicting COVID-19 was 0.63 (95% CI 0.52–0.74); nevertheless, only older adults aged above 65 years were included, and the number of subjects was not large. Another review study [67] summarized the literature of AI in chest X-rays for predicting COVID-19, albeit without synthesis evidence.
This meta-analysis verified the prediction value of AI in chest X-rays for COVID-19 patients (Table 2); however, there were some limitations and shortcomings. First, there was notable but not significant heterogeneity (I2 = 36.212, p = 0.129) on the diagnostic odds ratio in the review; the random effect results of the pooled sensitivity and specificity were 94.72 (p = 0.0338, 95% CI 90.09–99.59) and 96.10 (p < 0.0001, 95% CI 94.28–97.95), respectively. The funnel plot of the sensitivity and specificity may show a certain selection bias for recruited studies (Figure S2). However, such heterogeneity may have resulted from dissimilarity in areas, the cut-off value of AI in chest X-rays, the severity of COVID-19, and the resolution of the chest X-ray (Figure S2). Moreover, the predictive value of AI in chest X-rays was not consistent, and the predictive value of AI in chest X-rays had not been considered in the preceding studies, which may have negatively influenced the contributed sensitivity and specificity. Deep learning is a type of machine learning. Moreover, the goal of machine learning is to enable computers to think and behave independently of human input. However, deep learning is the process of teaching computers to reason by using structures inspired by the human brain. The recruited nine studies, which focus on a deep learning model (Table 1), may limit the results of this study.
5. Conclusions
In conclusion, AI can play a significant auxiliary role in utilizing chest X-rays when diagnosing COVID-19. Considering that COVID-19 is a complicated syndrome involving complex pathophysiological mechanisms, AI can assist chest X-rays though should not be considered as the single decisive signal for perceiving COVID-19. Other elements such as medical history, physical examination, and pathogenic microorganism tests should be implemented during the clinical diagnostic procedure. Future studies should consider clinical comparisons and external validation.
Conceptualization, I.-S.T.; methodology, I.-S.T. and P.-C.H.; software, T.-H.H.; validation, I.-S.T. and P.-C.H.; formal analysis, I.-S.T.; investigation, I.-S.T. and W.-L.S.; resources, S.-C.C.; data curation, I.-S.T., P.-C.H. and W.-L.S.; writing—original draft preparation, I.-S.T.; writing—review and editing, I.-S.T. and.; visualization, I.-S.T. and T.-H.H.; funding acquisition. All authors have read and agreed to the published version of the manuscript.
Not applicable.
Not applicable.
The data used to support the findings of this study are including within the article.
We acknowledge valuables comments and suggestions from three anonymous reviewers.
The authors declare no conflict of interest.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Enlarge this image.Figure 3. The forest plot of sensitivity and specificity results of the nine studies.
Enlarge this image.Figure 4. Pooled estimates for sensitivity and false positive rate analysis. Crosshairs wider with increased weight and colored arbitrarily.
Enlarge this image.Figure 5. The summary operating characteristic curve and the AUC were 0.98 (95% CI 0.94–1.00). By default, the point estimate of the pair of sensitivity and false positive rate is also plotted together with a confidence region (i.e., conf.region).
The general characteristics of the recruited studies.
| Study Number | First Author | Publication Year | Country | Study |
Dataset | Deep Learning Model | All Data | COVID | Non-COVID | Sensitivity | Specificity |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Borkowski [ |
2020 | United |
Case |
COVID-19/non- |
Microsoft |
1000 | 500 | 500 | 100 | 95 |
| 2 | Zokaeinikoo [ |
2021 | United |
Case |
COVID-19/non- |
AIDCOV using |
5801 | 269 | 5532 | 99.3 | 99.98 |
| 3 | Keidar [ |
2021 | Israel | Retrospective | COVID- |
RetNet50 | 2427 | 360 | 2067 | 87.1 | 92.4 |
| 4 | Ahmed [ |
2021 | Japan | Case |
COVID/non- |
HRNet | 1410 | 410 | 1000 | 98.53 | 98.52 |
| 5 | Kikkisetti [ |
2020 | United |
Retrospective | COVID/bacterial |
VGG-16 | 2031 | 445 | 1586 | 79 | 93 |
| 6 | Shibly [ |
2020 | Bangladesh | Case |
COVID/non- |
Faster R-CNN | 19,250 | 283 | 18,967 | 97.65 | 95.48 |
| 7 | Gomes [ |
2020 | Brazil | Case |
COVID- |
IKONOS | 6320 | 464 | 5856 | 97.7 | 99.3 |
| 8 | Ko [ |
2020 | South |
Case |
COVID- |
DarkNet-19 | 1125 | 125 | 1000 | 95.13 | 95.3 |
| 9 | Sharma [ |
2020 | United |
Case |
COVID-19/non |
Residual Att |
239 | 120 | 119 | 100 | 96 |
The characteristics and performance of diagnostic test results.
| Test Result | |||
| Total (n = 39,603) | Positive | Negative | |
| True condition | COVID-19 (n = 2976) | 2804.79 | 171.21 |
| Non COVID-19 (n = 36,627) | 1259.0788 | 35,367.9212 |
Supplementary Materials
The following supporting information can be downloaded at:
References
1. Hu, B.; Guo, H.; Zhou, P.; Shi, Z.L. Characteristics of SARS-CoV-2 and COVID-19. Nat. Rev. Microbiol.; 2021; 19, pp. 141-154. [DOI: https://dx.doi.org/10.1038/s41579-020-00459-7]
2. Wu, C.; Chen, X.; Cai, Y.; Xia, J.; Zhou, X.; Xu, S.; Huang, H.; Zhang, L.; Zhou, X.; Du, C. et al. Risk Factors Associated with Acute Respiratory Distress Syndrome and Death in Patients with Coronavirus Disease 2019 Pneumonia in Wuhan, China. JAMA Intern. Med.; 2020; 180, pp. 934-943. [DOI: https://dx.doi.org/10.1001/jamainternmed.2020.0994] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32167524]
3. Ayoubkhani, D.; Khunti, K.; Nafilyan, V.; Maddox, T.; Humberstone, B.; Diamond, I.; Banerjee, A. Post-covid syndrome in individuals admitted to hospital with COVID-19: Retrospective cohort study. BMJ; 2021; 372, n693. [DOI: https://dx.doi.org/10.1136/bmj.n693]
4. Han, J.; Shi, L.X.; Xie, Y.; Zhang, Y.J.; Huang, S.P.; Li, J.G.; Wang, H.R.; Shao, S.F. Analysis of factors affecting the prognosis of COVID-19 patients and viral shedding duration. Epidemiol. Infect.; 2020; 148, e125. [DOI: https://dx.doi.org/10.1017/S0950268820001399]
5. Sauter, A.P.; Andrejewski, J.; De Marco, F.; Willer, K.; Gromann, L.B.; Noichl, W.; Kriner, F.; Fischer, F.; Braun, C.; Koehler, T. et al. Optimization of tube voltage in X-ray dark-field chest radiography. Sci. Rep.; 2019; 9, 8699. [DOI: https://dx.doi.org/10.1038/s41598-019-45256-2] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31213645]
6. Berk, I.A.H.V.D.; Kanglie, M.M.N.P.; van Engelen, T.S.R.; Altenburg, J.; Annema, J.T.; Beenen, L.F.M.; Boerrigter, B.; Bomers, M.K.; Bresser, P.; Eryigit, E. et al. Ultra-low-dose CT versus chest X-ray for patients suspected of pulmonary disease at the emergency department: A multicentre randomised clinical trial. Thorax; 2022; pp. 1-8. [DOI: https://dx.doi.org/10.1136/thoraxjnl-2021-218337]
7. Souid, A.; Sakli, N.; Sakli, H. Classification and Predictions of Lung Diseases from Chest X-rays Using MobileNet V2. Appl. Sci.; 2021; 11, 2751. [DOI: https://dx.doi.org/10.3390/app11062751]
8. Xie, Y.; Xu, E.; Bowe, B.; Al-Aly, Z. Long-term cardiovascular outcomes of COVID-19. Nat. Med.; 2022; 28, pp. 583-590. [DOI: https://dx.doi.org/10.1038/s41591-022-01689-3]
9. Reisi-Vanani, V.; Lorigooini, Z.; Dayani, M.A.; Mardani, M.; Rahmani, F. Massive intraperitoneal hemorrhage in patients with COVID-19: A case series. J. Thromb. Thrombolysis; 2021; 52, pp. 338-344. [DOI: https://dx.doi.org/10.1007/s11239-021-02447-x]
10. Fan, E.; Beitler, J.R.; Brochard, L.; Calfee, C.S.; Ferguson, N.D.; Slutsky, A.S.; Brodie, D. COVID-19-associated acute respiratory distress syndrome: Is a different approach to management warranted?. Lancet Respir. Med.; 2020; 8, pp. 816-821. [DOI: https://dx.doi.org/10.1016/S2213-2600(20)30304-0]
11. Guan, W.J.; Ni, Z.Y.; Hu, Y.; Liang, W.H.; Ou, C.Q.; He, J.X.; Liu, L.; Shan, H.; Lei, C.L.; Hui, D.S.C. et al. Clinical characteristics of coronavirus disease 2019 in China. N. Engl. J. Med.; 2020; 382, pp. 1708-1720. [DOI: https://dx.doi.org/10.1056/NEJMoa2002032] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32109013]
12. Dabravolski, S.A.; Kavalionak, Y.K. SARS-CoV-2: Structural diversity, phylogeny, and potential animal host identification of spike glycoprotein. J. Med. Virol.; 2020; 92, pp. 1690-1694. [DOI: https://dx.doi.org/10.1002/jmv.25976] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32374452]
13. Murray, E.; Tomaszewski, M.; Guzik, T.J. Binding of SARS-CoV-2 and angiotensinconverting enzyme 2: Clinical implications. Cardiovasc. Res.; 2020; 116, pp. e87-e89. [DOI: https://dx.doi.org/10.1093/cvr/cvaa096]
14. Bernheim, A.; Mei, X.; Huang, M.; Yang, Y.; Fayad, Z.A.; Zhang, N.; Diao, K.; Lin, B.; Zhu, X.; Li, K. et al. Chest CT findings in coronavirus disease-19 (COVID-19): Relationship to duration of infection. Radiology; 2020; 295, 200463. [DOI: https://dx.doi.org/10.1148/radiol.2020200463] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32077789]
15. Heidinger, B.H.; Kifjak, D.; Prayer, F.; Beer, L.; Milos, R.I.; Röhrich, S.; Arndt, H.; Prosch, H. [Radiological manifestations of pulmonary diseases in COVID-19]. Radiologe; 2020; 60, pp. 908-915. [DOI: https://dx.doi.org/10.1007/s00117-020-00749-4]
16. Gravell, R.J.; Theodoreson, M.D.; Buonsenso, D.; Curtis, J. Radiological manifestations of COVID-19: Key points for the physician. Br. J. Hosp. Med.; 2020; 81, pp. 1-11. [DOI: https://dx.doi.org/10.12968/hmed.2020.0231]
17. Xie, X.; Zhong, Z.; Zhao, W.; Zheng, C.; Wang, F.; Liu, J. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: Relationship to negative RT-PCR testing. Radiology; 2020; 296, pp. E41-E45. [DOI: https://dx.doi.org/10.1148/radiol.2020200343]
18. Wong, H.Y.F.; Lam, H.Y.S.; Fong, A.H.; Leung, S.T.; Chin, T.W.; Lo, C.S.Y.; Lui, M.M.; Lee, J.C.Y.; Chiu, K.W.; Chung, T.W. et al. Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiology; 2020; 296, pp. E72-E78. [DOI: https://dx.doi.org/10.1148/radiol.2020201160]
19. Tahir, A.M.; Qiblawey, Y.; Khandakar, A.; Rahman, T.; Khurshid, U.; Musharavati, F.; Islam, M.T.; Kiranyaz, S.; Al-Maadeed, S.; Chowdhury, M.E.H. Deep Learning for Reliable Classification of COVID-19, MERS, and SARS from Chest X-ray Images. Cognit. Comput.; 2022; 14, pp. 1752-1772. [DOI: https://dx.doi.org/10.1007/s12559-021-09955-1]
20. Alsharif, M.H.; Alsharif, Y.H.; Chaudhry, S.A.; Albreem, M.A.; Jahid, A.; Hwang, E. Artificial intelligence technology for diagnosing COVID-19 cases: A review of substantial issues. Eur. Rev. Med. Pharmacol. Sci.; 2020; 24, pp. 9226-9233.
21. Menze, B.H.; Jakab, A.; Bauer, S.; Kalpathy-Cramer, J.; Farahani, K.; Kirby, J.; Burren, Y.; Porz, N.; Slotboom, J.; Wiest, R. et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging; 2015; 34, pp. 1993-2024. [DOI: https://dx.doi.org/10.1109/TMI.2014.2377694]
22. Li, C.; Xie, Y.; Sun, J. 3D intracranial artery segmentation using a convolutional autoencoder. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM); Kansas City, MO, USA, 13–16 November 2017; pp. 714-717.
23. Meijs, M.; Patel, A.; van de Leemput, S.C.; Prokop, M.; van Dijk, E.J.; de Leeuw, F.-E.; Meijer, F.J.A.; van Ginneken, B.; Manniesing, R. Robust segmentation of the full cerebral vasculature in 4D CT of suspected stroke patients. Sci. Rep.; 2017; 7, 15622. [DOI: https://dx.doi.org/10.1038/s41598-017-15617-w] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29142240]
24. Jacobi, A.; Chung, M.; Bernheim, A.; Eber, C. Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review. Clin. Imaging; 2020; 64, pp. 35-42. [DOI: https://dx.doi.org/10.1016/j.clinimag.2020.04.001] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32302927]
25. Cozzi, A.; Schiaffino, S.; Arpaia, F.; Della Pepa, G.; Tritella, S.; Bertolotti, P.; Menicagli, L.; Monaco, C.G.; Carbonaro, L.A.; Spairani, R. et al. Chest X-ray in the COVID-19 pandemic: Radiologists’ real-world reader performance. Eur. J. Radiol.; 2020; 132, 109272. [DOI: https://dx.doi.org/10.1016/j.ejrad.2020.109272] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32971326]
26. Ozsahin, I.; Sekeroglu, B.; Musa, M.S.; Mustapha, M.T.; Ozsahin, D.U. Review on Diagnosis of COVID-19 from Chest CT Images Using Artificial Intelligence. Comput. Math. Methods Med.; 2020; 2020, 9756518. [DOI: https://dx.doi.org/10.1155/2020/9756518]
27. Whiting, P.; Rutjes, A.W.; Reitsma, J.B.; Bossuyt, P.M.; Kleijnen, J. The development of QUADAS: A tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med. Res. Methodol.; 2003; 3, 25. [DOI: https://dx.doi.org/10.1186/1471-2288-3-25]
28. Doebler, P.; Holling, H. Meta-Analysis of Diagnostic Accuracy with mada. Cran.r-project.org Web site. Available online: https://cran.r-project.org/web/packages/mada/vignettes/mada.pdf (accessed on 15 October 2022).
29. Borkowski, A.A.; Viswanadham, N.A.; Thomas, L.B.; Guzman, R.D.; Deland, L.A.; Mastorides, S.M. Using Artificial Intelligence for COVID-19 Chest X-ray Diagnosis. Fed. Pract.; 2020; 37, pp. 398-404. [DOI: https://dx.doi.org/10.12788/fp.0045]
30. Zokaeinikoo, M.; Kazemian, P.; Mitra, P.; Kumara, S. AIDCOV: An Interpretable Artificial Intelligence Model for Detec-tion of COVID-19 from Chest Radiography Images. ACM Trans. Manag. Inf. Syst.; 2021; 12, pp. 1-20. [DOI: https://dx.doi.org/10.1145/3466690]
31. Keidar, D.; Yaron, D.; Goldstein, E.; Shachar, Y.; Blass, A.; Charbinsky, L.; Aharony, I.; Lifshitz, L.; Lumelsky, D.; Neeman, Z. et al. COVID-19 Classification of X-ray Images Using Deep Neural Networks. Eur. Radiol.; 2021; 31, pp. 9654-9663. [DOI: https://dx.doi.org/10.1007/s00330-021-08050-1]
32. Ahmed, S.; Hossain, T.; Hoque, O.B.; Sarker, S.; Rahman, S.; Shah, F.M. Automated COVID-19 Detection from Chest X-RayImages: A High Resolution Network (HRNet) Approach. SN Comput. Sci.; 2021; 2, 294. [DOI: https://dx.doi.org/10.1007/s42979-021-00690-w]
33. Kikkisetti, S.; Zhu, J.; Shen, B.; Li, H.; Duong, T. Deep-learning convolutional neural networks with transfer learning accurately classify COVID19 lung infection on portable chest radiographs. PeerJ; 2020; 8, e10309. [DOI: https://dx.doi.org/10.7717/peerj.10309] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33194447]
34. Shibly, K.H.; Dey, S.K.; Islam, M.T.U.; Rahman, M.M. COVID Faster R-CNN: A Novel Framework to Diagnose Novel Coronavirus Disease (COVID-19) in X-ray Images. Inform. Med. Unlocked; 2020; 20, 100405. [DOI: https://dx.doi.org/10.1016/j.imu.2020.100405] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32835082]
35. Gomes, J.C.; Barbosa, V.A.d.F.; Santana, M.A.; Bandeira, J.; Valenca, M.J.S.; de Souza, R.E.; Ismael, A.M.; dos Santos, W.P. IKONOS. An intelligent tool to support diagnosis of COVID-19 by texture analysis of X-ray images. Res. Biomed. Eng.; 2020; 38, pp. 15-28. [DOI: https://dx.doi.org/10.1007/s42600-020-00091-7]
36. Ko, H.; Chung, H.; Kang, W.S.; Kim, K.W.; Shin, Y.; Kang, S.J.; Lee, J.H.; Kim, Y.J.; Kim, N.Y.; Jung, H. et al. COVID-19 Pneumonia Diagnosis Using a Simple 2D Deep Learning Framework with a Single Chest CT Image: Model Development and Validation. J. Med. Internet Res.; 2020; 22, e19569. [DOI: https://dx.doi.org/10.2196/19569] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32568730]
37. Sharma, V.; Dyreson, C. COVID-19 Screening Using Residual Attention Network an Artificial Intelligence Approach. arXiv; 2020; arXiv: 2006.16106
38. Deeks, J.J. Systematic Reviews of Evaluations of Diagnostic and Screening Tests. Br. Med. J.; 2001; 323, pp. 157-162. [DOI: https://dx.doi.org/10.1136/bmj.323.7305.157] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/11463691]
39. Atzrodt, C.L.; Maknojia, I.; McCarthy, R.D.P.; Oldfield, T.M.; Po, J.; Ta, K.T.L.; Stepp, H.E.; Clements, T.P. A Guide to COVID-19: A global pandemic caused by the novel coronavirus SARS-CoV-2. FEBS J.; 2020; 287, pp. 3633-3650. [DOI: https://dx.doi.org/10.1111/febs.15375]
40. Futoma, J.; Simons, M.; Panch, T.; Doshi-Velez, F.; Celi, L.A. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit. Health; 2020; 2, pp. e489-e492. [DOI: https://dx.doi.org/10.1016/S2589-7500(20)30186-2]
41. Krarup, M.M.K.; Krokos, G.; Subesinghe, M.; Nair, A.; Fischer, B.M. Artificial Intelligence for the Characterization of Pulmonary Nodules, Lung Tumors and Mediastinal Nodes on PET/CT. Semin. Nucl. Med.; 2021; 51, pp. 143-156. [DOI: https://dx.doi.org/10.1053/j.semnuclmed.2020.09.001]
42. Roberts, M.; Driggs, D.; Thorpe, M.; Gilbey, J.; Yeung, M.; Ursprung, S.; Aviles-Rivero, A.I.; Etmann, C.; McCague, C.; Beer, L. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell.; 2021; 3, pp. 199-217. [DOI: https://dx.doi.org/10.1038/s42256-021-00307-0]
43. Zheng, C.; Deng, X.; Fu, Q.; Zhou, Q.; Feng, J.; Ma, H.; Liu, W.; Wang, X. Deep Learning-based Detection for COVID-19 from Chest CT using Weak Label. medRxiv; 2020; [DOI: https://dx.doi.org/10.1101/2020.03.12.20027185]
44. Alizadehsani, R.; Roshanzamir, M.; Hussain, S.; Khosravi, A.; Koohestani, A.; Zangooei, M.H.; Abdar, M.; Beykikhoshk, A.; Shoeibi, A.; Zare, A. et al. Handling of uncertainty in medical data using machine learning and probability theory techniques: A review of 30 years (1991–2020). Ann. Oper. Res.; 2021; pp. 1-42. advance online publication [DOI: https://dx.doi.org/10.1007/s10479-021-04006-2] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33776178]
45. Cohen, J.P.; Morrison, P.; Dao, L. COVID-19 Image Data Collection. arXiv; 2020; arXiv: 2003.11597
46. Calderon-Ramirez, S.; Yang, S.; Moemeni, A.; Colreavy-Donnelly, S.; Elizondo, D.A.; Oala, L.; Rodriguez-Capitan, J.; Jimenez-Navarro, M.; Lopez-Rubio, E.; Molina-Cabello, M.A. Improving uncertainty estimation with semi-supervised deep learning for COVID-19 detection using chest X-ray images. IEEE Access; 2021; 9, pp. 85442-85454. [DOI: https://dx.doi.org/10.1109/ACCESS.2021.3085418] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34812397]
47. Zhou, J.; Jing, B.; Wang, Z.; Xin, H.; Tong, H. SODA: Detecting COVID-19 in chest X-rays with semi-supervised open set domain adaptation. IEEE/ACM Trans. Comput. Biol. Bioinform.; 2021; 19, pp. 2605-2612. [DOI: https://dx.doi.org/10.1109/TCBB.2021.3066331]
48. Santosh, K.; Ghosh, S. COVID-19 imaging tools: How Big Data is big?. J. Med. Syst.; 2021; 45, 71. [DOI: https://dx.doi.org/10.1007/s10916-021-01747-2]
49. Wei, Q.; Dunbrack, R.L. The role of balanced training and testing data sets for binary classifiers in bioinformatics. PLoS ONE; 2013; 8, e67863. [DOI: https://dx.doi.org/10.1371/journal.pone.0067863]
50. Tabik, S.; Gomez-Rios, A.; Martin-Rodriguez, J.L.; Sevillano-Garcia, I.; Rey-Area, M.; Charte, D.; Guirado, E.; Suarez, J.L.; Luengo, J.; Valero-Gonzalez, M.A. et al. COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on chest X-ray images. IEEE J. Biomed. Health Inform.; 2020; 24, pp. 3595-3605. [DOI: https://dx.doi.org/10.1109/JBHI.2020.3037127] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33170789]
51. Chen, L.; Rezaei, T. A New Optimal Diagnosis System for Coronavirus (COVID-19) Diagnosis Based on Archimedes optimization algorithm on chest X-ray images. Comput. Intell. Neurosci.; 2021; 2021, 7788491. [DOI: https://dx.doi.org/10.1155/2021/7788491]
52. Sharifrazi, D.; Alizadehsani, R.; Roshanzamir, M.; Joloudari, J.H.; Shoeibi, A.; Jafari, M.; Hussain, S.; Sani, Z.A.; Hasanzadeh, F.; Khozeimeh, F. et al. Fusion of convolution neural network, support vector machine and Sobel filter for accurate detection of COVID-19 patients using X-ray images. Biomed. Signal Process. Control; 2021; 68, 102622. [DOI: https://dx.doi.org/10.1016/j.bspc.2021.102622]
53. Blain, M.; Kassin, M.T.; Varble, N.; Wang, X.; Xu, Z.; Xu, D.; Carrafiello, G.; Vespro, V.; Stellato, E.; Ierardi, A.M. et al. Determination of disease severity in COVID-19 patients using deep learning in chest X-ray images. Diagn. Interv. Radiol.; 2021; 27, pp. 20-27. [DOI: https://dx.doi.org/10.5152/dir.2020.20205]
54. Bai, H.X.; Wang, R.; Xiong, Z.; Hsieh, B.; Chang, K.; Halsey, K.; Tran, T.M.L.; Choi, J.W.; Wang, D.C.; Shi, L.B. et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology; 2020; 296, pp. E156-E165. [DOI: https://dx.doi.org/10.1148/radiol.2020201491] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32339081]
55. Zhang, K.; Liu, X.; Shen, J.; Li, Z.; Sang, Y.; Wu, X.; Zha, Y.; Liang, W.; Wang, C.; Wang, K. et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography. Cell; 2020; 181, pp. 1423-1433.e11. [DOI: https://dx.doi.org/10.1016/j.cell.2020.04.045]
56. Harmon, S.A.; Sanford, T.H.; Xu, S.; Turkbey, E.B.; Roth, H.; Xu, Z.; Yang, D.; Myronenko, A.; Anderson, V.; Amalou, A. et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat. Commun.; 2020; 11, 4080. [DOI: https://dx.doi.org/10.1038/s41467-020-17971-2]
57. Belfiore, M.P.; Urraro, F.; Grassi, R.; Giacobbe, G.; Patelli, G.; Cappabianca, S.; Reginelli, A. Artificial intelligence to codify lung CT in COVID-19 patients. Radiol. Med.; 2020; 125, pp. 500-504. [DOI: https://dx.doi.org/10.1007/s11547-020-01195-x]
58. Li, L.; Qin, L.; Xu, Z.; Yin, Y.; Wang, X.; Kong, B.; Bai, J.; Lu, Y.; Fang, Z.; Song, Q. et al. Using artificial intelligence to detect COVID-19 and community-acquired pneumonia based on pulmonary CT: Evaluation of the diagnostic accuracy. Radiology; 2020; 296, pp. E65-E71. [DOI: https://dx.doi.org/10.1148/radiol.2020200905] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32191588]
59. Wang, S.; Zha, Y.; Li, W.; Wu, Q.; Li, X.; Niu, M.; Wang, M.; Qiu, X.; Li, H.; Yu, H. et al. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis. Eur. Respir. J.; 2020; 2, 56. [DOI: https://dx.doi.org/10.1183/13993003.00775-2020] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32444412]
60. Wu, X.; Hui, H.; Niu, M.; Li, L.; Wang, L.; He, B.; Yang, X.; Li, L.; Li, H.; Tian, J. et al. Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: A multicentre study. Eur. J. Radiol.; 2020; 128, 109041. [DOI: https://dx.doi.org/10.1016/j.ejrad.2020.109041] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32408222]
61. Mei, X.; Lee, H.C.; Diao, K.Y.; Huang, M.; Lin, B.; Liu, C.; Xie, Z.; Ma, Y.; Robson, P.M.; Chung, M. et al. Artificial intelligence-enabled rapid diagnosis of patients with COVID-19. Nat. Med.; 2020; 26, pp. 1224-1228. [DOI: https://dx.doi.org/10.1038/s41591-020-0931-3]
62. Neri, E.; Miele, V.; Coppola, F.; Grassi, R. Use of CT and artificial intelligence in suspected or COVID-19 positive patients: Statement of the Italian Society of Medical and Interventional Radiology. Radiol. Med.; 2020; 125, pp. 505-508. [DOI: https://dx.doi.org/10.1007/s11547-020-01197-9]
63. Wang, X.; Deng, X.; Fu, Q.; Zhou, Q.; Feng, J.; Ma, H.; Liu, W.; Zheng, C. A weakly-supervised framework for COVID-19 classification and lesion localization from chest CT. IEEE Trans. Med. Imaging; 2020; 39, pp. 2615-2625. [DOI: https://dx.doi.org/10.1109/TMI.2020.2995965] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33156775]
64. Singh, D.; Kumar, V.; Vaishali Kaur, M. Classification of COVID-19 patients from chest CT images using multi-objective differential evolution-based convolutional neural networks. Eur. J. Clin. Microbiol. Infect. Dis.; 2020; 39, pp. 1379-1389. [DOI: https://dx.doi.org/10.1007/s10096-020-03901-z]
65. Javor, D.; Kaplan, H.; Kaplan, A.; Puchner, S.B.; Krestan, C.; Baltzer, P. Deep learning analysis provides accurate COVID-19 diagnosis on chest computed tomography. Eur. J. Radiol.; 2020; 133, 109402. [DOI: https://dx.doi.org/10.1016/j.ejrad.2020.109402] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33190102]
66. Soda, P.; D’Amico, N.C.; Tessadori, J.; Valbusa, G.; Guarrasi, V.; Bortolotto, C.; Akbar, M.U.; Sicilia, R.; Cordelli, E.; Fazzini, D. et al. AIforCOVID: Predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre study. Med. Image Anal.; 2021; 74, 102216. [DOI: https://dx.doi.org/10.1016/j.media.2021.102216] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34492574]
67. Mulrenan, C.; Rhode, K.; Fischer, B.M. A Literature Review on the Use of Artificial Intelligence for the Diagnosis of COVID-19 on CT and Chest X-ray. Diagnostics; 2022; 12, 869. [DOI: https://dx.doi.org/10.3390/diagnostics12040869] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35453917]
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Po-Chun Hsieh 2
; Wen-Lin, Su 3
; Tsung-Han Hsieh 1 ; Sheng-Chang, Chang 4 1 Department of Research, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
2 Department of Chinese Medicine, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
3 Division of Pulmonary Medicine, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan
4 Department of Medical Imaging, Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, New Taipei City 23142, Taiwan