Deep learning based screening model for hip diseases on plain radiographs

Abstract

Introduction

The interpretation of plain hip radiographs can vary widely among physicians. This study aimed to develop and validate a deep learning-based screening model for distinguishing normal hips from severe hip diseases on plain radiographs.

Methods

Electronic medical records and plain radiograph from 2004 to 2012 were used to construct two patient groups: the hip disease group (those who underwent total hip arthroplasty) and normal group. A total of 1,726 radiographs (500 normal hip radiographs and 1,226 radiographs with hip diseases, respectively) were included and were allocated for training (320 and 783), validation (80 and 196), and test (100 and 247) groups. Four different models were designed–raw image for both training and test set, preprocessed image for training but raw image for the test set, preprocessed images for both sets, and change of backbone algorithm from DenseNet to EfficientNet. The deep learning models were compared in terms of accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1-score, and area under the receiver operating characteristic curve (AUROC).

Results

The mean age of the patients was 54.0 ± 14.8 years in the hip disease group and 49.8 ± 14.9 years in the normal group. The final model showed highest performance in both the internal test set (accuracy 0.96, sensitivity 0.96, specificity 0.97, PPV 0.99, NPV 0.99, F1-score 0.97, and AUROC 0.99) and the external validation set (accuracy 0.94, sensitivity 0.93, specificity 0.96, PPV 0.95, NPV 0.93, F1-score 0.94, and AUROC 0.98). In the gradcam image, while the first model depended on unrelated marks of radiograph, the second and third model mainly focused on the femur shaft and sciatic notch, respectively.

Conclusion

The deep learning-based model showed high accuracy and reliability in screening hip diseases on plain radiographs, potentially aiding physicians in more accurately diagnosing hip conditions.

Introduction

Hip joint-related pain is widely recognized, affecting both young athletes and elderly patients, and can cause severe functional disabilities [1–3]. Among the various causes of hip pain, diseases such as osteonecrosis of the femoral head (ONFH) and osteoarthritis of the hip can lead to loss of ambulatory function if not treated appropriately [4,5]. A bedridden state due to these conditions can result in numerous complications, including aspiration pneumonia, pressure sores, venous thromboembolism, pulmonary embolism, and major adverse cardiac events (MACE) [6–10].

However, diagnosing the cause of hip pain is often very challenging. Because the hip joint is located deep beneath thick layers of large muscles, the pain is frequently referred to the groin, trochanter, or buttock regions [11]. In addition, the differential diagnosis of hip pain widely varies among different conditions, and even orthopedic surgeons often struggle to diagnose accurately [12–15]. In a retrospective study of 150 patients, over 80% were misdiagnosed by primary physicians [16]. Interestingly, the study also found that the problems of lumbar spine such as spinal stenosis were the third most common cause of hip pain [16]. Conversely, Nakamura et al. found that 8% of patients with ONFH and 17% of those with osteoarthritis of hip exhibited low back pain [17,18]. In clinical settings, many patients who have problems in the hip joint are commonly misdiagnosed as simple muscle sprain, spine problems, or arthrosis of other joints even with the plain hip radiographs taken. These findings underline the importance of screening model in patients with hip pain to distinguish whether the hip joint should be thoroughly investigated.

For the diagnosis of hip diseases, plain hip radiographs remain to be the keystone. The plain radiographs are usually the first imaging test to be performed in patients with hip pain. However, the accuracy of plain hip radiographs in diagnosing the specific hip diseases is known to be limited, demanding the use of more advanced imaging techniques [19]. Although contemporary imaging modalities such as MRI or CT scans are more sensitive, they are not currently routinely recommended because they are expensive and time-consuming. Moreover, the high acquisition and maintenance costs of these advanced modalities limits their use in developing countries or rural areas [20–22].

With the advent of artificial intelligence (AI) technologies, the potential practicality of plain hip radiographs could be revisited. Deep learning algorithm, notably convolutional neural network (CNN) model, has displayed high performance in medical image classification task through supervised learning [23,24]. Although plain hip radiographs are difficult to provide final and specific diagnosis, the high availability of plain radiographs makes them a practical choice for screening hip disease with the aid of the CNN.

Previous studies have focused on detecting specific diseases or conditions using deep learning models on radiographs. However, to the best of our knowledge, no study has developed a screening model for a wide range of hip diseases using plain radiographs and deep learning algorithms. Therefore, the purpose of this study was to develop and validate a deep learning-based screening model to distinguish normal hips from those with significant pathological changes, using plain radiographs as a preliminary step toward diagnosing severe hip diseases. This study aims to contribute to the field by developing a novel deep learning-based screening model for detecting hip diseases using plain radiographs, providing a practical tool that can be applied in clinical settings. Unlike previous studies that focused on specific conditions, this model is designed to screen a wide range of hip diseases with high accuracy.

Materials and methods

The study is organized into several key phases: data collection, model development, model training using different algorithms, and validation using external data. Each phase is designed to ensure that the model is robust and generalizable across different clinical settings.

Datasets

This study was conducted as a multicenter study on two tertiary hospitals. The institutional review board of the local medical institution approved the design and protocol of this retrospective study (No B-2307-843-106) and waived the requirement for informed consent. The radiographs were retrospectively collected in both centers. The deep learning model was constructed using the data from one hospital and the external validation was performed in the other hospital. The normal hip radiographs were identified using a clinical data warehouse (CDW) database. The electronic medical record (EMR) program was adopted in 2004 in both hospitals and CDW database covers all the radiographic data from 2004. Eight hundred ninety-seven patients who had plain hip anteroposterior (AP) radiographs taken between 2004 and 2012, with radiologists’ reports indicating "no significant bony abnormality," "no bony abnormality," "no specific abnormal findings," "within normal limits," or "no definite bony lesion," were identified. Two of the authors (JWP, SMR) reviewed all radiographs and only included normal radiographs, excluding the patients who received previous hip surgery (n = 174). Among 723 radiographs, we selected 500 radiographs for analysis in the control group.

One thousand six hundred and thirty one patients who had hip diseases were identified as those who received total hip arthroplasty (THA) from 2004 to 2012 in the same institution. Hip radiographs of these patients were reviewed by two of the authors (JWP, SMR). Patients who received bilateral THA (n = 257) and who had prior hip surgery (n = 148) were excluded. A total of 1,226 radiographs with hip disease were included for the screening model construction.

The dataset was stratified as follows: training set (320 normal, 783 hip disease), validation set (80 normal, 196 hip disease), and test set (100 normal, 247 hip disease). To address the class imbalance issue, we employed PyTorch’s ImbalancedDatasetSampler during the training and validation phases. This approach facilitated oversampling of the minority class and undersampling of the majority class, ensuring balanced representation of both normal and abnormal cases in each epoch.

Model training

For model training, two authors (JWP, SMR) reviewed all radiographs and confirmed the diagnosis whether the radiograph showed evidences of hip diseases or was normal. Furthermore, they confirmed the diagnosis of each patient with their respective diagnosis on the radiographic findings and data from the EMR. The diagnoses of the included patients were listed in Table 1. The binary classification model (hip disease vs normal) was constructed through supervised learning. Among the total of 1,726 hip radiographs, training set, validation set, and test set included 1,103, 276, and 347 radiographs, respectively. Training set was used for the training of the screening model, while validation set was used for tuning to enhance the model performance, and test set was used for internal validation. The radiograph images were preprocessed by resizing to a standard resolution of 1024x1024 pixels, normalizing the pixel values to a range of 0 to 1.

[Figure omitted. See PDF.]

The neural network architecture employed in this study is based on the EfficientNet-B3 model, pre-trained on the ImageNet dataset. EfficientNet is a family of convolutional neural network models designed for efficient performance and scalability, making it a suitable choice for our medical image analysis task [25]. Notably, the model implementation was conducted using open-source software, specifically Python version 3.8.3, facilitated by libraries such as PyTorch (version 1.7) and MONAI (version 0.6.0), a dedicated library for medical image analysis.

During the training phase, a meticulously crafted algorithm iterated through batches of input data, calculating the loss using binary cross-entropy as the primary metric. This loss was then utilized to update the model parameters through the process of backpropagation, with the Adam optimizer employed for efficient convergence. Data augmentation and early stopping were employed to enhance generalization and prevent overfitting. These computational resources operated within a Linux environment running Ubuntu version 18.04.5 and utilized the CUDA platform version 11.0 from Nvidia for GPU acceleration. The system comprised of a 16-thread Intel Core i9-9900K CPU clocked at 3.60 GHz, complemented by 32 GB of DDR4 Synchronous Dynamic Random Access Memory (DRAM), and powered by an Nvidia GeForce RTX 2080 Ti graphics processing unit (GPU).

Model development

The screening model was developed through four consecutive versions. In the first model, raw image files (DICOM files) were used for both the training and test sets, and the DenseNet CNN model served as the backbone algorithm for classification. In the second model, images were preprocessed with center-cropping, resizing, and zero padding for training, but raw images were used for testing. The third model utilized preprocessed images for both the training and test sets. In the fourth model, the backbone algorithm was changed from DenseNet to EfficientNet, and a nonlocal block network was added to the algorithm [25–27]. The specific codes to train and develop the models are listed in the following link.

(https://github.com/smryuphd/Hip2D_SNUBH/blob/main/Binary_Classifier_240807_EN_NLB_github.ipynb)

Our final model utilizes an EfficientNet-B3 backbone followed by a nonlocal block. The input to the model is a preprocessed image of size 512x512x3. The EfficientNet-B3 backbone consists of multiple mobile inverted bottleneck convolutional (MBConv) blocks, which efficiently balance model size and accuracy. This backbone extracts hierarchical features from the input image. Following the EfficientNet-B3 backbone, we implement a nonlocal block. This block is designed to capture long-range dependencies in the feature space, which is particularly beneficial for analyzing complex medical images like hip radiographs. The output from the nonlocal block is then passed through a global average pooling layer to reduce spatial dimensions, followed by a fully connected layer with sigmoid activation for binary classification (S1 Fig., S1 File).

External validation

The external validation was performed using the data from the other hospital. The radiographs of the normal group and the hip disease group were collected in the same manner as the data obtained to construct the model. Both normal and disease groups were selected from the hip radiographs of the patients who were treated between 2004 and 2012 for hip pain. The hip disease group consisted of patients who received THA and were selected ensuring a representative sample of the disease population. This approach allowed us to capture the full spectrum of disease severity and progression, while minimizing potential selection biases. Two hundred radiographs were selected from each group, respectively. The final (4^th) screening model was externally validated in this cohort.

Statistical analysis

The best-performing model was selected based on the validation loss on the validation dataset, with the optimal epoch determined by the lowest validation loss. The area under the receiver operating characteristic curve (AUROC) was calculated using the predicted values and true labels. The Youden index was then used to determine the optimal threshold, which facilitated the calculation of accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1-score [28].AUROC was calculated using the trapezoidal rule

Where TP = true positives, TN = true negatives, FP = false positives, FN = false negatives, PPV = positive predictive value, NPV = negative predictive value, AUROC = area under the receiver operating characteristic curve.

To improve the evaluation process, Gradient-weighted Class Activation Mapping (Grad-CAM) was used to highlight the areas of the input that had the most impact on the model’s decision-making [29].

Results

Patient demographics

Plain hip radiographs of 1,226 patients in hip disease group and 500 patients in normal group were included for the development of screening model (Table 2).

[Figure omitted. See PDF.]

Performance of each screening model

The result of first model showed accuracy, sensitivity, specificity, and F1-score of 0.96, 0.96, 0.98, and 0.97, respectively. However, the Grad-CAM image showed that the screening model trained using the unrelated marks such as “stand” or “R” (indicating right side) from the uncropped radiograph (Fig 1).

[Figure omitted. See PDF.]

The second model in which the preprocessed radiographs were used for the training set but raw radiographs were used for the test set showed the accuracy, sensitivity, specificity, and F1-score of 0.93, 0.90, 0.99, and 0.94, respectively. The performance decreased compared to the first model and the gradcam image showed that the second model mainly focused on the femur shaft rather than the hip joint (Fig 2).

[Figure omitted. See PDF.]

The third model in which the preprocessed radiographs were used both for training set and the test set showed the accuracy, sensitivity, specificity, and F1-score of 0.92, 0.89, 0.98, and 0.94, respectively. The gradcam image showed that the third model mainly focused on the sciatic notch rather than the hip joint (Fig 3).

[Figure omitted. See PDF.]

The fourth model in which the backbone algorithm was changed from DenseNet to EfficientNet with the addition of nonlocal block, showed the accuracy, sensitivity, specificity, and F1-score of 0.96, 0.96, 0.97, and 0.97, respectively (Fig 4).

[Figure omitted. See PDF.]

The outcomes of the four models are summarized in Table 3 and Fig 5.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

External validation

There were 73 (37%) men in the normal group and 62 (31%) men in the hip disease group. The mean age of patients in normal group and hip disease group was 63.4 ± 9.8 years and 62.4 ± 16.1 years, respectively. The accuracy, sensitivity, specificity, F1 score, and AUROC of the final model was 0.94, 0.93, 0.96, 0.94, and 0.98, respectively.

Discussion

The fourth screening model for hip disease developed in this study demonstrated very high performance with an AUROC of 0.99. External validation also confirmed the model’s outstanding performance with an AUROC of 0.98. This is a very encouraging outcome, especially considering that the diagnostic accuracy of plain hip radiographs has been historically limited [19].

Recently, deep learning-based diagnostic models for certain diseases have shown favorable results (Table 4) [30–40]. Most of the studies have focused on detection of the specific conditions such as osteoporosis [31,32,34,35,38,39,41] or fractures [30,36,40,42,43]. The prominent performance of deep learning-based models in diagnosis was abundantly studied using the chest radiograph, which is one of the most commonly taken radiographs [30,34,39,44]. Guo L. et al suggested that the diagnostic performance of chest radiographs were enhanced in recognition of various disease of the chest region with the AI assistance [44]. In 2022, Chae HD et al. found that the deep learning model and observers showed AUC of 0.85 and 0.84, respectively (p = 0.739) in the task of detecting cervical ossification of the posterior longitudinal ligament in plain radiographs [45]. In current study, we also found outstanding performance of screening model based on deep learning based algorithm. In fact, the performance of the developed screening model was even better than the previously reported diagnostic models because the aim of this model was to screen and differentiate only whether the patient has the hip disease or not rather than specifically classifying the diagnoses.

[Figure omitted. See PDF.]

The screening models on a wide range of conditions by the plain radiographs are not commonly reported compared to the detection of the specific diagnosis. This may be due to the broader range of target diseases potentially leading to lower performance in a clinical setting, which could hinder its clinical utility [46]. However, binary classification on only the presence of diseases—not diagnosing each—could be a feasible task. In 2019, Hwang et al. used a deep learning algorithm to identify abnormal chest radiographs and reported AUC of 0.95, sensitivity of 0.89, and specificity of 0.70 [47]. The authors suggested that this screening model could inform physicians and radiologists of crucial diseases that might require urgent diagnosis and management [47]. More recently, Niehues et al. used bedside chest radiographs of the patients in intensive care unit to screen 8 findings (cardiac congestion, pleural effusion, air-space opacification, pneumothorax, central venous catheter, thoracic drain, gastric tube, and tracheal tube/cannula) and reported AUC of 0.85 to 0.99 [48]. Unlike the diagnosis of a specific disease, these studies are conducted regarding certain clinical situations such as emergency departments or intensive care units [47,48]. The screening model developed in the present study could be useful in the smaller medical institutions where the hip specialists are not present. After the screening, the patients who require further evaluation and timely appropriate management could be referred to hip specialists, reducing the possibility of misdiagnosis and the delay of diagnosis.

There are a few limitations to this study. First, the hospitals where the investigation was performed were tertiary referral hospitals and the diagnoses of the hip disease might differ from those of the patients in local hospitals. However, the radiographs of the various conditions used to develop the screening model could enhance the sensitivity, which is the main advantage for the screening model. Second, the scores of external validation were slightly lower than those of the internal validation. This might be due to the overfitting of the internal cohort [49]. However, the scores of the external validation were still excellent, indicating the high generalizability of the developed screening method.

In conclusion, the screening model developed in this study showed excellent performance both in internal and external validation. This screening model for hip disease is expected to be highly beneficial for primary physicians treating patients who present with hip pain in clinical settings. Building on the results of this study, future work could focus on developing more advanced diagnostic models capable of automatically classifying specific hip diseases using deep learning algorithms on plain radiographs. Long-term clinical validation across diverse populations and healthcare settings is also essential to ensure the generalizability and practical application of these models in real-world clinical environments.

Supporting information

S1 Fig. Architecture of the proposed deep learning model for hip disease screening.

The model consists of an EfficientNet-B3 backbone for feature extraction, followed by a nonlocal block for capturing long-range dependencies. The extracted features are then processed through global average pooling, a fully connected layer, and a sigmoid activation for binary classification of normal vs. abnormal hip radiographs.

https://doi.org/10.1371/journal.pone.0318022.s001

(TIF)

S1 File. Model architectures.

https://doi.org/10.1371/journal.pone.0318022.s002

(DOCX)

References

1. 1. van der Waal JM, Bot SD, Terwee CB, van der Windt DA, Bouter LM, Dekker J. The course and prognosis of hip complaints in general practice. Ann Behav Med. 2006;31(3):297–308. Epub 2006/05/17. pmid:16700644.

* View Article

* PubMed/NCBI

* Google Scholar

2. 2. Impellizzeri FM, Jones DM, Griffin D, Harris-Hayes M, Thorborg K, Crossley KM, et al. Patient-reported outcome measures for hip-related pain: a review of the available evidence and a consensus statement from the International Hip-related Pain Research Network, Zurich 2018. Br J Sports Med. 2020;54(14):848–57. Epub 2020/02/19. pmid:32066573.

* View Article

* PubMed/NCBI

* Google Scholar

3. 3. Thorborg K, Rathleff MS, Petersen P, Branci S, Holmich P. Prevalence and severity of hip and groin pain in sub-elite male football: a cross-sectional cohort study of 695 players. Scand J Med Sci Sports. 2017;27(1):107–14. Epub 2015/12/09. pmid:26643978.

* View Article

* PubMed/NCBI

* Google Scholar

4. 4. van Berkel AC, Schiphof D, Waarsing JH, Runhaar J, van Ochten JM, Bindels PJE, et al. 10-Year natural course of early hip osteoarthritis in middle-aged persons with hip pain: a CHECK study. Ann Rheum Dis. 2021;80(4):487–93. Epub 2021/01/17. pmid:33451999; PubMed Central PMCID: PMC7958083.

* View Article

* PubMed/NCBI

* Google Scholar

5. 5. Mont MA, Zywiel MG, Marker DR, McGrath MS, Delanois RE. The natural history of untreated asymptomatic osteonecrosis of the femoral head: a systematic literature review. J Bone Joint Surg Am. 2010;92(12):2165–70. Epub 2010/09/17. pmid:20844158.

* View Article

* PubMed/NCBI

* Google Scholar

6. 6. Chatterton BD, Moores TS, Ahmad S, Cattell A, Roberts PJ. Cause of death and factors associated with early in-hospital mortality after hip fracture. Bone Joint J. 2015;97-B(2):246–51. Epub 2015/01/30. pmid:25628290.

* View Article

* PubMed/NCBI

* Google Scholar

7. 7. Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest. 2008;133(6 Suppl):381S–453S. Epub 2008/07/24. pmid:18574271.

* View Article

* PubMed/NCBI

* Google Scholar

8. 8. Hoffmann B, Dehkordi P, Khosrow-Khavar F, Goswami N, Blaber AP, Tavakolian K. Mechanical deconditioning of the heart due to long-term bed rest as observed on seismocardiogram morphology. NPJ Microgravity. 2022;8(1):25. Epub 2022/07/13. pmid:35821029; PubMed Central PMCID: PMC9276739

* View Article

* PubMed/NCBI

* Google Scholar

9. 9. Park JW, Lee YK, Kim HS, Kim JT, Ha YC, Koo KH. Perioperative Concerns and Complaints of Patients Undergoing Total Hip Arthroplasty. Clin Orthop Surg. 2023;15(1):37–41. Epub 2023/02/14. pmid:36778993; PubMed Central PMCID: PMC9880498.

* View Article

* PubMed/NCBI

* Google Scholar

10. 10. Kim YV, Song JH, Lim YW, Jo WL, Ha SH, Lee KH. Prevalence of Venous Thromboembolism after Immediate Screening in Hip Fracture Patients. Hip Pelvis. 2024;36(1):47–54. Epub 2024/02/29. pmid:38420737; PubMed Central PMCID: PMC10929542.

* View Article

* PubMed/NCBI

* Google Scholar

11. 11. Martin HD, Palmer IJ. History and physical examination of the hip: the basics. Curr Rev Musculoskelet Med. 2013;6(3):219–25. Epub 2013/07/09. pmid:23832778; PubMed Central PMCID: PMC4094013.

* View Article

* PubMed/NCBI

* Google Scholar

12. 12. Skendzel JG, Weber AE, Ross JR, Larson CM, Leunig M, Kelly BT, et al. The approach to the evaluation and surgical treatment of mechanical hip pain in the young patient: AAOS exhibit selection. J Bone Joint Surg Am. 2013;95(18):e133. Epub 2013/09/21. pmid:24048564.

* View Article

* PubMed/NCBI

* Google Scholar

13. 13. Battaglia PJ, D’Angelo K, Kettner NW. Posterior, Lateral, and Anterior Hip Pain Due to Musculoskeletal Origin: A Narrative Literature Review of History, Physical Examination, and Diagnostic Imaging. J Chiropr Med. 2016;15(4):281–93. Epub 2016/11/20. pmid:27857636; PubMed Central PMCID: PMC5106442.

* View Article

* PubMed/NCBI

* Google Scholar

14. 14. Karrasch C, Lynch S. Practical approach to hip pain. Med Clin North Am. 2014;98(4):737–54, xi. Epub 2014/07/06. pmid:24994049.

* View Article

* PubMed/NCBI

* Google Scholar

15. 15. Lee S, Kim HJ, Yoo JJ. Factors Affecting the Clinical Course of Subchondral Fatigue Fracture of the Femoral Head. Clin Orthop Surg. 2023;15(2):203–10. Epub 2023/04/04. pmid:37008972; PubMed Central PMCID: PMC10060783.

* View Article

* PubMed/NCBI

* Google Scholar

16. 16. Lee YJ, Kim SH, Chung SW, Lee YK, Koo KH. Causes of Chronic Hip Pain Undiagnosed or Misdiagnosed by Primary Physicians in Young Adult Patients: a Retrospective Descriptive Study. J Korean Med Sci. 2018;33(52):e339. Epub 2018/12/26. pmid:30584417; PubMed Central PMCID: PMC6300658.

* View Article

* PubMed/NCBI

* Google Scholar

17. 17. Nakamura J, Konno K, Orita S, Hagiwara S, Shigemura T, Nakajima T, et al. Distribution of hip pain in patients with idiopathic osteonecrosis of the femoral head. Mod Rheumatol. 2017;27(3):503–7. Epub 2016/07/28. pmid:27459136.

* View Article

* PubMed/NCBI

* Google Scholar

18. 18. Nakamura J, Oinuma K, Ohtori S, Watanabe A, Shigemura T, Sasho T, et al. Distribution of hip pain in osteoarthritis patients secondary to developmental dysplasia of the hip. Mod Rheumatol. 2013;23(1):119–24. Epub 2012/04/12. pmid:22492057.

* View Article

* PubMed/NCBI

* Google Scholar

19. 19. Clohisy JC, Carlisle JC, Trousdale R, Kim YJ, Beaule PE, Morgan P, et al. Radiographic evaluation of the hip has limited reliability. Clin Orthop Relat Res. 2009;467(3):666–75. Epub 2008/12/03. pmid:19048356; PubMed Central PMCID: PMC2635468.

* View Article

* PubMed/NCBI

* Google Scholar

20. 20. Atalabi OM, Lagunju IA, Tongo OO, Akinyinka OO. Cranial magnetic resonance imaging findings in kwashiorkor. Int J Neurosci. 2010;120(1):23–7. Epub 2010/02/05. pmid:20128668.

* View Article

* PubMed/NCBI

* Google Scholar

21. 21. Chabanova E, Larsen L, Logager VB, Moller JM, Thomsen HS. [Use of magnetic resonance imaging]. Ugeskr Laeger. 2014;176(1):50–4. Epub 2014/03/19. pmid:24629608.

* View Article

* PubMed/NCBI

* Google Scholar

22. 22. Ogbole GI, Adeyinka OA, Okolo CA, Ogun AO, Atalabi OM. Low field MR imaging of sellar and parasellar lesions: experience in a developing country hospital. Eur J Radiol. 2012;81(2):e139–46. Epub 2011/02/09. pmid:21300499.

* View Article

* PubMed/NCBI

* Google Scholar

23. 23. Shen D, Wu G, Suk HI. Deep Learning in Medical Image Analysis. Annu Rev Biomed Eng. 2017;19:221–48. Epub 2017/03/17. pmid:28301734; PubMed Central PMCID: PMC5479722.

* View Article

* PubMed/NCBI

* Google Scholar

24. 24. Ajagbe SA, Adigun MO. Deep learning techniques for detection and prediction of pandemic diseases: a systematic literature review. Multimed Tools Appl. 2023:1–35. Epub 2023/06/26. pmid:37362693; PubMed Central PMCID: PMC10226029.

* View Article

* PubMed/NCBI

* Google Scholar

25. 25. Tan MX, Le QV. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Pr Mach Learn Res. 2019;97. WOS:000684034306026.

* View Article

* Google Scholar

26. 26. Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely Connected Convolutional Networks. Proc Cvpr Ieee. 2017:2261–9. WOS:000418371402035.

* View Article

* Google Scholar

27. 27. Wang XL, Girshick R, Gupta A, He KM. Non-local Neural Networks. 2018 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (Cvpr). 2018:7794–803. WOS:000457843607098.

* View Article

* Google Scholar

28. 28. Fluss R, Faraggi D, Reiser B. Estimation of the Youden index and its associated cutoff point. Biometrical J. 2005;47(4):458–72. WOS:000231492300005. pmid:16161804

* View Article

* PubMed/NCBI

* Google Scholar

29. 29. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vision. 2020;128(2):336–59. WOS:000511807100003.

* View Article

* Google Scholar

30. 30. Cheng CT, Kuo LW, Ouyang CH, Hsu CP, Lin WC, Fu CY, et al. Development and evaluation of a deep learning-based model for simultaneous detection and localization of rib and clavicle fractures in trauma patients’ chest radiographs. Trauma Surg Acute Care Open. 2024;9(1):e001300. Epub 2024/04/22. pmid:38646620; PubMed Central PMCID: PMC11029226.

* View Article

* PubMed/NCBI

* Google Scholar

31. 31. Ho CS, Chen YP, Fan TY, Kuo CF, Yen TY, Liu YC, et al. Application of deep learning neural network in predicting bone mineral density from plain X-ray radiography. Arch Osteoporos. 2021;16(1):153. Epub 2021/10/10. pmid:34626252.

* View Article

* PubMed/NCBI

* Google Scholar

32. 32. Hsieh CI, Zheng K, Lin C, Mei L, Lu L, Li W, et al. Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning. Nat Commun. 2021;12(1):5472. Epub 2021/09/18. pmid:34531406; PubMed Central PMCID: PMC8446034.

* View Article

* PubMed/NCBI

* Google Scholar

33. 33. Ito S, Nakashima H, Segi N, Ouchida J, Oda M, Yamauchi I, et al. Automated Detection of the Thoracic Ossification of the Posterior Longitudinal Ligament Using Deep Learning and Plain Radiographs. Biomed Res Int. 2023;2023:8495937. Epub 2023/12/06. pmid:38054045; PubMed Central PMCID: PMC10695689 publication of this paper.

* View Article

* PubMed/NCBI

* Google Scholar

34. 34. Jang M, Kim M, Bae SJ, Lee SH, Koh JM, Kim N. Opportunistic Osteoporosis Screening Using Chest Radiographs With Deep Learning: Development and External Validation With a Cohort Dataset. J Bone Miner Res. 2022;37(2):369–77. Epub 2021/11/24. pmid:34812546.

* View Article

* PubMed/NCBI

* Google Scholar

35. 35. Jang R, Choi JH, Kim N, Chang JS, Yoon PW, Kim CH. Prediction of osteoporosis from simple hip radiography using deep learning algorithm. Sci Rep. 2021;11(1):19997. Epub 2021/10/09. pmid:34620976; PubMed Central PMCID: PMC8497544.

* View Article

* PubMed/NCBI

* Google Scholar

36. 36. Kim JH, Mo YC, Choi SM, Hyun Y, Lee JW. Detecting Ankle Fractures in Plain Radiographs Using Deep Learning with Accurately Labeled Datasets Aided by Computed Tomography: A Retrospective Observational Study. Appl Sci-Basel. 2021;11(19). ARTN 8791 WOS:000709119000001.

* View Article

* Google Scholar

37. 37. Magneli M, Borjali A, Takahashi E, Axenhus M, Malchau H, Moratoglu OK, et al. Application of deep learning for automated diagnosis and classification of hip dysplasia on plain radiographs. BMC Musculoskelet Disord. 2024;25(1):117. Epub 2024/02/10. pmid:38336666; PubMed Central PMCID: PMC10854089.

* View Article

* PubMed/NCBI

* Google Scholar

38. 38. Mao L, Xia Z, Pan L, Chen J, Liu X, Li Z, et al. Deep learning for screening primary osteopenia and osteoporosis using spine radiographs and patient clinical covariates in a Chinese population. Front Endocrinol (Lausanne). 2022;13:971877. Epub 2022/10/01. pmid:36176468; PubMed Central PMCID: PMC9513384.

* View Article

* PubMed/NCBI

* Google Scholar

39. 39. Sato Y, Yamamoto N, Inagaki N, Iesaki Y, Asamoto T, Suzuki T, et al. Deep Learning for Bone Mineral Density and T-Score Prediction from Chest X-rays: A Multicenter Study. Biomedicines. 2022;10(9). Epub 2022/09/24. pmid:36140424; PubMed Central PMCID: PMC9496220.

* View Article

* PubMed/NCBI

* Google Scholar

40. 40. Yoon AP, Lee YL, Kane RL, Kuo CF, Lin C, Chung KC. Development and Validation of a Deep Learning Model Using Convolutional Neural Networks to Identify Scaphoid Fractures in Radiographs. JAMA Netw Open. 2021;4(5):e216096. Epub 2021/05/07. pmid:33956133; PubMed Central PMCID: PMC8103226

* View Article

* PubMed/NCBI

* Google Scholar

41. 41. Lee KS, Jung SK, Ryu JJ, Shin SW, Choi J. Evaluation of Transfer Learning with Deep Convolutional Neural Networks for Screening Osteoporosis in Dental Panoramic Radiographs. J Clin Med. 2020;9(2). Epub 2020/02/07. pmid:32024114; PubMed Central PMCID: PMC7074309.

* View Article

* PubMed/NCBI

* Google Scholar

42. 42. Gao Y, Soh NYT, Liu N, Lim G, Ting D, Cheng LT, et al. Application of a deep learning algorithm in the detection of hip fractures. iScience. 2023;26(8):107350. Epub 2023/08/09. pmid:37554447; PubMed Central PMCID: PMC10404720.

* View Article

* PubMed/NCBI

* Google Scholar

43. 43. Hong N, Cho SW, Shin S, Lee S, Jang SA, Roh S, et al. Deep-Learning-Based Detection of Vertebral Fracture and Osteoporosis Using Lateral Spine X-Ray Radiography. J Bone Miner Res. 2023;38(6):887–95. Epub 2023/04/12. pmid:37038364.

* View Article

* PubMed/NCBI

* Google Scholar

44. 44. Guo L, Zhou C, Xu J, Huang C, Yu Y, Lu G. Deep Learning for Chest X-ray Diagnosis: Competition Between Radiologists with or Without Artificial Intelligence Assistance. J Imaging Inform Med. 2024. Epub 2024/02/09. pmid:38332402.

* View Article

* PubMed/NCBI

* Google Scholar

45. 45. Chae HD, Hong SH, Yeoh HJ, Kang YR, Lee SM, Kim M, et al. Improved diagnostic performance of plain radiography for cervical ossification of the posterior longitudinal ligament using deep learning. PLoS One. 2022;17(4):e0267643. Epub 2022/04/28. pmid:35476649; PubMed Central PMCID: PMC9045646

* View Article

* PubMed/NCBI

* Google Scholar

46. 46. Park SH, Han K. Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction. Radiology. 2018;286(3):800–9. Epub 2018/01/09. pmid:29309734.

* View Article

* PubMed/NCBI

* Google Scholar

47. 47. Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019;293(3):573–80. Epub 2019/10/23. pmid:31638490.

* View Article

* PubMed/NCBI

* Google Scholar

48. 48. Niehues SM, Adams LC, Gaudin RA, Erxleben C, Keller S, Makowski MR, et al. Deep-Learning-Based Diagnosis of Bedside Chest X-ray in Intensive Care and Emergency Medicine. Invest Radiol. 2021;56(8):525–34. Epub 2021/04/08. pmid:33826549.

* View Article

* PubMed/NCBI

* Google Scholar

49. 49. Cawley GC, Talbot NLC. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. J Mach Learn Res. 2010;11:2079–107. WOS:000282523000006.

* View Article

* Google Scholar

Citation: Park J-W, Ryu SM, Kim H-S, Lee Y-K, Yoo JJ (2025) Deep learning based screening model for hip diseases on plain radiographs. PLoS ONE 20(2): e0318022. https://doi.org/10.1371/journal.pone.0318022

About the Authors:

Jung-Wee Park

Roles: Data curation, Writing – original draft

Affiliations: Department of Orthopaedic Surgery, Seoul National University College of Medicine, Seoul, South Korea, Department of Orthopaedic Surgery, Seoul National University Bundang Hospital, Seongnam, South Korea

Seung Min Ryu

Roles: Formal analysis

Affiliation: Department of Orthopaedic Surgery, Seoul Medical Center, Seoul, Republic of Korea

Hong-Seok Kim

Roles: Data curation

Affiliations: Department of Orthopaedic Surgery, Seoul National University College of Medicine, Seoul, South Korea, Department of Orthopaedic Surgery, Seoul National University Hospital, Seoul, South Korea

Young-Kyun Lee

Roles: Supervision, Visualization

Affiliations: Department of Orthopaedic Surgery, Seoul National University College of Medicine, Seoul, South Korea, Department of Orthopaedic Surgery, Seoul National University Bundang Hospital, Seongnam, South Korea

Jeong Joon Yoo

Roles: Conceptualization, Validation, Writing – review & editing

E-mail: [email protected]

Affiliations: Department of Orthopaedic Surgery, Seoul National University College of Medicine, Seoul, South Korea, Department of Orthopaedic Surgery, Seoul National University Hospital, Seoul, South Korea

ORICD: https://orcid.org/0000-0002-6304-0101

References

1. van der Waal JM, Bot SD, Terwee CB, van der Windt DA, Bouter LM, Dekker J. The course and prognosis of hip complaints in general practice. Ann Behav Med. 2006;31(3):297–308. Epub 2006/05/17. pmid:16700644.

2. Impellizzeri FM, Jones DM, Griffin D, Harris-Hayes M, Thorborg K, Crossley KM, et al. Patient-reported outcome measures for hip-related pain: a review of the available evidence and a consensus statement from the International Hip-related Pain Research Network, Zurich 2018. Br J Sports Med. 2020;54(14):848–57. Epub 2020/02/19. pmid:32066573.

3. Thorborg K, Rathleff MS, Petersen P, Branci S, Holmich P. Prevalence and severity of hip and groin pain in sub-elite male football: a cross-sectional cohort study of 695 players. Scand J Med Sci Sports. 2017;27(1):107–14. Epub 2015/12/09. pmid:26643978.

4. van Berkel AC, Schiphof D, Waarsing JH, Runhaar J, van Ochten JM, Bindels PJE, et al. 10-Year natural course of early hip osteoarthritis in middle-aged persons with hip pain: a CHECK study. Ann Rheum Dis. 2021;80(4):487–93. Epub 2021/01/17. pmid:33451999; PubMed Central PMCID: PMC7958083.

5. Mont MA, Zywiel MG, Marker DR, McGrath MS, Delanois RE. The natural history of untreated asymptomatic osteonecrosis of the femoral head: a systematic literature review. J Bone Joint Surg Am. 2010;92(12):2165–70. Epub 2010/09/17. pmid:20844158.

6. Chatterton BD, Moores TS, Ahmad S, Cattell A, Roberts PJ. Cause of death and factors associated with early in-hospital mortality after hip fracture. Bone Joint J. 2015;97-B(2):246–51. Epub 2015/01/30. pmid:25628290.

7. Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest. 2008;133(6 Suppl):381S–453S. Epub 2008/07/24. pmid:18574271.

8. Hoffmann B, Dehkordi P, Khosrow-Khavar F, Goswami N, Blaber AP, Tavakolian K. Mechanical deconditioning of the heart due to long-term bed rest as observed on seismocardiogram morphology. NPJ Microgravity. 2022;8(1):25. Epub 2022/07/13. pmid:35821029; PubMed Central PMCID: PMC9276739

9. Park JW, Lee YK, Kim HS, Kim JT, Ha YC, Koo KH. Perioperative Concerns and Complaints of Patients Undergoing Total Hip Arthroplasty. Clin Orthop Surg. 2023;15(1):37–41. Epub 2023/02/14. pmid:36778993; PubMed Central PMCID: PMC9880498.

10. Kim YV, Song JH, Lim YW, Jo WL, Ha SH, Lee KH. Prevalence of Venous Thromboembolism after Immediate Screening in Hip Fracture Patients. Hip Pelvis. 2024;36(1):47–54. Epub 2024/02/29. pmid:38420737; PubMed Central PMCID: PMC10929542.

11. Martin HD, Palmer IJ. History and physical examination of the hip: the basics. Curr Rev Musculoskelet Med. 2013;6(3):219–25. Epub 2013/07/09. pmid:23832778; PubMed Central PMCID: PMC4094013.

12. Skendzel JG, Weber AE, Ross JR, Larson CM, Leunig M, Kelly BT, et al. The approach to the evaluation and surgical treatment of mechanical hip pain in the young patient: AAOS exhibit selection. J Bone Joint Surg Am. 2013;95(18):e133. Epub 2013/09/21. pmid:24048564.

13. Battaglia PJ, D’Angelo K, Kettner NW. Posterior, Lateral, and Anterior Hip Pain Due to Musculoskeletal Origin: A Narrative Literature Review of History, Physical Examination, and Diagnostic Imaging. J Chiropr Med. 2016;15(4):281–93. Epub 2016/11/20. pmid:27857636; PubMed Central PMCID: PMC5106442.

14. Karrasch C, Lynch S. Practical approach to hip pain. Med Clin North Am. 2014;98(4):737–54, xi. Epub 2014/07/06. pmid:24994049.

15. Lee S, Kim HJ, Yoo JJ. Factors Affecting the Clinical Course of Subchondral Fatigue Fracture of the Femoral Head. Clin Orthop Surg. 2023;15(2):203–10. Epub 2023/04/04. pmid:37008972; PubMed Central PMCID: PMC10060783.

16. Lee YJ, Kim SH, Chung SW, Lee YK, Koo KH. Causes of Chronic Hip Pain Undiagnosed or Misdiagnosed by Primary Physicians in Young Adult Patients: a Retrospective Descriptive Study. J Korean Med Sci. 2018;33(52):e339. Epub 2018/12/26. pmid:30584417; PubMed Central PMCID: PMC6300658.

17. Nakamura J, Konno K, Orita S, Hagiwara S, Shigemura T, Nakajima T, et al. Distribution of hip pain in patients with idiopathic osteonecrosis of the femoral head. Mod Rheumatol. 2017;27(3):503–7. Epub 2016/07/28. pmid:27459136.

18. Nakamura J, Oinuma K, Ohtori S, Watanabe A, Shigemura T, Sasho T, et al. Distribution of hip pain in osteoarthritis patients secondary to developmental dysplasia of the hip. Mod Rheumatol. 2013;23(1):119–24. Epub 2012/04/12. pmid:22492057.

19. Clohisy JC, Carlisle JC, Trousdale R, Kim YJ, Beaule PE, Morgan P, et al. Radiographic evaluation of the hip has limited reliability. Clin Orthop Relat Res. 2009;467(3):666–75. Epub 2008/12/03. pmid:19048356; PubMed Central PMCID: PMC2635468.

20. Atalabi OM, Lagunju IA, Tongo OO, Akinyinka OO. Cranial magnetic resonance imaging findings in kwashiorkor. Int J Neurosci. 2010;120(1):23–7. Epub 2010/02/05. pmid:20128668.

21. Chabanova E, Larsen L, Logager VB, Moller JM, Thomsen HS. [Use of magnetic resonance imaging]. Ugeskr Laeger. 2014;176(1):50–4. Epub 2014/03/19. pmid:24629608.

22. Ogbole GI, Adeyinka OA, Okolo CA, Ogun AO, Atalabi OM. Low field MR imaging of sellar and parasellar lesions: experience in a developing country hospital. Eur J Radiol. 2012;81(2):e139–46. Epub 2011/02/09. pmid:21300499.

23. Shen D, Wu G, Suk HI. Deep Learning in Medical Image Analysis. Annu Rev Biomed Eng. 2017;19:221–48. Epub 2017/03/17. pmid:28301734; PubMed Central PMCID: PMC5479722.

24. Ajagbe SA, Adigun MO. Deep learning techniques for detection and prediction of pandemic diseases: a systematic literature review. Multimed Tools Appl. 2023:1–35. Epub 2023/06/26. pmid:37362693; PubMed Central PMCID: PMC10226029.

25. Tan MX, Le QV. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Pr Mach Learn Res. 2019;97. WOS:000684034306026.

26. Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely Connected Convolutional Networks. Proc Cvpr Ieee. 2017:2261–9. WOS:000418371402035.

27. Wang XL, Girshick R, Gupta A, He KM. Non-local Neural Networks. 2018 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (Cvpr). 2018:7794–803. WOS:000457843607098.

28. Fluss R, Faraggi D, Reiser B. Estimation of the Youden index and its associated cutoff point. Biometrical J. 2005;47(4):458–72. WOS:000231492300005. pmid:16161804

29. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vision. 2020;128(2):336–59. WOS:000511807100003.

30. Cheng CT, Kuo LW, Ouyang CH, Hsu CP, Lin WC, Fu CY, et al. Development and evaluation of a deep learning-based model for simultaneous detection and localization of rib and clavicle fractures in trauma patients’ chest radiographs. Trauma Surg Acute Care Open. 2024;9(1):e001300. Epub 2024/04/22. pmid:38646620; PubMed Central PMCID: PMC11029226.

31. Ho CS, Chen YP, Fan TY, Kuo CF, Yen TY, Liu YC, et al. Application of deep learning neural network in predicting bone mineral density from plain X-ray radiography. Arch Osteoporos. 2021;16(1):153. Epub 2021/10/10. pmid:34626252.

32. Hsieh CI, Zheng K, Lin C, Mei L, Lu L, Li W, et al. Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning. Nat Commun. 2021;12(1):5472. Epub 2021/09/18. pmid:34531406; PubMed Central PMCID: PMC8446034.

33. Ito S, Nakashima H, Segi N, Ouchida J, Oda M, Yamauchi I, et al. Automated Detection of the Thoracic Ossification of the Posterior Longitudinal Ligament Using Deep Learning and Plain Radiographs. Biomed Res Int. 2023;2023:8495937. Epub 2023/12/06. pmid:38054045; PubMed Central PMCID: PMC10695689 publication of this paper.

34. Jang M, Kim M, Bae SJ, Lee SH, Koh JM, Kim N. Opportunistic Osteoporosis Screening Using Chest Radiographs With Deep Learning: Development and External Validation With a Cohort Dataset. J Bone Miner Res. 2022;37(2):369–77. Epub 2021/11/24. pmid:34812546.

35. Jang R, Choi JH, Kim N, Chang JS, Yoon PW, Kim CH. Prediction of osteoporosis from simple hip radiography using deep learning algorithm. Sci Rep. 2021;11(1):19997. Epub 2021/10/09. pmid:34620976; PubMed Central PMCID: PMC8497544.

36. Kim JH, Mo YC, Choi SM, Hyun Y, Lee JW. Detecting Ankle Fractures in Plain Radiographs Using Deep Learning with Accurately Labeled Datasets Aided by Computed Tomography: A Retrospective Observational Study. Appl Sci-Basel. 2021;11(19). ARTN 8791 WOS:000709119000001.

37. Magneli M, Borjali A, Takahashi E, Axenhus M, Malchau H, Moratoglu OK, et al. Application of deep learning for automated diagnosis and classification of hip dysplasia on plain radiographs. BMC Musculoskelet Disord. 2024;25(1):117. Epub 2024/02/10. pmid:38336666; PubMed Central PMCID: PMC10854089.

38. Mao L, Xia Z, Pan L, Chen J, Liu X, Li Z, et al. Deep learning for screening primary osteopenia and osteoporosis using spine radiographs and patient clinical covariates in a Chinese population. Front Endocrinol (Lausanne). 2022;13:971877. Epub 2022/10/01. pmid:36176468; PubMed Central PMCID: PMC9513384.

39. Sato Y, Yamamoto N, Inagaki N, Iesaki Y, Asamoto T, Suzuki T, et al. Deep Learning for Bone Mineral Density and T-Score Prediction from Chest X-rays: A Multicenter Study. Biomedicines. 2022;10(9). Epub 2022/09/24. pmid:36140424; PubMed Central PMCID: PMC9496220.

40. Yoon AP, Lee YL, Kane RL, Kuo CF, Lin C, Chung KC. Development and Validation of a Deep Learning Model Using Convolutional Neural Networks to Identify Scaphoid Fractures in Radiographs. JAMA Netw Open. 2021;4(5):e216096. Epub 2021/05/07. pmid:33956133; PubMed Central PMCID: PMC8103226

41. Lee KS, Jung SK, Ryu JJ, Shin SW, Choi J. Evaluation of Transfer Learning with Deep Convolutional Neural Networks for Screening Osteoporosis in Dental Panoramic Radiographs. J Clin Med. 2020;9(2). Epub 2020/02/07. pmid:32024114; PubMed Central PMCID: PMC7074309.

42. Gao Y, Soh NYT, Liu N, Lim G, Ting D, Cheng LT, et al. Application of a deep learning algorithm in the detection of hip fractures. iScience. 2023;26(8):107350. Epub 2023/08/09. pmid:37554447; PubMed Central PMCID: PMC10404720.

43. Hong N, Cho SW, Shin S, Lee S, Jang SA, Roh S, et al. Deep-Learning-Based Detection of Vertebral Fracture and Osteoporosis Using Lateral Spine X-Ray Radiography. J Bone Miner Res. 2023;38(6):887–95. Epub 2023/04/12. pmid:37038364.

44. Guo L, Zhou C, Xu J, Huang C, Yu Y, Lu G. Deep Learning for Chest X-ray Diagnosis: Competition Between Radiologists with or Without Artificial Intelligence Assistance. J Imaging Inform Med. 2024. Epub 2024/02/09. pmid:38332402.

45. Chae HD, Hong SH, Yeoh HJ, Kang YR, Lee SM, Kim M, et al. Improved diagnostic performance of plain radiography for cervical ossification of the posterior longitudinal ligament using deep learning. PLoS One. 2022;17(4):e0267643. Epub 2022/04/28. pmid:35476649; PubMed Central PMCID: PMC9045646

46. Park SH, Han K. Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction. Radiology. 2018;286(3):800–9. Epub 2018/01/09. pmid:29309734.

47. Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019;293(3):573–80. Epub 2019/10/23. pmid:31638490.

48. Niehues SM, Adams LC, Gaudin RA, Erxleben C, Keller S, Makowski MR, et al. Deep-Learning-Based Diagnosis of Bedside Chest X-ray in Intensive Care and Emergency Medicine. Invest Radiol. 2021;56(8):525–34. Epub 2021/04/08. pmid:33826549.

49. Cawley GC, Talbot NLC. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. J Mach Learn Res. 2010;11:2079–107. WOS:000282523000006.