Detecting COVID-19 from digitized ECG printouts using 1D convolutional neural networks

Abstract

The COVID-19 pandemic has exposed the vulnerability of healthcare services worldwide, raising the need to develop novel tools to provide rapid and cost-effective screening and diagnosis. Clinical reports indicated that COVID-19 infection may cause cardiac injury, and electrocardiograms (ECG) may serve as a diagnostic biomarker for COVID-19. This study aims to utilize ECG signals to detect COVID-19 automatically. We propose a novel method to extract ECG signals from ECG paper records, which are then fed into one-dimensional convolution neural network (1D-CNN) to learn and diagnose the disease. To evaluate the quality of digitized signals, R peaks in the paper-based ECG images are labeled. Afterward, RR intervals calculated from each image are compared to RR intervals of the corresponding digitized signal. Experiments on the COVID-19 ECG images dataset demonstrate that the proposed digitization method is able to capture correctly the original signals, with a mean absolute error of 28.11 ms. The 1D-CNN model (SEResNet18), which is trained on the digitized ECG signals, allows to identify between individuals with COVID-19 and other subjects accurately, with classification accuracies of 98.42% and 98.50% for classifying COVID-19 vs. Normal and COVID-19 vs. other classes, respectively. Furthermore, the proposed method also achieves a high-level of performance for the multi-classification task. Our findings indicate that a deep learning system trained on digitized ECG signals can serve as a potential tool for diagnosing COVID-19.

Introduction

Since the initial infection was reported in Wuhan, China at the end of 2019, COVID-19 has become a global pandemic, infecting more than 500 million individuals and causing more than 6 million deaths (until the end of April, 2022) [1]. COVID-19 is becoming an endemic disease that will always be with us [1, 2]. COVID-19 disease, when it becomes an endemic disease, may still pose problems for many people, especially vulnerable groups like the elderly and those with pre-existing medical conditions, owing to its rapid contagious [3]. As a result, detecting COVID-19 and developing an effective treatment strategy for patients require a fast and accurate diagnosis.

To diagnose and screen COVID-19 infections, previous studies used a combination of genome sequencing, nucleic acid molecular testing, clustered regularly interspaced short palindromic repeats editing techniques, antigen/antibody detection, and computed tomography imaging. Because of their excellent sensitivity and specificity, polymerase chain reaction (PCR)-based assays are considered the gold standard for virus identification, and Reverse Transcription PCR (RT-PCR) is the most commonly used diagnostic method [4]. However, PCR-based detection has some drawbacks, including the need for a high-purity sample, expensive laboratory equipment, expert training, and a long response time [5]. Other molecular diagnostic techniques, such as clustered regularly interspaced short palindromic repeats (CRISPR) and gene sequencing, suffer from the same limitations. Chest CT detection is also a significant method to diagnose patients suspected of having a COVID-19 infection. Most COVID-19 patients have characteristic radiologic findings in chest CT scans, such as multifocal plaque consolidation and ground glass opacity, according to several studies [6, 7]. However, Bernheim et al. [8] discovered that CT scans are unable to diagnose 56% of patients in the early stages of symptom development. Some studies have also reported that some patients who tested positive for RT-PCR show initially normal X-rays or chest CT scans due to the lung may not be the target organ of the SARS-CoV-2 infection [8, 9].

Previous studies have shown that COVID-19 may be responsible for various cardiovascular complications, which has been the impetus for studies using ECG signals to diagnose COVID-19 [10]. The ECG is an excellent indication of pathological abnormalities in the cardiovascular system. This research is aimed at the possibility of diagnosing COVID-19 using paper-based ECG data and Deep Learning (DL) techniques, taking advantage of the most promising ECG qualities including accessibility, dependability, cheap cost, real-time monitoring, and harmlessness. The technique presented in this research is expected to be an accurate and effective alternative to fill the COVID-19 detection gap.

Several previous studies have also been conducted to evaluate the feasibility of using ECG signals as a diagnostic method for COVID-19. Khan et al. [11] built a dataset of 1937 ECG images of Cardiac and COVID-19 patients, which covered 5 classes: COVID-19, Abnormal Heartbeat, Myocardial Infarction (MI), Previous History of MI (RMI), and Normal Person. This dataset facilitates subsequent studies to investigate the ability to detect COVID-19 based on ECG signals. Ozdemir et al. proposed a method called hexaxial feature mapping to represent 12-lead ECG to 2D colorful images [12]. Specifically, Gray-Level Co-Occurrence Matrix (GLCM) method was used to extract features and generate hexaxial mapping images. These generated images are then fed into a new CNN architecture to diagnose COVID-19. Irmak in [13] introduced a novel deep CNN model to demonstrate the feasibility of using ECG signals to diagnose COVID-19 and yielded an overall classification accuracy of 98.57%, 93.20%, 96.74% for COVID-19 vs. Normal, COVID-19 vs. Abnormal Heartbeats and COVID-19 vs. MI binary classification tasks, respectively. Attallah in [14] proposed a pipeline called ECG-BiCoNet, which makes use of five different deep learning models with different structural designs. Nevertheless, most of the abovementioned studies utilized ECG images as direct input into machine learning algorithms. These approaches do not exploit the advantages of 1D signals such as high temporal resolution, saving computational costs and storage space. Furthermore, feeding images directly into a deep learning model makes the model sensitive to the type of ECG report form (various ECG printout templates available depending on the type of ECG machine and ECG file reading program) and image resolution.

In this study, a novel image-to-signal conversion algorithm is proposed, which extracts 1D ECG signal from ECG printout images. This algorithm is then tested in COVID-19 classification application to evaluate its efficiency and accuracy. To the best of our knowledge, this is the first time the ECG signal extracted from ECG printouts was used to detect COVID-19. This method allows us to normalize data from many sources, removes the model’s reliance on image quality, and enhances computing performance by using 1D convolution.

In addition, the image-to-signal conversion also benefits ECG data storage. The majority of ECG records were collected in paper-based form over the years [15]. However, it has the drawbacks of being damaged over time and being difficult to archive data in order to establish an electronic medical record database. A good storage effort may be a viable answer to these issues. The digitization procedure is one of the most efficient ways to save ECG data. The ECG printouts can be digitized to reduce storage space and to convey various vital morphological and functional information for clinical and diagnostic scientific study. This digital data can be then analyzed with a machine learning algorithm to detect heart abnormalities quickly. Previous studies have developed algorithms to convert paper data into 1D signals [15–18]. These algorithms, however, were either too complicated or inadequately conceived for our experimental data. In this study, we developed a simple, rapid, and effective method, based on a pixel tracking algorithm for converting an ECG image to signals. After being extracted, the 12 signal segments from 12 leads will be concatenated and fed into the SEResNet18 model [19], which will classify ECG signals of individuals with COVID-19 and other CVDs. Several scenarios were undertaken to demonstrate the capability of identifying COVID-19 based on ECG signals. In addition, we evaluated our approach by comparing it against previous studies. The findings demonstrated that our approach performs competitively and ECG could be a useful diagnostic tool for COVID-19.

In summary, our main contributions are threefold:

* First, we develop a high-precision COVID-19 vs. other CVDs vs. healthy individuals classification system that takes the ECG image as direct input, then the ECG signal is extracted and fed into a 1D-CNN.

* Second, a novel algorithm for converting ECG paper records to digital signals was developed, which took ECG images as input and converted them into electronic ECG recording forms.

* Third, extracted 1D ECG signals are used as input for the machine learning model in the COVID-19 diagnosis task for the first time, to the best of our knowledge. Extracting ECG signals from images provides the following advantages: (i) computational cost savings due to the use of 1D-CNN; (ii) application in converting ECG paper into soft files for archiving; (iii) hand-crafted feature extraction methods can be applied to ECG signals for further analysis.

* Last, by simulating the data settings of prior research and conducting extensive experiments, we demonstrate that models using ECG signal as input outperform the current state-of-the-art, which does the classification directly on the ECG image. Our codes for signal-to-image conversion and modeling are made available at https://github.com/vinuni/DiECG-Cov.

This paper is structured as following. In Methods section, the image-to-signal conversion method is explained. In section Experiments, the used dataset is presented first, followed by a description of the experimental settings and evaluation criteria that we employ to assess model performance. In Results section, the experimental results are presented and analyzed. The main findings and limitations of the study are summarized and some useful suggestions are provided in Discussions. Finally, the entire study are summarized in the Conclusion section.

Methods

Our proposed approach for identifying COVID-19 from ECG images involves two steps: converting images into 1D signals and feeding these signals into a CNN model that distinguishes between COVID-19, other CVDs, and normal ECG. The schematic diagram for the proposed system is illustrated in Fig 1.

[Figure omitted. See PDF.]

The process consists of 2 steps: converting ECG images to signals and classifying the resulting ECG signals.

Image-to-signal conversion algorithm

ECG data is often stored in paper form, which consumes storage space, and as a machine learning model input, 2D image data requires more computational cost than time series data (1D signal). For those reasons, we extract the 1D ECG signal from the scanned ECG paper and use that signal as input to the classifier. The procedure for extracting the ECG signal consists of 5 steps as depicted in Fig 2. First, the scanned image is converted to a binary image. From the binary image, the isoelectric lines of the ECG signal are found, and from the position of these lines, the image of each lead is separated. Since the binary image contains noise and artifacts, a noise removal algorithm is required. Finally, from the signal image of each lead, the ECG signal is extracted and concatenated to form a time series ECG signal.

[Figure omitted. See PDF.]

The process of converting image to signal consists of 5 steps: image binarization, isoelectric line detection, lead separation, noise removal and signal extraction.

Image binarization.

ECG scans are converted from colored to binary by Otsu’s method [20] to facilitate further processing.

Isoelectric lines detection.

It is critical to find the position of the isoelectric lines in the ECG recording, which can deduce the area of each lead on the image. Due to the characteristic of the isoelectric lines, when projecting the ECG image in the vertical direction, the data points (black points as shown in Fig 3a) will be concentrated with high intensity at their position. As shown in Fig 3b, the vertical histogram of the image has peaks at the positions of the isoelectric lines. Thus, finding the position of the isoelectric lines can be done by finding the peaks of the histogram. As shown in Fig 3d, the isoelectric lines are detected (the black horizontal lines) and shared by four signals corresponding to four ECG leads each.

[Figure omitted. See PDF.]

(b) and (c) are the vertical and horizontal histograms of image (a), respectively. (b) has peaks corresponding to positions of the horizontal lines at the top of the paper and the isoelectric lines of the ECG signal. (c) has three peaks corresponding to lines separating the leads. In (d), the black line depicts the positions of all isoelectric lines.

Lead seperation.

From the position of the detected isoelectric lines, we separate the image of each lead by zoning vertically from the upper isoelectric line minus L pixels to the lower isoelectric line plus L pixels (L depends on the characteristics of ECG images, specifically the thickness of ECG lines). The boundary between leads sharing an isoelectric line is detected based on the horizontal histogram of the image. Fig 3c shows the horizontal histogram of the image in Fig 3a. Due to the presence of characters referring to the name of leads in the ECG paper, the segment of image containing those characters is cropped. Once the vertical and horizontal boundaries of the leads found, we crop the image of each lead for further processing. Fig 4a and 4b are instances of an ECG printout and image of lead V2 cropped from it.

[Figure omitted. See PDF.]

(a) is a binary image of the ECG signal, (b) is an image of the lead V2 signal cropped from (a), (c) is the image of signal after removing noise and artifacts. In (c), there are small noise spots around the signal. These spots were filtered out in (d), leaving only the ECG signal image.

Noise and artifacts removal.

The binary image is embedded in noise and artifacts from the grid and adjacent signals (as shown in Fig 4b). We built a simple yet effective algorithm that separates the image of the ECG signal from noise and artifacts. The first step involves initializing a point that belongs to the isoelectric line and has a pixel value of 0. This point has coordinate (n, b) (b is the y-intercept of the isoeletric line). Pixels satisfying the following three conditions i) located on the same column as the starting point; ii) having a value of 0 and iii) less than N pixels away from the starting point; are retained. Pixels with a value of 0 in the n^th column of the image and are less than N pixels away from the isoelectric line are considered pixels of the ECG signal in that column. Similar to L, the value of N depends on the characteristics of ECG images, specifically the thickness of ECG lines. The maximum coordinate (c_max) and the minimum coordinate (c_min) of pixels 0 in that column are calculated. In the next column, pixels with coordinates from c_min − N to c_max + N are considered; and among them, pixels with value 0 are retained. c_max and c_min values of this column are again calculated, and the search for ECG pixels in the next column is carried out.

From the starting point, we track in two directions: to the left edge of the image and to the right edge of the image. At the end of the process, only ECG pixels are retained. The noise removal algorithm is detailed in Algorithm 1 and Fig 5. Fig 4c is the result of the ECG tracking algorithm applied to Fig 4b with N = 10. This image contains no noise and artifacts, and is the input for the ECG signal extraction.

[Figure omitted. See PDF.]

(a) Signal image with noise and artifacts; (b) ECG pixel tracking algorithm with N = 6; (c) Image of signal without noise and artifacts.

Algorithm 1 Noise and artifacts removal algorithm

Ensure: Separate the image of ECG signal from noise and artifacts in img and save it to img_copy

 W = img.shape[1]

img_copy = 255*np.ones_like(img)

Initialize n      ▹Random point for ECG image tracking

col = img[b − N: b + N, n]      ▹b: baseline position

col = np.where(col)

img_copy[col + c_min, n] = 0

c_max = max(col)

c_min = min(col)

for i in range(n+1, W) do

col = img[c_min − N: c_max + N, n]

col = np.where(col==0)

img_copy[col + c_min, n] = 0

c_max = max(col)

c_min = min(col)

end for

for i in range(n-1, -1, -1) do

col = img[c_min − N: c_max + N, n]

col = np.where(col==0)

img_copy[col + c_min, n] = 0

c_max = max(col)

c_min = min(col)

end for

Signal extraction.

The image containing only the ECG pixels is the subject of the image-to-signal conversion.

Extraction of 1D ECG signals from images is conducted by scanning the image horizontally. The amplitude of the ECG signal at a specific column is equal to the average of the pixel 0 positions of that column minus the isoelectric line position. For columns missing pixel 0, the signal amplitude is taken as the value of the signal point immediately preceding it. Once the scanning is done, the length of the resulting signal is equal to the number of image’s columns.

Each ECG printout has reference pulses in the form of a square pulse (as shown in Fig 6). In Fig 6a, 6b and 6d, the reference square pulses have a width corresponding to 0.1 s and a height corresponding to 0.5 mV. In Fig 6c the reference pulse is 0.2 s and 0.5 mV for width and height, respectively. Determining the width and height of these square pulses makes the image-to-signal conversion consistent in time and amplitude. In addition, the scale conversion also helps to display the signal accurately in terms of amplitude and time. In this study, we convert all the records to the same sampling frequency of 200 Hz. Fig 7a depicts an ECG signal image, and Fig 7b depicts the signal extracted from Fig 7a.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Signal quality assessment.

Once the 1D ECG data have been obtained, we assess its quality to guarantee that the image-to-signal conversion algorithm functions properly and that the resulting signal can be used as input for the machine learning model. After randomly selecting 50 samples from each category, we hand-labeled the R peaks of the lead II signal in these images and regarded the resulting positions as ground truths to evaluate our algorithm against. The selected samples are also fed into our proposed image-to-signal conversion, and the generated signals were exposed to NeuroKit2’s [21] R peak detection algorithm to obtain the positions of R peaks. Finally, we evaluate our algorithm by comparing its output against the ground truths, specifically by quantifying the difference in the RR intervals (the time elapsed between two successive R-waves) using the mean absolute error (MAE) metric.

Experiments

Experimental setups

Dataset.

COVID-19 ECG images dataset—a publicly available dataset of ECG images of cardiac and COVID-19 patients was used in this study. Khan et al. from the University of Management and Technology shared the dataset to Mendeley Data [11]. The dataset contains photos of 1937 paper-based ECG reports. ECG records for 250 COVID-19 patients, 77 MI patients, 548 individuals with irregular heartbeats (recovered from COVID-19 or MI), 203 patients with MI history (RMI), and 859 people without any cardiac abnormalities were analyzed by experts. The provided dataset is the first ECG dataset for COVID-19 disease that has been shared and is completely anonymized. We did not seek approval from an ethics committee and participants’ consent for this study because (i) the previous study during which the dataset was collected already obtained necessary approval, (ii) the dataset is completely anonymized and contains no personal information which can be used to identify patients, and (iii) no further data are collected during this study.

The ECG images in the dataset are from a 12-lead system (I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, and V6) at a sampling rate of 500 Hz. ECG data were gathered using an EDAN SE-3 series 3-channel electrocardiograph, and some of the signals were applied with a 0.67–25 Hz bandpass filter, while others were processed with a 0.5–100 Hz bandpass filter and a 50 Hz notch filter, according to the paper-based ECG reports.

The dataset has some limitations. In particular, the dataset is considerably imbalanced, with 859 samples of normal ECG (the class with the most samples) and 77 samples of MI ECG (the class with the fewest samples). Besides, the dataset consists of four different types of reports, which are different in (i) image size and resolution; (ii) ECG report template; (iii) grid background, and (iv) reference pulse. Four types of ECG printout templates are shown in Fig 6. Using the signal extracted from the ECG image is not only reduces the computational cost but also removes disparities in ECG printout templates, making the prediction more objective.

Experiments.

To evaluate the capability of diagnosing COVID-19 based on ECG data, three classification tasks are covered: (i) COVID-19 and normal ECG; (ii) COVID-19 and others; (iii) COVID-19, normal and others. As the number of samples in MI and RMI classes is relatively small, the merging of MI, RMI, and abnormal into a class could alleviate the effects of data imbalance. The number of images used in each problem is detailed in Table 3.

To evaluate the effectiveness of this method, we also compare its performance against the results of previous studies performed on the COVID-19 ECG images dataset. We simulate the data splitting of previous studies and execute similar classification tasks [12–14].

Implementation details.

In this study, all experiments were conducted using Intel core i9-10900X CPU @ 3.70GHz, RAM 31 GB hardware and NVIDIA GeForce RTX 3080 Ti GPU. We trained 110 epochs for each model, with a mini-batch size of 16, and evaluated model performance after each epoch. Checkpoint with the highest F1-score against the test set was considered the best model for each training procedure. As the backbone of the COVID-19 vs. Normal classifier, we apply multiple architectures, then select the model with the best performance to train other classification tasks. The utilized architectures are SEResNet18 [19], ResNet18 [22], ResCNN [23], Sequence Stroke Net [24], LSTM Fully Convolutional Network (LSTM FCN) [25], GRU Fully Convolutional Network (GRU FCN) [26], InceptionTime [27], XceptionTime [28] and Time Series Transformer (TST) [29]. We adopted Adam optimizer [30] (β1 = 0.9, β2 = 0.999 and learning rate = 1e-3), cooperating with Cosine annealing learning rate [31] with the maximum number of iteration set to 25 and the learning rate stop decaying at 100^th epoch. As a criterion for training, the label smoothing cross-entropy loss function with a smoothing factor of 0.1 was used. We selected the most suitable hyper-parameters using simple grid search strategy and experimental results.

To evaluate the performance of the models, a nested cross-validation method was employed for each trial. The dataset was first divided into 5 folds, with 1 set used for testing and the rest used for training and validation. In each iteration, we used 4-fold cross-validation to further split the data between training and validation. Classification performance was evaluated for each fold, and then the average classification performance of each model was calculated. To ensure that the data distributions in training, validation and test sets were the same as in the original dataset, stratified splitting was adopted for both cross-validation steps.

We used all of the samples in the dataset that are relevant to the considered classes for each classification task. The number of images used for each task is listed in Table 3.

Evaluation metrics

To evaluate the quality of the ECG signals extracted from the images, we used mean absolute error (MAE) as the evaluation metric. The mean of the RR intervals in the ground truth and the resulting signal is calculated, then MAE value is calculated. Eq 1 gives the formula for computing the MAE.(1)Where N is total number of ECG images; mRR_i is the mean of RR intervals calculated from i^th image; is the mean of RR intervals of the corresponding signal.

In terms of the CNN model, model performance is assessed utilizing well-known evaluation metrics such as Accuracy, F1-score, Specificity, Sensitivity, and AUC (Area Under the Curve of the Receiver Operating Characteristic).

Results

Experimental results and implications

ECG signal quality assessment.

Extracted signal quality assessment was evaluated on a certain number of samples in the dataset. The number of samples in each class, as well as the MAE of the RR interval, are listed in Table 1. Despite starting with 50 samples in each class, the RR interval cannot be calculated for samples with only one R peak and thus the reported sample counts were less than 50.

[Figure omitted. See PDF.]

Evaluation was performed on 226 random samples belonging to 5 classes, with mean absolute error of RR interval of 28.11 ms.

The MAE values range from 11.09 ms to 63.75 ms, as shown in Table 1. Normal class has the lowest MAE (11.09 ms) due to the regular and stable ECG waveform. Class MI has the highest MAE of 63.75 ms, due to the unstable of ECG form of patients with MI disease. The MAE values for COVID-19, Abnormal, and RMI classes are 16.07 ms, 25.73 ms, and 27.57 ms, respectively. Overall, the mean MAE of the tested samples is 28.11 ms, which is 3.7% of the RR interval for a person with an 80 bpm heart rate (equivalent to a mean RR interval of 750 ms).

ECG signal classification.

Table 2 compares the COVID-19 vs. normal classification results between different models. With an f1-score of 0.9842 and an AUC of 0.9735, the SEResNet18 model outperforms the others. The XceptionTime model has the best classification accuracy (98.58%), while the ResNet18 model has the fastest inference time (18.34 ms per sample). According to the obtained results, we choose the SEResNet18 with some modifications as the backbone for other classification tasks. Convolution (Conv) layers are modified with a larger kernel size (kernel_size = 7) and a stride parameter of 1 in order to capture longer patterns in ECG data. For ECG data in specific and time-series data in general, this technique has been recommended to be more effective [27, 32].

[Figure omitted. See PDF.]

Table 3 covers all of the experimental results for each classification task. Patients positive for COVID-19 were distinguished from normal patients and patients negative for COVID-19 with high accuracy of 98.42% and 98.50%, respectively. The classification accuracy and AUC were 83.17% and 0.8748 in the case of classifying patients with COVID-19, Normal and remaining classes. The majority of misclassified samples include COVID-19 ECGs with identical characteristics to those of normal ECGs, indicating that the ECG waveform of the patients has not changed.

[Figure omitted. See PDF.]

Three classification tasks were completed in order to determine whether ECG signals might be used to diagnose COVID-19.

It is also crucial to discover whether the network is making the decision based on the relevant segment of the ECG signal or somewhere else. For different classes of ECG data, heat maps based on the Grad-CAM technique [33] were constructed. Fig 8 shows heat maps generated with the best-performing models and Fig 9 depicts t-SNE plots [34] to visually highlight the capacity to distinguish between COVID-19 and other conditions.

[Figure omitted. See PDF.]

(a, b) Grad-CAM visualization of ECG image with COVID-19. (c, d) Grad-CAM visualization of ECG image with MI. Signal segments significant to the classifier are depicted in darker color.

[Figure omitted. See PDF.]

Comparison with state-of-the-arts

Table 4 summarizes the experimental results and comparisons against previous studies using the same ECG dataset. The proposed method has an accuracy of 0.6% higher than that of previous studies in the COVID-19 vs. Normal classification problem and up to 6.72% and 6.43% higher in the other classification problems in terms of to accuracy and F1-score, respectively.

[Figure omitted. See PDF.]

To evaluate the efficacy of our method in comparison to others, data was split in a way that simulates related studies. Dashes (-) represent values not reported in the original paper. Our results are in bold.

Discussions

In addition to having hazardous effects on human health, the COVID-19 pandemic has placed a significant strain on the healthcare system. To alleviate this problem, a rapid and straightforward method for COVID-19 diagnosis is required. The use of a deep learning methodology using 1D signals converted from ECG images to detect COVID-19 shows potential as a novel diagnostic procedure.

When compared against previous works on the same dataset, our method performed competitively. The following reasons may contribute to the proposed method’s better performance: (i) white pixels contain no information, although they account for a major portion of the image; (ii) the amount of data required to train a 2D CNN is greater than that of 1D CNN [35] while the experimental dataset contains a modest number of samples (250 images for COVID-19 class), and (iii) it is thought that for ECG signal processing, a CNN with a larger kernel size and a lower stride value (compared to values often used in image processing) would be more effective in capturing the temporal correlation of adjacent signal points [32].

This study contributes to the literature in numerous ways in this regard. These contributions can be summarized as follows:

* Our work, together with previous research, demonstrated the capability to automatically distinguish COVID-19 patients by ECG signals with high sensitivity, cost-effectiveness, and non-invasiveness.

* To classify paper-based ECG data, a new and effective technique was developed, in which signals from a standard 12-lead ECG are extracted from the image and utilized as input to a 1D CNN model. This is the first time the ECG signal extracted from ECG printouts was used to detect COVID-19, to the best of our knowledge.

* The following benefits are provided by the image-to-signal conversion: (i) Less computational costs;(ii) More robust classification, regardless of the ECG report template and the resolution of ECG images; (iii) Less storage costs by using 1D data instead of 2D;

* Our experiments confirmed the differences in ECG data of patients with COVID-19 and persons without any cardiac findings, as well as patients with other CVDs. The findings of the experiments might indicate the presence of COVID-19-induced cardiovascular changes.

There are some drawbacks to our proposed method. Firstly, similar to all current image-signal conversion algorithms, our proposed algorithm is sensitive to the shape of the ECG signal. The algorithm cannot differentiate the ECG signal in different leads if their amplitude is too high and combine together (signals with excessive amplitude are caused by noise or abnormal electrical activity of the heart due to CVDs). Furthermore, although the proposed method has been tested with variety of experimental scenarios, it still has to be evaluated with different datasets. In future works, we expect to integrate a COVID-19 diagnosis module into a mobile application, which will help medical practitioners by providing a quick and accurate approach to identifying COVID-19. Finally, although our method only produced the final diagnosis, an additional output that highlights the anomalies in the ECG [36] of COVID-19 patients can also be helpful to cardiologists. Other potential approaches will be considered include training deep learning models with hierarchical disease dependencies [37] or demographic data [38].

Conclusion

In this work, a novel and effective method for automatically detecting COVID-19 utilizing ECG printouts is proposed. The proposed approach is based on an image-to-signal conversion algorithm, which extracts 1D ECG signal from scanned ECG printouts and feeds it into the 1D SEResNet18 model. We have run experiments on various scenarios and analyzed the findings. We also simulated the data set-up of prior research to develop classification models, then compared the performance of those models with their findings to evaluate the effectiveness of utilizing the extracted signal. The findings suggest that the ECG signal may be used to distinguish COVID-19 patients from other groups, and the proposed technique outperforms previous studies using the same dataset. Additionally, the capacity of the proposed method to distinguish COVID-19 ECG can serve as evidence that there are changes in ECG signals of COVID-19 patients. The proposed ECG-based COVID-19 diagnose is easily extended to real-time systems and executed on mobile applications in our future works. In addition, we also intend to further interpret the model’s decision and discover how COVID-19 patients’ ECG changes compared to healthy people. As a result, it may benefit healthcare providers by offering a quick and accurate way to detect COVID-19, as well as reducing hospital costs by avoiding unnecessary hospital visits.

Citation: Nguyen T, Pham HH, Le KH, Nguyen A-T, Thanh T, Do C (2022) Detecting COVID-19 from digitized ECG printouts using 1D convolutional neural networks. PLoS ONE 17(11): e0277081. https://doi.org/10.1371/journal.pone.0277081

About the Authors:

Thao Nguyen

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Visualization, Writing – original draft

Affiliation: College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam

https://orcid.org/0000-0003-0811-8620

Hieu H. Pham

Roles: Conceptualization, Writing – review & editing

Affiliations College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam, VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam

https://orcid.org/0000-0003-4851-2518

Khiem H. Le

Roles: Writing – review & editing

Affiliation: College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam

Anh-Tu Nguyen

Roles: Conceptualization, Writing – review & editing

Affiliation: College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam

Tien Thanh

Roles: Conceptualization, Writing – review & editing

Affiliation: Pulmonology, Vinmec International Hospital, Hanoi, Vietnam

https://orcid.org/0000-0001-8438-1995

Cuong Do

Roles: Conceptualization, Writing – review & editing

E-mail: [email protected]

Affiliation: College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam

References

1. The Lancet Infectious Diseases. Transitioning to endemicity with covid-19 research. In The Lancet Infectious Diseases, 22(3):297, 2022. pmid:35151377

2. Antia Rustom and Halloran M Elizabeth. Transition to endemicity: Understanding covid-19. In Immunity, 54(10):2172–2176, 2021. pmid:34626549

3. Mao Ren, Liang Jie, Shen Jun, Ghosh Subrata, Zhu Liang-Ru, et al. Implications of covid-19 for patients with pre-existing digestive diseases. In The lancet Gastroenterology & hepatology, 5(5):425–427, 2020. pmid:32171057

4. Shen Minzhe, Zhou Ying, Ye Jiawei, Al-Maskri Abdu Ahmed Abdullah, et al. Recent advances and perspectives of nucleic acid detection for coronavirus. In Journal of Pharmaceutical Analysis, 10(2):97–101, 2020. pmid:32292623

5. Corman Victor M, Landt Olfert, Kaiser Marco, Molenkamp Richard, Meijer Adam, et al. Detection of 2019 novel coronavirus (2019-ncov) by real-time rt-pcr. In Eurosurveillance, 25(3):2000045, 2020.

6. Chung Michael, Bernheim Adam, Mei Xueyan, Zhang Ning, Huang Mingqian, et al. CT imaging features of 2019 novel coronavirus (2019-nCoV). In Radiology, 2020.

7. Li Xiaoming, Zeng Wenbing, Li Xiang, Chen Haonan, Shi Linping, et al. CT imaging changes of corona virus disease 2019(COVID-19): A multi-center study in Southwest China. In Journal of Translational Medicine, 18(1):1–8, 2020. pmid:32252784

8. Bernheim Adam, Mei Xueyan, Huang Mingqian, Yang Yang, Fayad Zahi A et al. Chest CT findings in coronavirus disease 2019 (COVID-19): Relationship to duration of infection. In Radiology, 2020. pmid:32077789

9. Ai Tao, Yang Zhenlu, Hou Hongyan, Zhan Chenao, Chen Chong, et al. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases. In Radiology, 2020. pmid:32101510

10. RS Soumya, Unni T Govindan, and KG Raghu. Impact of COVID-19 on the Cardiovascular System: A Review of Available Reports In Cardiovascular Drugs and Therapy, 35(3):411–425, 2021.

11. Khan Ali Haider, Hussain Muzammil, and Malik Muhammad Kamran. ECG Images dataset of Cardiac and COVID-19 Patients. In Data Brief, 34:106762, 2021. pmid:33521183

12. Ozdemir Mehmet Akif, Ozdemir Gizem Dilara, and Guren Onan. Classification of COVID-19 electrocardiograms by using hexaxial feature mapping and deep learning. In BMC Med Inform Decis Mak, 21(1):1–20, 2021. pmid:34034715

13. Emrah Irmak. COVID-19 disease diagnosis from paper-based ECG trace image data using a novel convolutional neural network model. In Physical and Engineering Sciences in Medicine, pages 1–13, 2022.

14. Attallah Omneya. ECG-BiCoNet: An ECG-based pipeline for COVID-19 diagnosis using Bi-Layers of deep features integration. In Computers in Biology and Medicine, page 105210, 2022. pmid:35026574

15. Sun Xiaohan, Li Qince, Wang Kuanquan, He Runnan, and Zhang Henggui. A Novel Method for ECG Paper Records Digitization. In 2019 Computing in Cardiology (CinC), 2019; ages Page–1. IEEE, 2019.

16. Mishra Siddharth, Khatwani Gaurav, Patil Rupali, Sapariya Darshan, Shah Vruddhi, et al. ECG Paper Record Digitization and Diagnosis Using Deep Learning. In J Med Biol Eng, 127:104077, 2020. 2021.

17. Li Yao, Qu Qixun, Wang Meng, Yu Liheng, Wang Jun, et al. Deep learning for digitizing highly noisy paper-based ECG records. In Computers in Biology and Medicine, 127:104077, 2020. pmid:33171291

18. Tahmida Tabassum and Mohiuddin Ahmad. Numerical Data Extraction from ECG Paper Recording Using Image Processing Technique In 2020 11th International Conference on Electrical and Computer Engineering (ICECE), pages 355–358. IEEE, 2020.

19. Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitation Networks In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.

20. Yousefi Jamileh. Image Binarization using Otsu Thresholding Algorithm. In Ontario, Canada: University of Guelph, 2011.

21. Makowski Dominique, Pham Tam, Lau Zen J, Brammer Jan C, Lespinasse Francois, et al. NeuroKit2: A Python toolbox for neurophysiological signal processing In Behavior Research Methods, 53(4):1689–1696, 2021. pmid:33528817

22. Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN), 1578–1585. IEEE, 2017.

23. Zou Xiaowu, Wang Zidong, Li Qi, and Sheng Weiguo. Integration of residual network and convolutional neural network along with various activation functions and global pooling for time series classification. In Neurocomputing, 367:39–45, 2019.

24. Liu Xin, Hu Baotian, Chen Qingcai, Wu Xiangping, and You Jinghan. Stroke sequence-dependent deep convolutional neural network for online handwritten Chinese character recognition. In IEEE Transactions on Neural Networks and Learning Systems, 31(11):4637–4648, 2020. pmid:31905151

25. Karim Fazle, Majumdar Somshubra, Darabi Houshang, and Chen Shun. LSTM fully convolutional networks for time series classification. In IEEE access, 6:1662–1669, 2017.

26. Nelly Elsayed, Anthony S Maida, and Magdy Bayoumi. Deep gated recurrent and convolutional network hybrid model for univariate time series classification. In arXiv preprint arXiv:1812.07683., 2018

27. Fawaz Hassan Ismail, Lucas Benjamin, Forestier Germain, Pelletier Charlotte, Schmidt Daniel F, et al. Inceptiontime: Finding alexnet for time series classification. In Data Mining and Knowledge Discovery, 34(6):1936–1962, 2020.

28. Rahimian, Elahe and Zabihi, Soheil and Atashzar, Seyed Farokh and Asif, Amir et al. Xceptiontime: A novel deep architecture based on depthwise separable convolutions for hand gesture classification. In arXiv preprint arXiv:1911.03803, 2019

29. George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2114–2124, 2021.

30. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014

31. Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.

32. Cheng Jinyong, Zou Qingxu, and Zhao Yunxiang. Ecg signal classifcation based on deep cnn and bilstm In BMC Medical Informatics and Decision Making, 21(1):1–12, 2021.

33. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,” In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.

34. Van der Maaten Laurens and Hinton Geoffrey. Viualizing data using t-SNE. In Journal of Machine Learning Research. 9(11), 2008.

35. Yunan Wu, Feng Yang, Ying Liu, Xuefan Zha, and Shaofeng Yuan. A comparison of 1-d and 2-d deep convolutional neural networks in ecg classification. A Comparison of 1-D and 2-D Deep Convolutional Neural Networks in ECG Classification In arXiv preprint arXiv:1810.07088, 2018.

36. Le, Khiem H and Pham, Hieu H and Nguyen, Thao BT and Nguyen, Tu A and Thanh, Tien N and Do, Cuong D. LightX3ECG: A Lightweight and eXplainable Deep Learning System for 3-lead Electrocardiogram Classification arXiv preprint arXiv:2207.12381, 62:39–45, 2022.

37. Pham Hieu H and Le Tung T and Tran Dat Q and Ngo Dat T and Nguyen Ha Q. Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels Neurocomputing, 437, 5, 186–194, 2021.

38. Le, Khiem H and Pham, Hieu H and Nguyen, Thao BT and Nguyen, Tu A and Do, Cuong D. Enhancing Deep Learning-based 3-lead ECG Classification with Heartbeat Counting and Demographic Data Integration arXiv preprint arXiv:2208.07088, 2022.