The Fusion of MRI and CT Medical Images Using Variational Mode Decomposition

Abstract

In medical image processing, magnetic resonance imaging (MRI) and computed tomography (CT) modalities are widely used to extract soft and hard tissue information, respectively. However, with the help of a single modality, it is very challenging to extract the required pathological features to identify suspicious tissue details. Several medical image fusion methods have attempted to combine complementary information from MRI and CT to address the issue mentioned earlier over the past few decades. However, existing methods have their advantages and drawbacks. In this work, we propose a new multimodal medical image fusion approach based on variational mode decomposition (VMD) and local energy maxima (LEM). With the help of VMD, we decompose source images into several intrinsic mode functions (IMFs) to effectively extract edge details by avoiding boundary distortions. LEM is employed to carefully combine the IMFs based on the local information, which plays a crucial role in the fused image quality by preserving the appropriate spatial information. The proposed method’s performance is evaluated using various subjective and objective measures. The experimental analysis shows that the proposed method gives promising results compared to other existing and well-received fusion methods.

1. Introduction

Medical image analysis plays a crucial role in clinical assessment. However, the success rate of the diagnosis depends upon the visual quality and the information present in medical images [1]. In real-world medical imaging, denoising [2,3] or texture information processing [4,5] is a necessary preprocessing step to improve the fused image’s visual quality further.

Nowadays, several imaging modalities are available to capture specific medical information of a given organ [6,7,8]. X-ray, MRI, CT, positron emission tomography (PET), and single-photon emission computed tomography (SPECT) of a human brain displayed in Figure 1 are crucial medical imaging modalities among them. For example, the magnetic resonance imaging (MRI) modality captures the anatomical information of the soft tissue. In contrast, computed tomography (CT) significantly provides hard tissue information such as bones structures and tumors [8]. Moreover, for clinical needs, the information provided by a single modality may not be sufficient, especially during the diagnosis of diseases [9]. The image fusion mechanism can effectively address this problem, enhancing the information by combining the complementary details provided by two or more modalities into a single image.

Image fusion can be categorized into spatial and transform domain techniques [10]. In spatial domain methods, the fusion takes place between the pixels of the source images directly. The maximum, minimum, average, weighted average and PCA are examples of the spatial domain fusion methods, which are easy to implement and computationally efficient. Direct pixel-based fusion methods use a weighted pixel of input images to form a fused image [11]. The activity level of the pixels determines these weights. In the literature, various machine learning methods such as neural networks and support vector machines (SVM) are also used to select the pixels with the highest activity [12,13]. In [14], an iterative block-level fusion method is proposed. First, the source images are decomposed into small square blocks, and PCA is computed on those blocks. Next, weights are found using the average of the PCA components. Finally, a maximum average mutual information fusion rule is employed for the final blending of input images. In [15], a pixel-level image fusion method is proposed using PCA. Here, the first PCA components from both the input images are multiplied individually, and those weighted images are added for fusion. However, these methods might exhibit spatial color, information loss, and brightness distortions [16,17].

Image fusion methods based on the transform domain techniques are receiving much consideration [18]. Pyramid [19], wavelet [20], and multi-resolution singular value decomposition (MSVD) are examples of traditional methods [21] in this category. However, transform domain fusion methods have a few drawbacks [18]. For example, most pyramid methods suffer from blocking artifacts and a loss of source information, even producing artifacts around edges [22]. Wavelets suffer shift sensitivity, poor directionality, an absence of phase information, poor performance at edges and texture regions, and produce artifacts around edges because of the shift-variant nature [22]. Despite the reliable quantification results, MSVD fusion methods might result in poor visual quality [23].

To address the issues mentioned above, other transform domain fusion techniques such as À Trous wavelet transform (ATWT), curvelet transform (CVT), and ridgelet transform are suggested in [24]. These methods provide better results concerning the visual aspect, preserving spatial and spectral information. Nevertheless, these techniques suffer from artifacts around the edges in the fused image [25].

In [26], a new pixel-level image fusion approach using convolutional sparsity-based morphological component analysis (CS-MCA) is introduced. This method achieves sparse representation by combining MCA and sparse convolutional representation into the unified optimization method. This approach might suffer from a spatial consistency problem, resulting in the degradation of spatial details [27]. An NSST-based fusion scheme is proposed in [28].This approach used a blend of NSST with weighted local energy (WLE) and a weighted sum of eight- neighborhood-based modified Laplacian (WSEML) to integrate MRI and CT images. However, this method is a non-adaptive approach. A summary of different types of image fusion methods, their advantages and drawbacks are tabulated in Table 1.

An adaptive transform-domain fusion technique might provide a better solution to the challenges mentioned above. In these fusion approaches, the basis function of the transform technique depends on the source image’s characteristics. With the help of adaptive wavelets, the image’s crucial features can be highlighted, which helps in the fusion process. Hence, adaptive wavelets turned out to be a preferable representation compared to standard wavelets. Similar works based on VMD decomposition-based techniques can be found in [35,36]. However, this paper proposes a new adaptive multimodal image fusion strategy based on the combination of variational mode decomposition (VMD) and local energy maxima (LEM) to address the challenges mentioned above. The highlights of the proposed method are as follows:

1. VMD is an adaptive decomposition scheme that decomposes the images as band-limited sub-bands called intrinsic mode functions (IMFs) without introducing boundary distortions and mode-mixing problems. Indeed, the band-limited sub-bands characterize the edge and line features of source images. This decomposition technique can effectively extract the image features from the other transform methods such as wavelet transform (WT), bi-dimensional empirical mode decomposition (BEMD), and empirical wavelet transform (EWT);

2. The LEM fusion rule extracts the local information from decomposed modes corresponding to two source images pixel by pixel using a windowing operation (3 × 3) and then measures the maximum information value. Hence, using the LEM fusion rule, we can preserve the required complementary visual, edge, and texture information in the IMFs;

3. The proposed approach aims to preserve the information and details of both MRI and CT images into the fused image using VMD and LEM. From visual perception and objective assessment of the fusion results, it is evident that our new image fusion method accomplishes good performance over other existing fusion methods.

The remainder of the paper is arranged as follows: The proposed framework and its mathematical representation are presented in Section 2. The detailed analysis of the simulation results and necessary discussion is presented in Section 3. A final note on the proposed method and future directions is given in Section 4.

2. Proposed Methodology

Our proposed work aims to integrate the details of the soft tissue and dense bone structure provided by MRI and CT medical imaging technologies into a unique image. For this, we have proposed a multimodal medical image fusion based on a blend of VMD and LEM, as shown in Figure 2.

The main steps involved in our fusion methodology are:

• VMD-based image decomposition;

• A fusion strategy depending on the LEM;

• Synthesizing the fused image.

A. VMD-Based Image Decomposition

The traditional decomposition approaches, such as wavelets [37,38], BEMD [39], and EWT [40], suffer from various problems such as boundary distortions and mode-mixing. With these issues, we may fail to achieve an appropriate fusion result. To address these problems, we employed VMD [41], a robust adaptive decomposition approach, highlighting meaningful details in the form of sub-images.

The VMD finds applications in image denoising [42] and texture decomposition [43]. VMD is a non-stationary and adaptive signal processing technique. Unlike EMD and its variants, VMD is not a recursive analysis approach, and it decomposes the signal/image into bandlimited sub-bands based on its frequency content. This work uses VMD to obtain distinct and significant IMFs from the source images (MRI and CT). The derived IMFs reduce mode-mixing and boundary distortions, which are the major concerns in the above mentioned transform domain methods. With this VMD decomposition, we can extract prominent edge information. Initially, we decomposed the input images into six IMFs, which are illustrated in Figure 3.

From Figure 3, it can be observed that the first IMF ((b) and (i)) captures prominent information from the source images, whereas the remaining IMFs encompass the line and edge information. We can note from Figure 3 that as the mode number increases, the visual details are not significant.

Mathematical Details of VMD:

The main goal of VMD is to subdivide an input signal () into a specific number of sub-bands (IMFs or Modes) ($b_l$), and each sub-band is bandlimited to specific frequencies in the spectral domain (Fourier domain) by maintaining sparsity. Each of the sub-bands is bandlimited to its center frequencies. VMD involves the following steps to get the bandlimited sub-bands [41]:

1. For each sub-band, its analytical counterpart needs to be computed using Hilbert transform to get the one-sided frequency spectrum;

2. An exponential is used to mix with each mode to shift its frequency spectrum to the baseband;

3. Finally, the bandwidth of the mode estimates using the squared L²-norm of the gradient. The constrained variational problem can be represented as below.

(1)$$\begin{gathered} \underset{\left\{b_l\right\},\left\{{\omega }_l\right\}}{\min }\left\{{\displaystyle \sum _l\Vert {\partial }_t}\left[\left(\delta (t)+\frac{j}{\pi t}\right)\ast b_l(t)\right]{e^{-j{\omega }_{l}t}\Vert }_2^2\right\} \\ \left\{b_l\right\}{ \mathrm{and} \{\mathrm{w}}_l\} \end{gathered}$$

where $l^{th}$ indicates the $l^{th}$ sub-band and its center frequency, respectively. $\delta (t)$ represents the Dirac distribution, $\ast$ is the symbol of the convolution.

The constrained problem in Equation (1) is solved using the quadratic penalty term and Lagrangian multipliers $\lambda$ to make it an unconstrained problem given in Equation (2).

(2)$L(\{b_l\},\{{\omega }_l\},\lambda )=\alpha {{\displaystyle \sum _l\Vert {\partial }_t\left[\left(\delta (t)+\frac{j}{\pi t}\right)\ast b_l(t)\right]e^{-i{\omega }_{l}t}\Vert }}_2^2+{\Vert x(t)-{\displaystyle \sum _l{b}_l(t)}\Vert }_2^2+⟨\lambda ,x(t)-{\displaystyle \sum _l{b}_l(t)}⟩$

where $L$ represents augmented Lagrange matrix function, $\alpha$ is the penalty factor parameter, $\lambda$ indicates the Lagrange multiplier, and $x(t)$ is the input signal.

Now the solution of Equation (1) can be computed as the saddle point of Equation (2) using the method called an alternating direction method of multipliers (ADMM).

Equation (3) can be further solved using an alternating direction method of multipliers (ADMM) [41]. Finally, the estimate of the $l^{th}$ sub-band is computed as [44]:

(3)${\widehat{b}}_l{}^{n+1}(\omega )=(\widehat{x}(\omega )-{\displaystyle \sum _{j\ne l}{\widehat{b}}_j(\omega )+\frac{\widehat{\lambda }(\omega )}{2}})\frac{1}{1+2\alpha {(\omega -{\omega }_l)}^2}$

Similarly, the center frequency is updated as:

(4)${\omega }_l^{n+1}=\frac{{\displaystyle \underset{0}{\overset{\infty }{\int }}\omega {\left|{\widehat{b}}_l(\omega )\right|}^{2}d\omega }}{{\displaystyle \underset{0}{\overset{\infty }{\int }}{\left|{\widehat{b}}_l(\omega )\right|}^{2}d\omega }}$

In this work, we used the two-dimensional (2D)-VMD [45] method to decompose the MRI and CT images. As stated above, 2D-VMD is a helpful method in extracting useful information such as edges and curves from the source images. Furthermore, VMD is a reliable method to deal with noisy images. Therefore, it can improve the quality of the fusion process even without employing additional preprocessing techniques.

B. Fusion Strategy Depending on LEM

As discussed before, the VMD adaptively decomposes the input images into bandlimited sub-bands called IMFs. Indeed, these IMFs characterize the image features of source images. To highlight and extract relevant features in the fused image, we require appropriate fusion rules. As discussed in Section 1, many fusion rules [46], such as minima, maxima, averaging, and PCA, have been widely explored for this purpose over the past few years. Among them, minima and maxima cause brightness distortions, averaging rule blurs the fused image, and PCA degrades the spectral information [15]. Furthermore, the fusion rules mentioned above may produce low spatial resolution issues [47]. The LEM-based [47] fusion rule is adopted to tackle the issues discussed above in this work.

We have demonstrated the influence of these fusion rules visually in Figure 4 and quantitatively in Table 2. As shown in Figure 4, the VMD with LEM fusion rule achieves visually satisfying results compared to VMD with other fusion rules. Similarly, as shown in Table 2, the fusion metric values calculated over 10 data sets proved the efficacy of the chosen LEM fusion rule.

The technical details of the LEM fusion rule are discussed as follows. The principal idea behind using LEM is to extract and preserve vital information with the help of local information constraints from both the images pixel by pixel [47]. The entire process of LEM is described in Algorithm 1.

Algorithm 1

$\mathrm{Let} \mathrm{us} \mathrm{consider} \mathrm{the} \mathrm{IMFs} \mathrm{of} \mathrm{the} \mathrm{first} \mathrm{image} \mathrm{as} {\mathrm{IMFs}}_A^i$$, \mathrm{and} \mathrm{the} \sec \mathrm{ond} \mathrm{image} \mathrm{as} {\mathrm{IMFs}}_B^i. \mathrm{The}$$\mathrm{local} \mathrm{information} L{E}_{\alpha }\left(x,y\right)$$\mathrm{of} IMF{s}_{\alpha }^i(\alpha =A,B)$ is evaluated using the following steps.$\mathrm{Input}: \mathrm{Decomposed} \mathrm{modes} \mathrm{of} \mathrm{images} {\mathrm{IMFs}}_A^i$$, {\mathrm{IMFs}}_B^i$.$\mathrm{Output}: \mathrm{Enhanced} \mathrm{decomposition} \mathrm{modes} {\mathrm{F}}^i{}_{IMF{s}_{A,B}}(x,y)$.$\mathrm{Step} 1: \mathrm{Calculate} \mathrm{the} \mathrm{local} \mathrm{information} LE{M}_{\alpha }\left(x,y\right)$$\mathrm{of} \mathrm{individual} \mathrm{modes} {\mathrm{IMFs}}_{\alpha }^i(\alpha =A,B)$ (5)${\mathrm{LEM}}_{\alpha }\left(x,y\right)={\displaystyle \sum _{i=1}^w{\displaystyle \sum _{j=1}^w{\left[IMF{s}_{\alpha }^i\left(x+i,y+j\right)\right]}^2\times W_k(i,j)}}$ $\mathrm{where}, W_k$ is given by: ${\mathrm{W}}_k=\left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right]$ $\mathrm{Step} 2: \mathrm{Choose} \mathrm{the} \mathrm{maximum} \mathrm{value} \mathrm{in} \mathrm{the} \mathrm{local} \mathrm{information} LE{M}_{\alpha }\left(x,y\right)$  (6)${\mathrm{L}}_{\alpha }\left(x,y\right)=\max \left\{LE{M}_{\alpha }\left(x+i,y+j\right)|1\le i,j\le 3\right\}$ Step 3: Calculate the binary decision weight maps (7)$X_1\left(x,y\right)=\{\begin{array}{c}1, \mathrm{if} L_A(\mathrm{x},\mathrm{y})>L_B(\mathrm{x},\mathrm{y})\\ 0, \mathrm{otherwise}\hfill \end{array}$ (8)$X_2\left(x,y\right)=\{\begin{array}{c}1, \mathrm{if} L_B(\mathrm{x},\mathrm{y})>L_A(\mathrm{x},\mathrm{y})\\ 0, \mathrm{otherwise}\hfill \end{array}$ $\mathrm{Step} 4: \mathrm{Obtain} \mathrm{the} \mathrm{enhanced} \mathrm{decomposition} \mathrm{modes} {\mathrm{F}}^i{}_{IMF{s}_{A,B}}(x,y)$  (9)${\mathrm{F}}^i{}_{IMF{s}_{A,B}}=X_1(x,y)\times IMF{s}_A^i(x,y)+X_2(x,y)\times IMF{s}_B^i(x,y)$

C. Synthesizing the Fused Image

We linearly combine all the enhanced IMFs obtained from each LEM fusion rule to construct the fused image. The whole process of the proposed fusion framework is given in Algorithm 2.

Algorithm 2

Input: Image A (MRI), Image B (CT).Output: The fused image F.Step 1: Image decomposition using VMD:$\mathrm{Employ} \mathrm{VMD} \mathrm{on} \mathrm{the} \mathrm{source} \mathrm{images} (\mathrm{A} \mathrm{and} \mathrm{B}) \mathrm{to} \mathrm{obtain} {\mathrm{IMF}}_{\mathrm{S}}$ which are represented as (10)$$\begin{gathered} VMD(A)=\left\{IMF{s}_A^1,IMF{s}_A^2\dots IMF{s}_A^i\right\}.,i=(1,2,\dots N); \\ VMD(B)=\left\{IMF{s}_B^1,IMF{s}_B^2\dots IMF{s}_B^i\right\}.,i=(1,2,\dots N) \end{gathered}$$ Step 2: LEM-based image fusion:$\left(\mathrm{a}\right) \mathrm{Estimate} \mathrm{the} \mathrm{local} \mathrm{information} LE{M}_{\alpha }\left(x,y\right)$$\mathrm{from} \mathrm{each} \mathrm{sub}-\mathrm{band} IMF{s}_{\alpha }^i(\alpha =A,B)$ using Equation (5).$\left(\mathrm{b}\right) \mathrm{Consider} \mathrm{the} \mathrm{maximum} \mathrm{value} {\mathrm{L}}_{\alpha }\left(x,y\right)$$\mathrm{of} LE{M}_{\alpha }\left(x,y\right)$ by Equation (6).$\left(\mathrm{c}\right) \mathrm{Evaluate} \mathrm{the} \mathrm{binary} \mathrm{decision} \mathrm{weight} \mathrm{maps} X_1\left(x,y\right)$$, X_2\left(x,y\right)$ with Equations (7) and (8).$\left(\mathrm{d}\right) \mathrm{Fuse} \mathrm{the} \mathrm{decomposed} \mathrm{modes} {\mathrm{F}}^i{}_{IMF{s}_{A,B}}(x,y)$ using Equation (9).Step 3: Reconstruct the fused image by summing all the fused sub-bands obtained from Step 2. (11)$F={\displaystyle \sum _{i=1}^N{F}^i{}_{\alpha }(x,y)} , i=1,\dots N$

D. Image Fusion Evaluation Metrics

In this paper, we used a few state-of-the-art image fusion metrics to estimate the information contribution of each source image in the fusion process. They are edge intensity (EI) [48], mutual information (MI) [49], visual information fidelity (VIF) [50,51], edge-based similarity measure ($Q_P^{AB/F}$) [52], structural similarity index measure (SSIM) [51,53], average gradient (AG) [54], root mean square error (RMSE) [15], peak signal-to-noise ratio (PSNR) [13,42]. EI represents the difference of luminance along the gradient direction in images. MI is used to measure the relative information between the source and the fused images. VIF estimates the visual information fidelity between the fused and source images depending on the Gaussian mixture model. The edge-based similarity ($Q_P^{AB/F}$) measure will be useful to provide the edge details in the fused image. RMSE computes a difference measure between the reference image and fused image. In this work, the maximum value of RMSE of MRI-fused images and CT-fused images is considered. Similarly, PSNR is also computed. Except for RMSE, the higher values of all these metrics imply better fusion. In the case of the RMSE, the lowest value yields a better result.

3. Results and Discussion

This section presents the experimental setup, results and analysis of the proposed method. First, we explain the experimental setup and methods, followed by data analysis using both qualitative and quantitative methods. Finally, we compare the proposed method with the existing literature for a fair assessment.

The experiments are conducted on a PC with Intel(R) Core (TM) i5-5200U [email protected] and RAM 8GB using MATLAB2018b. We have considered a whole-brain atlas website (http://www.med.harvard.edu/AANLIB/home.html, accessed on 1 September 2021) to conduct our experiments. For this purpose, 23 MRI-CT medical image data sets are taken from this database. All these data sets are registered with a resolution of 256 × 256. Image registration [55] is a necessary step prior to image fusion. It is defined as the process of mapping the input images with the help of a reference image. Such mapping aims to match the corresponding images based on specific features to assist in the image fusion process. The database contains various cross-sectional multimodal medical images, such as MRI (T1 and T2 weighted), CT, single-photon emission computed tomography (SPECT), and positron emission tomography (PET).

Furthermore, it has a wide range of brain images ranging from healthy to different brain diseases, including cerebrovascular, neoplastic, degenerative, and infectious diseases. We have considered 23 pairs of MRI-CT from fatal stroke (cerebrovascular disease) to validate our proposed approach (Supplementary Materials). Interested readers can find more details of this database in [56].

The efficacy of any image fusion algorithm can be verified using subjective (qualitative) and objective (quantitative) analysis. In Section 3.1, we first verified the subjective performance of various fusion algorithms and then performed objective analysis using fusion metrics in Section 3.2.

3.1. Subjective Assessment

Visual results of various MRI and CT fusion methods are shown in Figure 5, Figure 6 and Figure 7. A good MRI- and CT-fused image should contain both the soft tissue information and dense structure information of the MRI and CT images. We can draw the following observations by examining the visual quality of the four sets of MRI-CT fusion results using various methods.

1. Compared to all the other methods, our proposed algorithm provides a brighter outer region representing the CT image’s dense structure;

2. From Figure 5, Figure 6 and Figure 7, it can be seen that the fused images of methods (c)–(g) are yielding poor contrast;

3. Though the method (h) in all the Figure 5, Figure 6 and Figure 7 provides better contrast details; still, it is suffering from artifacts, especially in the CT region.

From Figure 5, Figure 6 and Figure 7, it can be noticed that the ASR method transfers both the CT and MRI information partially with low contrast. Next, coming to the CVT contains more MRI details than the CT. In the DTCWT method, we can find a few fusion artifacts in and around the CT region. Similarly, we can observe information fusion loss in the MSVD method. Compared with the methods mentioned above, CSMCA gives better visual quality, but the overall contrast of the image is reduced. The fused images with the NSST method are visually degraded due to both the fusion loss and artifacts. Overall, our proposed method retains the necessary information from the MRI and CT with minimum fusion losses. The comparison results of the MRI-CT fusion using various methods, including the proposed method on the 23 pairs of fatal stroke images, are shown in Figure 8 and Figure 9.

3.2. Objective Assessment

Here, we assess the fused image quality objectively using fusion metrics. Table 3, Table 4 and Table 5 demonstrate the objective assessment of the three fatal-stroke images proposed and other existing approaches (sets: 7, 11, and 15) subjectively analyzed earlier. In addition, we have presented the average objective metric scores of all the 23 sets (fatal-stroke)in Table 6. Fusion metrics except for RMSE with the first highest values are highlighted in bold font, and the second-highest values are underlined. The first-lowest value of the RMSE is indicated in bold, and the second-lowest value is underlined. A number within the bracket at the end of the quantitative metric scores represents the rank of the fusion algorithm. In these Tables, the ranking scheme is considered for better quantitative analysis of fusion algorithms.

Comprehensively, the proposed framework is the only approach that occupies the first two ranks for all eight metrics among all the seven methods. It indicates that our method has robust performance (i.e., stable and promising performance) than other existing techniques. Specifically, our approach always remains in the first position on VIFF and RMSE for all four data sets, as shown in Table 3, Table 4 and Table 5.

Average quantitative analysis of the proposed and other state-of-the-art methods calculated over 23 pairs of MRI-CT (fatal stroke) are presented in Table 6. The proposed method occupied the first position by overperforming other fusion algorithms when average values are considered in fusion metrics.

In general, the consistent performance of any image fusion algorithm in quantitative results is mainly due to the good visual quality of fused images, fusion gain, and less fusion loss and fusion artifacts. We have already seen from the visual result analysis that the proposed method can transfer the source image information into the fused image with less fusion loss and artifacts compared to the other fusion algorithms. It is also evident from the fusion metrics that our method is giving a stable performance.

Hence, we can conclude that the proposed method is promising, stable, and efficient from qualitative and quantitative comparative analysis.

4. Conclusions and Future Scope

We proposed a multi-modal medical image fusion framework with VMD and LEM to fuse MRI and CT medical images in this work. By using an adaptive decomposition technique VMD, significant IMFs are derived from the source images. This decomposition process can preserve some details of source images. However, these details are not sufficient to fulfill the clinical needs of radiologists. Hence, we used a LEM fusion rule to preserve complementary information from IMFs, an essential criterion during medical image diagnosis. All the experiments are evaluated on the Whole Brain Atlas benchmark data sets to analyze the efficacy of the proposed methodology. The experimental results reveal that the proposed framework attained better visual perception. Even objective assessment in terms of average EI (64.582), MI (3.830), VIFF (0.498), $Q_P^{AB/F}$ (0.542), SSIM (0.6574), RMSE (0.020), AG (6.41), and PSNR (20.291) demonstrated quantitative fusion performance better than the existing multi-modal fusion approaches. In the future, we wish to conduct experiments with extensive data that contain images of MRI and CT with different disease information. Additionally, we consider extending this work to both 2D and 3D image clinical applications. Furthermore, we would like to verify the effectiveness of the proposed method for other image fusion applications such as digital photography, remote sensing, battlefield monitoring, and military.

Author Contributions

Conceptualization, S.P. and R.D.; methodology, S.P., D.P.B., R.D., K.N.V.P.S.R. and G.R.N.; software, S.P., D.P.B. and K.N.V.P.S.R.; validation, S.P., D.P.B. and K.N.V.P.S.R.; formal analysis, S.P. and G.R.N.; investigation, K.N.V.P.S.R. and G.R.N.; resources, S.P. and R.D.; data curation, K.N.V.P.S.R. and D.P.B.; writing—original draft preparation, S.P., D.P.B. and K.N.V.P.S.R.; writing—review and editing, S.P., D.P.B. and K.N.V.P.S.R.; visualization, S.P., R.D. and G.R.N.; supervision, R.D. and G.R.N.; project administration, K.N.V.P.S.R. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Imaging data can be downloaded from the link: https://www.med.harvard.edu/aanlib/, accessed on 15 November 2021.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Illustration of the classification of different medical brain imaging modalities.

Figure 2. Proposed MRI-CT medical image fusion scheme.

Figure 3. IMFs obtained after VMD decomposition: (a) MRI image, (b) and (c–g) are approximation and detail images of (a), respectively. (h) CT image, (i) and (j–n) are approximation and detail images of (h), respectively.

Figure 4. Visual quality analysis of various fusion rules on MRI-CT image pair. (a) MRI image, (b) CT image, (c) VMD-AVG, (d) VMD-MAX, (e) VMD-MIN (f) VMD-LEM.

Figure 5. Visual quality analysis of various fusion algorithms for MRI-CT (set-7). (a) MRI image, (b) CT image, (c) ASR, (d) CVT, (e) DTCWT, (f) MSVD, (g) CSMCA, (h) NSST, (i) proposed method.

Figure 6. Visual quality analysis of various fusion algorithms for MRI-CT (set-11). (a) MRI image, (b) CT image, (c) ASR, (d) CVT, (e) DTCWT, (f) MSVD, (g) CSMCA, (h) NSST, (i) proposed method.

Figure 7. Visual quality analysis of various fusion algorithms for MRI-CT (set-15). (a) MRI image, (b) CT image, (c) ASR, (d) CVT, (e) DTCWT, (f) MSVD, (g) CSMCA, (h) NSST, (i) proposed method.

Figure 8. The results of various methods on first 10 pairs of MRI-T images (fatal stroke).

Figure 9. The results of various methods on next 13 pairs of MRI-CT images (fatal stroke).

Table 1

Brief summary of the image fusion methods.

Image Fusion Types		Fusion Methods	Advantages	Drawbacks
Spatial domain		Average, minimum, maximum, morphological operators [11], Principal Component Analysis (PCA) [14], Independent Component Analysis (ICA) [29]	Easy to implement. Computationally efficient	Reduces the contrast, produces brightness or color distortions.May give desirable results for a few fusion datasets.
Transform domain	Pyramidal methods	Contrast Pyramid [30],Ratio of the low-pass pyramid [31],Laplacian [19]	Provides spectralinformation	May produce artifacts around edges. Suffer from blocking artifacts
	Wavelet transform	Discrete wavelet transform (DWT) [15],Shift invariant discrete wavelet transform (SIDWT) [32],Dual-tree complex wavelet transform (DcxDWT) [20]	Providesdirectionalinformation	May produce artifacts around edges because of shift variant nature.Computationally expensive and demands large memory.
	Multiscale geometricanalysis (MGA)	Curvelet [24],Contourlet [33],Shearlet [34],Nonsubsampled Shearlet transform (NSST) [28]	Provides the edges and texture region	Loss in texture parts, high memory requirement, demands high run time.

Table 2

Average quantitative analysis of various fusion rules on 10 pairs of MRI-CT images.

Metrics	Methods
	VMD-AVG	VMD-MAX	VMD-MIN	VMD-LEM
EI	48.439	58.322	36.487	71.751
MI	4.384	4.376	3.486	4.391
VIFF	0.335	0.397	0.063	0.428
$$Q_P^{AB/F}$$	0.307	0.356	0.198	0.443
SSIM	0.599	0.232	0.563	0.621
AG	4.845	5.714	3.735	6.973
RMSE	0.0296	0.005	0.036	0.020
PSNR	15.926	14.553	15.869	18.580

Table 3

Quantitative analysis of various fusion methods for MRI-CT (set-7).

Metrics	Methods
	ASR	CVT	DTCWT	MSVD	CSMCA	NSST	Proposed Method
EI	85.184	91.417 (1)	88.853	77.183	87.219	81.907	90.390 (2)
MI	3.948 (2)	3.548	3.656	3.490	3.811	3.703	4.079 (1)
VIFF	0.321	0.290	0.280	0.344 (2)	0.319	0.267	0.406 (1)
$$Q_P^{AB/F}$$	0.535	0.478	0.500	0.427	0.536 (2)	0.373	0.538 (1)
SSIM	0.563	0.376	0.499	0.548	0.629 (2)	0.520	0.697 (1)
AG	8.561	9.140 (1)	8.933	8.332	8.674	8.368	9.008 (2)
RMSE	0.034	0.034	0.034	0.034	0.035	0.027 (2)	0.020
PSNR	16.328	16.749	17.166	13.28	17.393 (2)	13.976	21.342 (1)

Table 4

Quantitative analysis of the various fusion methods for MRI-CT (set-11).

Metrics	Methods
	ASR	CVT	DTCWT	MSVD	CSMCA	NSST	Proposed Method
EI	67.026	79.944 (2)	75.086	64.169	70.435	75.318	80.087 (1)
MI	4.279	3.904	4.030	4.227	4.346 (1)	4.116	4.339 (2)
VIFF	0.272	0.254	0.249	0.286	0.297 (2)	0.241	0.356 (1)
$$Q_P^{AB/F}$$	0.472	0.421	0.435	0.392	0.481 (1)	0.421	0.480 (2)
SSIM	0.593	0.276	0.413	0.301	0.537	0.600 (1)	0.599 (2)
AG	6.662	7.887 (2)	7.421	6.812	6.877	7.471	7.980 (1)
RMSE	0.029	0.029	0.029	0.028	0.029	0.024 (2)	0.021 (1)
PSNR	16.857	17.171	17.720	15.804	17.892 (1)	13.981	17.794 (2)

Table 5

Quantitative analysis of the state-of-the-art methods for MRI-CT (set-15) dataset.

Metrics	Methods
	ASR	CVT	DTCWT	MSVD	CSMCA	NSST	Proposed Method
EI	51.347	63.877	58.355	49.732	51.899	65.474 (2)	65.802 (1)
MI	4.186	3.878	3.995	4.090	4.284 (2)	4.214	4.549 (1)
VIFF	0.356	0.362	0.365	0.348	0.412 (2)	0.340	0.484 (1)
$$Q_P^{AB/F}$$	0.465 (2)	0.418	0.431	0.380	0.461	0.446	0.478 (1)
SSIM	0.674 (2)	0.338	0.507	0.417	0.663	0.590	0.694 (1)
AG	5.065	6.231	5.719	5.197	5.045	6.349 (1)	6.326 (2)
RMSE	0.028	0.029	0.029	0.026	0.028	0.022 (2)	0.018 (1)
PSNR	17.396	17.268	17.649	16.392	18.644 (1)	14.096	18.024 (2)

Table 6

Average quantitative analysis of the proposed method (23 pairs of MRI-CT) and other state-of-the-art methods.

Metrics	Methods
	ASR	CVT	DTCWT	MSVD	CSMCA	NSST	ProposedMethod
EI	57.800	64.531	61.820	50.850	58.592	62.404	64.582
MI	3.666	3.360	3.446	3.694	3.657	3.740	3.830
VIFF	0.376	0.362	0.358	0.365	0.401	0.364	0.498
$$Q_P^{AB/F}$$	0.541	0.483	0.500	0.399	0.531	0.439	0.542
SSIM	0.651	0.350	0.503	0.614	0.634	0.586	0.657
RMSE	0.029	0.029	0.029	0.029	0.029	0.022	0.020
AG	5.772	6.390	6.148	5.427	5.771	6.217	6.412
PSNR	16.803	16.972	17.242	16.000	17.757	16.021	20.291

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/app112210975/s1. The qualitative (Figures S1–S23) and quantitative results (Tables S1–S23) of all the 23 pairs of images used in this work are given in the supplementary material.

References

1. Vishwakarma, A.; Bhuyan, M.K. Image Fusion Using Adjustable Non-subsampled Shearlet Transform. IEEE Trans. Instrum. Meas.; 2018; 68, pp. 3367-3378. [DOI: https://dx.doi.org/10.1109/TIM.2018.2877285]

2. Ouahabi, A. A review of wavelet denoising in medical imaging. Proceedings of the 2013 8th International Workshop on Systems, Signal Processing and their Applications (WoSSPA); Algiers, Algeria, 12–15 May 2013; IEEE: New York, NY, USA, 2013; pp. 19-26.

3. Ahmed, S.; Messali, Z.; Ouahabi, A.; Trepout, S.; Messaoudi, C.; Marco, S. Nonparametric Denoising Methods Based on Contourlet Transform with Sharp Frequency Localization: Application to Low Exposure Time Electron Microscopy Images. Entropy; 2015; 17, pp. 3461-3478. [DOI: https://dx.doi.org/10.3390/e17053461]

4. Unser, M. Texture classification and segmentation using wavelet frames. IEEE Trans. Image Process.; 1995; 4, pp. 1549-1560. [DOI: https://dx.doi.org/10.1109/83.469936]

5. Meriem, D.; Abdeldjalil, O.; Hadj, B.; Adrian, B.; Denis, K. Discrete wavelet for multifractal texture classification: Application to medical ultrasound imaging. Proceedings of the 2010 IEEE International Conference on Image Processing; Hong Kong, China, 26–29 September 2010; IEEE: New York, NY, USA, 2010; pp. 637-640.

6. Hatt, C.R.; Jain, A.K.; Parthasarathy, V.; Lang, A.; Raval, A.N. MRI—3D ultrasound—X-ray image fusion with electromagnetic tracking for transendocardial therapeutic injections: In-vitro validation and in-vivo feasibility. Comput. Med. Imaging Graph.; 2013; 37, pp. 162-173. [DOI: https://dx.doi.org/10.1016/j.compmedimag.2013.03.006]

7. Labat, V.; Remenieras, J.P.; BouMatar, O.; Ouahabi, A.; Patat, F. Harmonic propagation of finite amplitude sound beams: Experimental determination of the nonlinearity parameter B/A. Ultrasonics; 2000; 38, pp. 292-296. [DOI: https://dx.doi.org/10.1016/S0041-624X(99)00113-4]

8. Dasarathy, B.V. Medical image fusion: A survey of the state of the art. Inf. Fusion; 2014; 19, pp. 4-19. [DOI: https://dx.doi.org/10.1016/j.inffus.2013.12.002]

9. Zhao, W.; Lu, H. Medical Image Fusion and Denoising with Alternating Sequential Filter and Adaptive Fractional Order Total Variation. IEEE Trans. Instrum. Meas.; 2017; 66, pp. 2283-2294. [DOI: https://dx.doi.org/10.1109/TIM.2017.2700198]

10. El-Gamal, F.E.-Z.A.; Elmogy, M.; Atwan, A. Current trends in medical image registration and fusion. Egypt. Inf. J.; 2016; 17, pp. 99-124. [DOI: https://dx.doi.org/10.1016/j.eij.2015.09.002]

11. Li, S.; Kang, X.; Fang, L.; Hu, J.; Yin, H. Pixel-level image fusion: A survey of the state of the art. Inf. Fusion; 2017; 33, pp. 100-112. [DOI: https://dx.doi.org/10.1016/j.inffus.2016.05.004]

12. Li, S.; Kwok, J.T.; Wang, Y. Multifocus image fusion using artificial neural networks. Pattern Recognit. Lett.; 2002; 23, pp. 985-997. [DOI: https://dx.doi.org/10.1016/S0167-8655(02)00029-6]

13. Li, S.; Kwok, J.-Y.; Tsang, I.-H.; Wang, Y. Fusing Images with Different Focuses Using Support Vector Machines. IEEE Trans. Neural Netw.; 2004; 15, pp. 1555-1561. [DOI: https://dx.doi.org/10.1109/TNN.2004.837780]

14. Vijayarajan, R.; Muttan, S. Iterative block level principal component averaging medical image fusion. Optik; 2014; 125, pp. 4751-4757. [DOI: https://dx.doi.org/10.1016/j.ijleo.2014.04.068]

15. Naidu, V.; Raol, J. Pixel-level Image Fusion using Wavelets and Principal Component Analysis. Def. Sci. J.; 2008; 58, pp. 338-352. [DOI: https://dx.doi.org/10.14429/dsj.58.1653]

16. Singh, S.; Anand, R.S. Multimodal Medical Image Fusion Using Hybrid Layer Decomposition with CNN-Based Feature Mapping and Structural Clustering. IEEE Trans. Instrum. Meas.; 2020; 69, pp. 3855-3865. [DOI: https://dx.doi.org/10.1109/TIM.2019.2933341]

17. Du, J.; Li, W.; Lu, K.; Xiao, B. An overview of multi-modal medical image fusion. Neurocomputing; 2016; 215, pp. 3-20. [DOI: https://dx.doi.org/10.1016/j.neucom.2015.07.160]

18. Kappala, V.K.; Pradhan, J.; Turuk, A.K.; Silva, V.N.H.; Majhi, S.; Das, S.K. A Point-to-Multi-Point Tracking System for FSO Communication. IEEE Trans. Instrum. Meas.; 2021; 70, pp. 1-10. [DOI: https://dx.doi.org/10.1109/TIM.2021.3115202]

19. Mitianoudis, N.; Stathaki, T. Pixel-based and region-based image fusion schemes using ICA bases. Inf. Fusion; 2007; 8, pp. 131-142. [DOI: https://dx.doi.org/10.1016/j.inffus.2005.09.001]

20. Toet, A.; van Ruyven, L.J.; Valeton, J.M. Merging Thermal And Visual Images By A Contrast Pyramid. Opt. Eng.; 1989; 28, 287789. [DOI: https://dx.doi.org/10.1117/12.7977034]

21. Toet, A. Image fusion by a ratio of low-pass pyramid. Pattern Recognit. Lett.; 1989; 9, pp. 245-253. [DOI: https://dx.doi.org/10.1016/0167-8655(89)90003-2]

22. Li, X.; Guo, X.; Han, P.; Wang, X.; Li, H.; Luo, T. Laplacian Redecomposition for Multimodal Medical Image Fusion. IEEE Trans. Instrum. Meas.; 2020; 69, pp. 6880-6890. [DOI: https://dx.doi.org/10.1109/TIM.2020.2975405]

23. Li, H.; Manjunath, B.S.; Mitra, S.K. Multisensor Image Fusion Using the Wavelet Transform. Graph. Model. Image Process.; 1995; 57, pp. 235-245. [DOI: https://dx.doi.org/10.1006/gmip.1995.1022]

24. Lewis, J.J.; O’Callaghan, R.J.; Nikolov, S.G.; Bull, D.R.; Canagarajah, N. Pixel- and region-based image fusion with complex wavelets. Inf. Fusion; 2007; 8, pp. 119-130. [DOI: https://dx.doi.org/10.1016/j.inffus.2005.09.006]

25. Nencini, F.; Garzelli, A.; Baronti, S.; Alparone, L. Remote sensing image fusion using the curvelet transform. Inf. Fusion; 2007; 8, pp. 143-156. [DOI: https://dx.doi.org/10.1016/j.inffus.2006.02.001]

26. Yang, L.; Guo, B.L.; Ni, W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. Neurocomputing; 2008; 72, pp. 203-211. [DOI: https://dx.doi.org/10.1016/j.neucom.2008.02.025]

27. Miao, Q.; Shi, C.; Xu, P.; Yang, M.; Shi, Y. A novel algorithm of image fusion using shearlets. Opt. Commun.; 2011; 284, pp. 1540-1547. [DOI: https://dx.doi.org/10.1016/j.optcom.2010.11.048]

28. Yin, M.; Liu, X.; Liu, Y.; Chen, X. Medical Image Fusion With Parameter-Adaptive Pulse Coupled-Neural Network in Nonsubsampled Shearlet Transform Domain. IEEE Trans. Instrum. Meas.; 2018; 68, pp. 49-64. [DOI: https://dx.doi.org/10.1109/TIM.2018.2838778]

29. Kirankumar, Y.; Shenbaga Devi, S. Transform-based medical image fusion. Int. J. Biomed. Eng. Technol.; 2007; 1, pp. 101-110. [DOI: https://dx.doi.org/10.1504/IJBET.2007.014140]

30. Naidu, V.P.S. Image Fusion Technique using Multi-resolution Singular Value Decomposition. Def. Sci. J.; 2011; 61, 479. [DOI: https://dx.doi.org/10.14429/dsj.61.705]

31. Hermessi, H.; Mourali, O.; Zagrouba, E. Multimodal medical image fusion review: Theoretical background and recent advances. Signal Process.; 2021; 183, 108036. [DOI: https://dx.doi.org/10.1016/j.sigpro.2021.108036]

32. Wan, H.; Tang, X.; Zhu, Z.; Xiao, B.; Li, W. Multi-Focus Color Image Fusion Based on Quaternion Multi-Scale Singular Value Decomposition. Front. Neurorobot.; 2021; 15, 76. [DOI: https://dx.doi.org/10.3389/fnbot.2021.695960]

33. Singh, S.; Anand, R.S. Multimodal Medical Image Sensor Fusion Model Using Sparse K-SVD Dictionary Learning in Nonsubsampled Shearlet Domain. IEEE Trans. Instrum. Meas.; 2020; 69, pp. 593-607. [DOI: https://dx.doi.org/10.1109/TIM.2019.2902808]

34. Liu, Y.; Chen, X.; Ward, R.K.; Wang, Z.J. Medical Image Fusion via Convolutional Sparsity Based Morphological Component Analysis. IEEE Signal Process. Lett.; 2019; 26, pp. 485-489. [DOI: https://dx.doi.org/10.1109/LSP.2019.2895749]

35. Maqsood, S.; Javed, U. Multi-modal Medical Image Fusion based on Two-scale Image Decomposition and Sparse Representation. Biomed. Signal Process. Control; 2020; 57, 101810. [DOI: https://dx.doi.org/10.1016/j.bspc.2019.101810]

36. Pankaj, D.; Sachin Kumar, S.; Mohan, N.; Soman, K.P. Image Fusion using Variational Mode Decomposition. Indian J. Sci. Technol.; 2016; 9, pp. 1-8. [DOI: https://dx.doi.org/10.17485/ijst/2016/v9i45/99068]

37. Vishnu Pradeep, V.; Sowmya, V.; Soman, K. Variational mode decomposition based multispectral and panchromatic image fusion. IJCTA; 2016; 9, pp. 8051-8059.

38. Pajares, G.; de la Cruz, J.M. A wavelet-based image fusion tutorial. Pattern Recognit.; 2004; 37, pp. 1855-1872. [DOI: https://dx.doi.org/10.1016/j.patcog.2004.03.010]

39. Ouahabi, A. Signal and Image Multiresolution Analysis; John Wiley & Sons: Hoboken, NJ, USA, 2012; ISBN 1118568664

40. Nunes, J.; Bouaoune, Y.; Delechelle, E.; Niang, O.; Bunel, P. Image analysis by bidimensional empirical mode decomposition. Image Vis. Comput.; 2003; 21, pp. 1019-1026. [DOI: https://dx.doi.org/10.1016/S0262-8856(03)00094-5]

41. Gilles, J. Empirical Wavelet Transform. IEEE Trans. Signal Process.; 2013; 61, pp. 3999-4010. [DOI: https://dx.doi.org/10.1109/TSP.2013.2265222]

42. Dragomiretskiy, K.; Zosso, D. Variational Mode Decomposition. IEEE Trans. Signal Process.; 2013; 62, pp. 531-544. [DOI: https://dx.doi.org/10.1109/TSP.2013.2288675]

43. Lahmiri, S.; Boukadoum, M. Biomedical image denoising using variational mode decomposition. Proceedings of the 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS); Lausanne, Switzerland, 22–24 October 2014; pp. 340-343. [DOI: https://dx.doi.org/10.1109/BioCAS.2014.6981732]

44. Lahmiri, S. Denoising techniques in adaptive multi-resolution domains with applications to biomedical images. Health Technol. Lett.; 2017; 4, pp. 25-29. [DOI: https://dx.doi.org/10.1049/htl.2016.0021]

45. Maheshwari, S.; Pachori, R.B.; Kanhangad, V.; Bhandary, S.V.; Acharya, U.R. Iterative variational mode decomposition based automated detection of glaucoma using fundus images. Comput. Biol. Med.; 2017; 88, pp. 142-149. [DOI: https://dx.doi.org/10.1016/j.compbiomed.2017.06.017]

46. Konstantin, D.; Zosso, D. Two-dimensional variational mode decomposition. Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition; Hong Kong, China, 13–16 January 2015; Springer: Berlin/Heidelberg, Germany, 2015; pp. 197-208.

47. Polinati, S.; Dhuli, R. A review on multi-model medical image fusion. Proceedings of the International Conference on Signal Processing, Communications and Computing (ICSPCC 2019); Liaoning, China, 20–22 September 2019; ICCSP: Tamilnadu, India, 2019.

48. Du, J.; Li, W.; Xiao, B. Anatomical-Functional Image Fusion by Information of Interest in Local Laplacian Filtering Domain. IEEE Trans. Image Process.; 2017; 26, pp. 5855-5866. [DOI: https://dx.doi.org/10.1109/TIP.2017.2745202]

49. Wang, Y.; Du, H.; Xu, J.; Liu, Y. A no-reference perceptual blur metric based on complex edge analysis. Proceedings of the 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content; Beijing, China, 21–23 September 2012; IEEE: New York, NY, USA, 2012; pp. 487-491.

50. Hossny, M.; Nahavandi, S.; Creighton, D. Comments on ‘Information measure for performance of image fusion’. Electron. Lett.; 2008; 44, pp. 2-4. [DOI: https://dx.doi.org/10.1049/el:20081754]

51. Sheikh, H.R.; Bovik, A.C. Image information and visual quality. IEEE Trans. Image Process.; 2006; 15, pp. 430-444. [DOI: https://dx.doi.org/10.1109/TIP.2005.859378] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/16479813]

52. Ferroukhi, M.; Ouahabi, A.; Attari, M.; Habchi, Y.; Taleb-Ahmed, A. Medical Video Coding Based on 2nd-Generation Wavelets: Performance Evaluation. Electronics; 2019; 8, 88. [DOI: https://dx.doi.org/10.3390/electronics8010088]

53. Xydeas, C.S.; Petrović, V. Objective image fusion performance measure. Electron. Lett.; 2000; 36, 308. [DOI: https://dx.doi.org/10.1049/el:20000267]

54. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process.; 2004; 13, pp. 600-612. [DOI: https://dx.doi.org/10.1109/TIP.2003.819861]

55. Singh, R.; Khare, A. Multiscale medical image fusion in wavelet domain. Sci. World J.; 2013; [DOI: https://dx.doi.org/10.1155/2013/521034]

56. Oliveira, F.P.M.; Tavares, J.M.R.S. Medical image registration: A review. Comput. Methods Biomech. Biomed. Engin.; 2014; 17, pp. 73-93. [DOI: https://dx.doi.org/10.1080/10255842.2012.670855] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22435355]