An Approach for the Pan Sharpening of Very High Resolution Satellite Images Using a CIELab Color Based Component Substitution Algorithm

Abstract

Recent very high spatial resolution (VHR) remote sensing satellites provide high spatial resolution panchromatic (Pan) images in addition to multispectral (MS) images. The pan sharpening process has a critical role in image processing tasks and geospatial information extraction from satellite images. In this research, CIELab color based component substitution Pan sharpening algorithm was proposed for Pan sharpening of the Pleiades VHR images. The proposed method was compared with the state-of-the-art Pan sharpening methods, such as IHS, EHLERS, NNDiffuse and GIHS. The selected study region included ten test sites, each of them representing complex landscapes with various land categories, to evaluate the performance of Pan sharpening methods in varying land surface characteristics. The spatial and spectral performance of the Pan sharpening methods were evaluated by eleven accuracy metrics and visual interpretation. The results of the evaluation indicated that proposed CIELab color-based method reached promising results and improved the spectral and spatial information preservation.

1. Introduction

The earth observation satellites with very high resolution (VHR) optical sensors provide a multispectral (MS) image and a panchromatic (Pan) image that are acquired simultaneously in order to provide essential accommodation between spectral and spatial resolution, which is an important consideration for optical satellite sensors due to their physical limitations [1,2]. Spectral diversity is important for modeling the spectral characteristics of different land cover/use classes and identifying them; on the other hand, spatial information is very crucial for identifying spatial details and geometric characteristics. The Pan image provides high spatial resolution with a single, wide range spectral band, whereas the MS image provides several spectral bands in different sections of the electromagnetic spectrum with low spatial resolution in order to meet the abovementioned requirements.

The fusion of Pan and MS images that are acquired over the same area from the single or multiple satellite system is referred to as Pan sharpening. The main aim of Pan sharpening is to create a high-resolution MS image, having the spatial resolution of Pan but preserving the spectral characteristics of MS [3]. Unlike the challenging problem of multi-sensor data fusion, single sensor Pan sharpening does not need image-to-image registration, as the Pan and MS sensors are mounted on the same platform and the images are acquired simultaneously with well-matching viewing geometry [4]. Several earth observation satellites, such as Geo-Eye, OrbView, QuickBird, WorldView, Pléiades and Spot, have this capability, and bundle (PAN+MS) products from these systems can be used directly as the input for Pan sharpening.

An ideal Pan sharpening algorithm leads to the best performance in spatial and spectral domains by keeping the spatial resolution of a Pan image and preserving the spectral characteristics of an MS image. Launching of VHR sensors led to the appearance of diverse Pan sharpening methods in recent decades [5,6,7]. In addition, Pan sharpening is a primary image enhancement step for many remote sensing applications, such as object detection [8], change detection [9], image segmentation and clustering [10,11], scene interpretation and visual image analysis [12]. Commonly, image fusion can be classified into three levels—pixel level, feature level and decision or knowledge level—while the Pan sharpening is categorized as a sub-pixel level process [13,14].

Pan sharpening algorithms can be divided into four groups: (1) rationing methods; (2) injection-based methods; (3) model-based methods; and (4) component substitution (CS) methods. Of these methods, CS algorithms are more practical because of their calculation speed and performance compatibility. The CS methods can be categorized into four classes according to the transform matrix used in the algorithm; which are principle component analysis (PCA) [15,16], intensity-hue-saturation (IHS) [7,17], Gram–Schmidt (GS) [18,19] and generalized component substitution (GCS) [5,20]. The common and general limitation of all CS-based methods is the distortion in spectral characteristics when compared to original MS image [21,22].

This research proposes a robust CS method for Pan sharpening the Pleiades VHR satellite images with the aim of enhanced spatial resolution and reduced spectral distortion. The principle of the proposed method is similar to the IHS method, where a uniform CIELab color space based on human eye spectral response is used instead of IHS color space [23]. The CIELab color space has been used for different image processing tasks. Wirth and Nikitenko, 2010 [24], investigated the performance of CIELab color space on the application of unsharp masking and fuzzy morphological sharpening algorithms. In the study of [25], the experiments of the content-based image retrieval (CBIR) were used to evaluate the performance of CIELab and the other three color spaces (RGB, CIELuv and HSV) on an image retrieval process. In addition, CIELab color space was used to help different image segmentation tasks [26,27]. In a previous Pan sharpening research, color normalization-based on CIELab color space aided the image fusion algorithm with sharpening a Hyperion hyperspectral image with an Ikonos Pan image using the spectral mixing-based color preservation model [28]. In another study, a remote sensing image fusion technique using CIELab color space was proposed by Jin et al. [29]. In that study, the authors improved the performance of image fusion techniques by combining non-subsampled shearlet transform and pulse coupled neural network. However, this approach is computationally complicated and there is lack of a specific satellite dataset.

Although the CIELab method is used in different image processing tasks applied on natural and satellite images, evaluation of its performance on the Pan sharpening process is limited and there is no detailed evaluation of this method on VHR image Pan sharpening by considering the different landscape characteristics and with use of spatial and spectral metrics in addition to visual interpretation yet, to our knowledge. This research focused on proposing a robust, CIELab-based Pan sharpening approach and aimed to fill the abovementioned gap in detailed investigation and accuracy assessment of CIELab-based Pan sharpening in the literature. In this research, results from the proposed method were compared with the results from the six well-known methods, which are Ehlers, Generalized IHS, IHS, Gram-Schmitt, HCS and NNDiffuse methods. Pleiades satellite images of ten different test sites having different landscape characteristics were comparatively evaluated to check the spatial and spectral performance of the proposed method with quantitative accuracy metrics. In addition, visual interpretation-based analyses were performed on the results in order to discuss the performances of methods. The results illustrated advantages of using uniform color space for the aim of the Pan sharpening application.

2. State of Art

A brief review of the state of the art methods that used for comparative analysis with respect to the proposed method are presented in Table 1.

3. Methodology and Accuracy Assessment

This research proposes a new Pan sharpening method that relies on the CIELab transform, which modifies the color components of images to be used for Pan sharpening of VHR satellite images. The flowchart of the proposed method is given in Figure 1, and the details of the proposed method are described in the sub sections.

3.1. Multispectral Image Transform to CIELab Color System

The RGB color system was designed in such a way that it includes nearly all primary colors and can be comprehended by human vision. Nevertheless, it is a tough task to deal with RGB color due to strong correlation between its components [29]. In this study, a uniform and complete color model, the CIELab color system, was used. In this uniform color system, a variation in the coordinates of the color component provides the same amount of variation in the luminance and saturation components [10]. Besides, this color space is projected to draw human vision, unlike the RGB and CMYK (cyan, magenta, yellow, black) color spaces [38]. The CIELab color system was used in the proposed Pan sharpening approach in order to reduce the spectral distortion, while maintaining the color perception of human vision [39].

The design of the CIELab color system is based on Hering’s theory, which indicates that only red (R), green (G), blue (B) and yellow (Y) are unique among the thousands of colors that are used to characterize the hue component [40]. Although, other colors can be produced using these unique colors (for example it is possible to obtain orange by mixing red and yellow), they (R, G, B and Y) can be described only with their own name. R, G, B and Y, with black (B) and white (W), constitutes a color system with six basic color properties and three opponent pairs: R/G, Y/B and B/W. The opponency idea rises from observation upon colors attributes, which proves no color could be characterized using both blue and yellow or red and green together [41]. A blue shade of yellow does not exist. These three opponent pairs are represented in the form of a three-dimensional color space, as illustrated in Figure 2. In this figure, the vertical axis L* represents the luminance, in which perfect black is represented by 0 and perfect white is represented by 100. The a* and the b* are the axes that are perpendicular to luminance indicated chromaticity, and stand for redness/greenness and yellowness/blueness, respectively. Positive values represent redness (for a* component) and yellowness (for b* component), whereas greenness and blueness are denoted with negative values.

The L*, a* and b* values are computed using XYZ values. The XYZ system that was based on the RGB color space was presented by the International Commission on Illumination, CIE (Commission international de l’éclairage), in the 1920s and patented in 1931. The difference of RGB and XYZ lies in the light sources. The R, G and B elements are real light sources of known characteristics, whereas the X, Y and Z elements are three theoretical sources, which are selected in a way that all visible colors can be defined as a density of just-positive units of the three primary sources [10].

Occasionally, the colorimetric calculations with the use of color matching functions produce negative lobs. This problem can be solved by transforming the real light sources to these theoretical sources. In this color space, red, green and blue colors are more saturated than any spectral RGB. X, Y and Z components, represent red, green and blue colors respectively. RGB to XYZ and its reverse transformations can be performed by following equations:

(1)$\left[\begin{array}{c}\mathrm{X}\\ \mathrm{Y}\\ \mathrm{Z}\end{array}\right]=\left[\begin{array}{ccc}0.4124564& 0.3575761& 0.1804375\\ 0.2126729& 0.7151522& 0.0721750\\ 0.0193339& 0.1191920& 0.9503041\end{array}\right]\text{•}\left[\begin{array}{c}\mathrm{R}\\ \mathrm{G}\\ \mathrm{B}\end{array}\right]$

(2) $\left[\begin{array}{c}\mathrm{R}\\ \mathrm{G}\\ \mathrm{B}\end{array}\right]=\left[\begin{array}{ccc}3.2404542& -1.5371385& -0.4985314\\ -0.9692660& 1.8760108& 0.0415560\\ 0.0556434& -0.2040259& 1.0572252\end{array}\right]\text{•}\left[\begin{array}{c}\mathrm{X}\\ \mathrm{Y}\\ \mathrm{Z}\end{array}\right].$

Lab system is calculated by the following equations [23]:

(3)$\mathrm{L}=116{\mathrm{F}}_{\mathrm{Y}}-16$

(4)${\mathrm{a}=500[\mathrm{F}}_{\mathrm{X}} {-\mathrm{F}}_{\mathrm{Y}}]$

(5)${\mathrm{b}=200[\mathrm{F}}_{\mathrm{Y}} {-\mathrm{F}}_{\mathrm{Z}}]$

(6)${\mathrm{Where}\mathrm{F}}_{\mathrm{X}}=(\frac{\mathrm{X}}{{\mathrm{X}}_{\mathrm{n}}}{)}^{\frac{1}{3}}\mathrm{if}(\frac{\mathrm{X}}{{\mathrm{X}}_{\mathrm{n}}})\text{>}(\frac{24}{116}{)}^3$

(7)${\mathrm{And}\mathrm{F}}_{\mathrm{X}}=(\frac{841}{108})(\frac{\mathrm{X}}{{\mathrm{X}}_{\mathrm{n}}})+\frac{16}{116}\mathrm{if}(\frac{\mathrm{X}}{{\mathrm{X}}_{\mathrm{n}}})\le {(\frac{24}{116})}^3,$

where ${\mathrm{X}}_{\mathrm{n}}$ is the tristimulus value of a perfect white object color stimulus, which the light reflected from a perfect diffuser under the chosen illuminant. ${\mathrm{F}}_{\mathrm{Y}}$ and ${\mathrm{F}}_{\mathrm{Z}}$ values are calculated in the same way as ${\mathrm{F}}_{\mathrm{X}}$.

3.2. Pan-Sharpening

In the Pan sharpening procedure, the MS image should be resampled to the same pixel size of the Pan image before converting it to CIELab color space. In this research, the bicubic interpolation method was used to resample 2 m resolution Pleiades MS images into 50 cm resolution to match the pixel size of Pleiades Pan image. This resampled dataset is used in all Pan sharpening methods used in this research, including the proposed one. After converting the MS image from RGB to CIELab space, the Pan sharpening process continues with replacing the Pan image with the L* component. Unlike the proposed method in [29] study, there is no need for color space conversion of Pan image in the proposed method, which leads to low computation and less data distortion. Before replacing the L* band of MS with the Pan image, there is a histogram matching step that could be considered as preprocessing step. After resampling the MS image to the same size of Pan image and converting the MS image color space, the histogram of Pan image has to be matched with the histogram of L* component in order to minimize the spectral differences [42]. For performing histogram matching task, mean and standard deviation normalizations were used [43]:

(8)$Pa{n}^{HM}=(Pan-{\mu }_{Pan})\frac{{\sigma }_I}{{\sigma }_{Pan}}+{\mu }_I,$

where $Pa{n}^{HM}$ stands for histogram matched Pan image, $\mu$ stands for mean and $\sigma$ represents standad deviation. After these preprocessing steps, the L* component is replaced with a Pan image. The Pan sharpened image is then produced by implementing inverse conversion of CIELab color system on the Pan*a*b* image and results in a new MS image with high spatial resolution.

3.3. Accuracy Assessment

Several metrics were proposed to assess the accuracy of Pan-sharpened images that use the precise, high-resolution MS image as a reference image. In this research, the first seven metrics provided in Table 2 were used for the spectral quality assessment, while the later four metrics were used for spatial quality assessment of the results. Although metrics provide important quantitative insights about the algorithm performance, qualitative assessment of the color preservation quality and spatial improvements in object representation is required. Thus, results obtained from the Pan sharpening algorithms were also evaluated with visual inspection.

4. Experimental Results

4.1. Dataset

The primary product of Pléiades satellite images were used for performing experimental analysis of the proposed method. The Pléiades program, which was launched by CNES (the French space agency), is the optical Earth imaging component of French–Italian ORFEO (Optical and Radar Federated Earth Observation). The Pléiades constellation consists of two satellites with VHR optical sensors. The Pléiades 1A launched on 17.12.2011 and Pléiades 1B launched on 2.12.2012. Both of the satellites provide 0.5 m spatial resolution for the Pan sensor and 2 m for MS sensor with 20 km swath width and 12 bit radiometric resolution [44].

The dataset used in this research consists of three Pleiades image scenes, which cover different landscape characteristics (Table 3). The locations of scenes are provided in Figure 3. Ten different sub frames were selected from these image scenes, in order to evaluate the performance of Pan sharpening methods for varying landscape characteristics and seasonal conditions (Table 4).

Four sub frames cover rural areas that contain forests with different types of trees with varying heights and barren roads between them, and mountains and agricultural fields. Three sub frames cover urban areas that include complex buildings, roads and highways. Finally, three sub frames cover sub-urban areas that include all the complex buildings, factories, vegetation, roads, trees and bare soil parts to perform a precise survey on effects of the proposed method in Pan sharpening of vegetation, impervious and soil surfaces simultaneously.

4.2. Performing the Algorithm and Accuracy Assessment

To measure the performance of Pan sharpening results using the metrics that were presented in Section 3.3, the Wald protocol was used, due to lack of reference a high-resolution MS image [53]. According to Wald protocol, all Pan sharpening experiments were done using degraded datasets, which are produced by decreasing spatial resolution of the original dataset (reduce MS and Pan, respectively, to 8 m and 2 m). The Pan sharpening results obtained that way, can be compared with the original MS images for an accuracy assessment procedure. In this paper, six Pan sharpening methods and eleven accuracy indexes are evaluated to perform a comparative accuracy assessment of the proposed method. The numerical results of the accuracy indexes are presented in Table A1, Table A2, Table A3, Table A4, Table A5 and Table A6. The visuals belonging to Pan sharpening results of ten frames are presented in Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6, Figure A7, Figure A8, Figure A9 and Figure A10. In each figure, parts a and b are the original Pléiades MS and Pan images, respectively. Parts c, d and e are the Pan sharpened results from the CIElab, GIHS and GS methods, respectively. The Pan sharpened images from the HCS, IHS, NNDiffuse and Ehlers methods are shown in parts f–i, respectively.

4.3. Experimental Results from Rural Areas

Figure A1, Figure A2, Figure A3 and Figure A4 belongs the Pan sharpening results of the rural test sites (frame F1 from D1 dataset, frames F5 and F6 from D2 and Frame F10 from D3 datasets). Each figure belongs to a representative part from the whole image focusing on rural areas and presents visual comparison different Pan sharpening techniques.

The visual comparison of the Pan sharpening methods reveals that spatial resolution of MS images improved significantly in all methods. As for spectral information, parts c, e and h show that the CIELab GS and NNDiffuse methods protect the spectral characteristics better; specifically, for the bands belonging to the visible region. The color-based visual interpretation in vegetated and forest areas in Figure A2, Figure A3 and Figure A4 inform us that the Pan-sharpened and original MS images are very similar to each other for GS and CIELab methods. Similar comments can be made on NNDiffuse and CIELab methods in Figure A1. On the other hand, visual comparison of part a with parts d, f, g and i reveals that the remaining four methods were not able to preserve the spectral characteristics of vegetated and forest areas. Particularly, IHS, Ehlers and HCS methods inherited the high frequency impact over vegetated area and could not preserve original spectral/color information for the first test site. In addition, the result of the HCS method is more blurred than the others. The GIHS and—in some cases—the NNDiffuse methods, preserved the color information better than the IHS, Ehlers and HCS; nevertheless, observable spectral distortion is apparent in their resulting products. The GS method has good performance in the case of vegetation except Figure A1 part e, while results are not satisfactory in pathways and their surroundings. In addition, obvious distortions are apparent in the shadowed areas. Detailed investigation on Figure A3 (e) reveals that there is an obvious distortion in snowy parts of the frame almost in all methods except the proposed CIELab, which resulted in a nearly blue color instead of white snow color. However, CIELab method could be able to preserve the texture and keep the small variances in the color when compared to original MS image. Besides, visual interpretation of the CIELab Pan sharpening results (part c in all figures) demonstrated that use of this color space for Pan sharpening could help to distinguish different tree types and vegetation from each other in the absence of NIR band.

The seven quality metrics, which were presented in Table 2, were used for spectral quality assessment of the Pan sharpening results. Numerical results from these metrics for the rural frames (F1, F5, F6 and F10) are presented in Table A1. The metric values were calculated band by band, and the average values of three bands were used for the accuracy assessment procedure. Numerical results of ERGAS, RASE, RMSE and SAM metrics indicated that the proposed CIELab method produced better results than the remaining methods and was followed by the GS method for most of the metrics. Metric-based results were in line with the visual interpretation. The CIELab method also provided the highest accuracies according to the QAVG and PSNR metrics. In addition, the proposed method provides the value 1 for the SSIM metric, which is the best possible value. Moreover, the IHS method provides worst results for all quality indexes, with respect to Table A1. Lastly, Ehlers, GIHS and HCS methods provide lower accuracies in some cases. This unstable manner of these methods across different scenes is another problem that should be considered.

To assess the spatial quality of Pan sharpened images, the CC, the Zhou index, Sobel RMSE and spatial ERGAS indexes that are presented in Section 3.3, were calculated by comparing the Pan image and the intensity component of the Pan sharpened images. Numerical results of these metrics are presented in Table A2. According to comparative evaluation, the Pan sharpened image from the proposed CIELab method provided the highest spatial CC and Zhou values and lowest SRMSE and SP ERGAS values. These results indicate that the proposed method has the best spatial performance among all methods tested. Ehlers, HIS and HCS methods provided the lowest spatial performances according to the values presented in Table A2.

As a result, the proposed CIELab method provided the best performance for the rural scenes based on the visual interpretation and spectral and spatial quality metrics results.

4.4. Experimental Results from Urban Areas

Figure A5 through Figure A7 belongs to the Pan sharpening results of the urban test sites (frames F2, F4 and F9 from D1, D2 and D3 datasets respectively). Each figure belongs to a representative part from the whole image focusing on the buildings and roads, and presents visual comparison between different Pan sharpening techniques on the differently sized and oriented buildings and roads in the urban areas.

Visual comparison results of urban areas revealed that all the Pan sharpened images inherited the high spatial information from the Pan image, and likewise, the results of rural areas. Roads and buildings could be better identified in all Pan sharpened images compared to original MS image. As for spectral information, Figure A5 c,e,h, informed us that CIELab, GS and NNDiffuse methods preserved the spectral characteristics and color information in urban areas. In particular, the color information from the buildings with brick rooves are similar to the original MS image. Visual comparison of Figure A5 part a with parts d, f, g and i illustrated that of IHS, HCS, GIHS and Ehlers methods are not able to preserve the original spectral characteristics of buildings as well as the other three approaches did. In particular, Ehlers, HCS and IHS methods provided blurred and smoggy results with faded and paled colors. Parts g and I from Figure A6 and Figure A7 support that HIS and Ehlers methods provide worst visual results among all methods tested. Part e in Figure A6 and Figure A7 reveals the weakest side of GS method; that is, the poor performance in the Pan sharpening of white tones. White colors tend to seem blueish in results of this method. It is obvious from part f in Figure A6 and Figure A7 that the HCS method provided the most blurred result. GIHS and NNDiffuse methods have acceptable results in comparison with the results from other methods (except CIELab method). Detailed investigation of Figure A6 and Figure A7, parts d and h, prove that NNDiffuse method produces distortion in the shadowed areas and GIHS method has poor performance in vegetated areas and trees. Visual interpretation of Figure A5(c), Figure A6(c) and Figure A7(c) reveal the fact that the proposed CIELab method protected spectral properties of original MS image more than the other methods.

Numerical results of spectral quality assessment of Pan sharpened images belonging to the urban test sites (F2, F4 and F9) are presented in Table A3. Metric values demonstrated that the CIELab method provided the most promising results among all Pan sharpening methods used in this research. This method presented the lowest values for the ERGAS, RASE, RMSE and SAM metrics and highest values for QAVG, PSNR and SSIM metrics (again, the highest possible value obtained for SSIM). HCS and IHS methods provided the worst results for most of the metrics. Once again, the second performance rank for spectral quality was obtained by GS method in most of the metrics.

Table A4 presents the spatial quality metrics results that were calculated from Pan image and the intensity component of Pan sharpened images for the urban test sites. Similar to the rural test sites, the proposed CIELab method provided the highest CC and Zhou values alongside of lowest SRMSE and SP ERGAS values for urban images, which demonstrated the high spatial quality. In particular, there is great gap between the numeric results of SRMSE and SP ERGAS indexes presented with CIELab method and other methods. Ehlers, IHS and HCS methods acted as the worst methods in the case of spatial indexes, which is consistent with the visual results. Consequently, the proposed CIELab method provided the best performance for the urban test sites (frames F2, F4 and F9) as well, based on the visual interpretation and spectral/spatial quality metrics.

4.5. Experimental Results from Suburban Areas

Figure A8, Figure A9 and Figure A10 present the representative portions of the original images and Pan sharpening results of the suburban test sites from F3, F7 and F8 frames respectively.

Similar to the urban and rural areas, visual comparison of original MS and Pan sharpened images of this category revealed that all the Pan sharpened images produced higher spatial information than original MS image and benefited from Pan image detail level. However, the visual performance of suburban areas was variable, unlike the urban and rural areas. Results from frame F3 (Figure A8) show that all methods had acceptable performance except Ehlers and HIS. Nevertheless, small amount of distortion in vegetation and shadowed areas is apparent in the results of NNDiffuse and HCS methods.

Figure A9 illustrates the effectiveness of the CIELab method in Pan sharpening process. There is an obvious color distortion in all methods except proposed Lab method’s result. Parts e, g and i demonstrate similar distortion in the results of Ehlers, GS and HIS methods with a green dominant color distortion, while other three methods, which are presented in parts d, f and h, have purple dominant color distortion. These color distortions are apparent for all surface types including roads, rooves and other objects. The CIELab was the only method that provided acceptable performance for this test frame. Ehlers and IHS methods could not provide good performance for the last test frame, as is observable in parts c and g of the Figure A10. Parts d and f prove that the results of the GIHS and HCS are blurred and not acceptable. Green and white tones distortion is obvious in the result of GS (Part e from Figure A10). The CIELab method illustrates the best performance again in this frame. Regardless of the distortion in shadowed areas, NNDiffuse provided most similar results to original MS image after the CIELab results.

Numerical results of spectral quality assessment of Pan sharpened images belonging to suburban areas (frames F3, F7 and F8) are presented in Table A5. Once again, CIELab method provides the highest values for the QAVG, PSNR and SSIM metrics, while it achieves the lowest values for the ERGAS, RASE, RMSE and SAM metrics. The GS method has the second place again, similar to the urban and rural test sites by achieving better numeric values for most of the metrics. Similar to previous test site results, the Ehlers and IHS methods have the worst performance between all tested methods.

Spatial quality assessment of Pan sharpening results for the frames F3, F7 and F8 are presented in Table A6. The proposed CIELab method illustrated an unrivaled performance in the case of the spatial quality metrics. For the CC and Zhou index, the highest correlation values that indicate high spatial quality are provided by the CIELab method. Moreover, the lowest SRMSE and SP ERGAS values achieved by the proposed method are another proof of high spatial quality of this method. Ehlers, HCS and IHS methods present the lowest CC and Zhou values with the highest SRMSE and SP ERGAS values, which indicate the poor spatial performances of these methods, like their spectral performances.

4.6. Thematic Accuracy Evaluation with Spectral Index

The performance of the conventional and proposed Pan sharpening methods were evaluated with several quality indices and visual interpretation in this research. However, the effects of Pan sharpening on the information extraction, process such as image classification, segmentation and index-based analysis is another important concern that requires conservation of spectral properties of the MS image after Pan sharpening. One of the indirect methods frequently used to evaluate the abovementioned situation is to apply the spectral index on the MS and Pan sharpened images, and investigate their consistency. As only visible bands of the images were used in this research, the visible atmospherically resistant index (VARI) proposed by Gitelson et al., 2002 [54] was used for the evaluation as it uses the all visible bands for calculation.

(9) $VARI=\frac{Green-Red}{Green+Red-Blue}.$

The test site F3 was selected for this evaluation, as it is one of the most challenging sites in the dataset due to complex and heterogeneous land cover characteristics. The VARI index was applied on both the MS and CIELab Pan sharpened images, and a binary classification was performed with the use of the same threshold to map the manmade and natural lands in the region. According to results presented in Figure 4, same level of information extraction could be achieved with CIELab Pan sharpened image and it even provided better thematic representation by providing better geometric representations of the objects and less of the salt and pepper effect observed in the vegetated areas located in the north and south parts of the image.

4.7. Overall Comments

When the numerical values from the seven spectral metrics for three different test sites (Table A1, Table A3 and Table A5) were evaluated, the proposed CIELab had a consistent behavior for different metrics and for different land categories, and ranked as the first among all methods. On the other hand, for the other Pan sharpening methods, different metrics provided various accuracies and did not show a consistent manner for different metrics and even, for different images, the same metric. As an example, the GS method generally had the second ranking for the ERGAS metric for different test sites. However, in the case of the RASE metric, the GS method had second ranking just for some images, while the GIHS method took the second rank for the remaining images. This phenomenon is similar for the worse results; there is no one method that can be mentioned as the worst for all test sites and all metrics. All facts about the consistent manner of the proposed method can also be asserted for the spatial metrics. The CIELab method had the best spatial performance for the all ten test sites, while second through seventh rankings were variable across different metrics and different images. As an outcome, it is evident that the proposed method presents the best results considering spectral and spatial quality metrics and visual interpretation for ten different sites having different landscape characteristics. Moreover, it provided efficient spectral conservation performance according to comparative evaluation performed with binary classification of VARI index. An overall ranking is provided in Table 5, according to expert judgement by considering the quantitative, metric-based results and visual interpretation results together for different landscapes across spectral and spatial domains. Lastly, in order to check the consistency of spectral quality indices through each band of the images, these indices were calculated band-by-band for the test site F3 (Table A7). According to this evaluation, the averaged values for each index are in accordance with the band-based calculations and indices, providing consistent characteristics across image bands in most of the cases.

5. Conclusions

This research proposed an effective, component substitution-based image Pan sharpening method that uses CIELab color space for Pan sharpening of the VHR Pléiades satellite images. Ten test sites with different landscape characteristics were selected to evaluate the performance of the proposed method in conjunction with six common Pan sharpening algorithms; namely, GS, HCS, IHS, EHLERS, NNDiffuse and GIHS. The comparative evaluation results from Pléiades VHR images supports that the proposed CS algorithm is powerful and ensures better performance compared to the other Pan sharpening methods according to the spectral and spatial accuracy assessment procedures and the visual interpretation. In addition, results indicated that proposed method provided comparatively consistent results, while the performance of other methods varyied with respect to land surface characteristics of the region. As an example for RMSE metric, the best values among the all ten sites were obtained for forest and vegetated areas. Pan sharpening in urban areas resulted in coarser metric values, which illustrate the impact of different land characteristics on the performance of Pan sharpening algorithms. Characteristics of unique CIELab color space, led to producing similar brightness characteristics in Pan sharpened images compared to original MS image. The nature of L* component of MS image helps to preserve spectral and spatial information of original MS and Pan images, respectively. Further improvement of the CIELab-based method could be the implementation of this approach for Pan sharpening of satellite images with more than three bands. In addition, further studies are planned to evaluate the performance of CIELab in fusions of satellite images from different sources. Lastly, other accuracy assessment approaches, such as comparisons of classification and segmentation results of Pan sharpened images, could also help future investigations.

Author Contributions

conceptualization, A.R., U.A. and C.G.; methodology, A.R., U.A. and C.G.; formal analysis, A.R.; investigation, A.R.; data curation, A.R.; writing—original draft preparation, A.R., U.A. and C.G.; writing—review and editing, A.R. and U.A.; visualization, A.R.; supervision, U.A. and C.G.; project administration, C.G.

Funding

This research received no external funding.

Acknowledgments

Great appreciation to ITU CSCRS for providing VHR “Pléiades” images. The authors would also like to acknowledge the many useful contributions of Elif Sertel.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Figure A1

Result and comparison of the proposed Pan sharpening method for the F1 (zoomed) area, which is represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A2

Result and comparison of the proposed Pan sharpening method for the F5 area (zoomed), which is represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A3

Result and comparison of the proposed Pan sharpening method for the F6 area (zoomed), which is represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A4

Result and comparison of the proposed Pan sharpening method for the F6 area (zoomed), which is represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Table A1

Numeric results of spectral quality metrics of the Pan-sharpened images produced by selected algorithms for rural test sites (blue: highest accuracy; red: lowest accuracy).

		ERGAS	QAVG	RASE	RMSE	SAM	PSNR	SSIM
F1	Ehlers	8.35973	0.70345	31.03779	0.00169	3.64069	55.45895	0.99423
	GS	3.53280	0.72884	12.85721	0.00070	2.17343	63.81712	0.99854
	GIHS	9.70735	0.68332	54.94083	0.00299	3.71061	50.49886	0.96170
	HCS	3.30104	0.72804	13.12935	0.00071	3.07602	62.93191	0.99913
	IHS	10.74394	0.67747	60.82506	0.00331	3.74898	49.61512	0.95743
	CIELab	2.07704	0.83561	11.24036	0.00061	1.33832	69.28116	1
	NNDiffuse	6.01008	0.68762	29.32491	0.00159	1.68020	55.95203	0.99065
F5	Ehlers	7.62863	0.67932	30.11675	0.00136	2.76553	57.32630	0.99653
	GS	3.43803	0.70832	15.06442	0.00069	1.96874	64.58066	0.99909
	GIHS	4.68240	0.70403	20.21354	0.00091	1.71451	60.78961	0.99738
	HCS	6.04601	0.67023	24.66611	0.00111	1.64450	59.06045	0.98721
	IHS	7.58878	0.66419	32.80574	0.00148	2.31403	56.58346	0.98436
	CIELab	3.1501	0.81033	12.37956	0.00056	1.25643	75.04835	1
	NNDiffuse	6.05444	0.69039	25.02772	0.00117	1.64355	59.53192	0.99658
F6	Ehlers	12.35511	0.68485	44.42476	0.00133	5.60838	57.50393	0.99650
	GS	4.95180	0.73240	18.30183	0.00059	2.46646	66.21170	0.99918
	GIHS	8.96652	0.71253	36.05239	0.00108	3.84187	59.31775	0.99551
	HCS	5.56829	0.71320	22.52386	0.00068	3.70496	63.40357	0.98920
	IHS	12.47236	0.68690	50.19509	0.00151	5.41973	56.44320	0.98278
	CIELab	4.53026	0.83580	16.05962	0.00048	1.86814	66.34172	1
	NNDiffuse	7.74954	0.69166	32.23618	0.00097	2.70472	60.28956	0.99680
F10	Ehlers	9.22107	0.61404	36.48889	0.00316	4.44999	49.99698	0.98567
	GS	4.78197	0.63099	18.07030	0.00157	2.86307	56.10088	0.99517
	GIHS	6.60284	0.62945	32.99550	0.00286	2.41971	50.87109	0.97682
	HCS	9.96965	0.57631	40.17887	0.00348	2.10395	49.16024	0.98781
	IHS	9.49649	0.60973	47.55454	0.00412	3.41879	47.69635	0.96361
	CIELab	4.3611	0.6478	17.14994	0.00149	1.77264	57.55494	1
	NNDiffuse	4.46541	0.63385	17.77395	0.00154	1.50089	56.24451	0.99540

Table A2

Numeric results of spatial quality metrics of the Pan-sharpened images produced by select algorithms for the rural test sites (blue: highest accuracy; red: lowest accuracy).

		CC	Zhou’s SP	SRMSE	SP ERGAS
F1	Ehlers	0.876695	0.939608	0.006527	26.540923
	GS	0.970317	0.973490	0.002298	25.222656
	GIHS	0.978149	0.979959	0.005213	25.699874
	HCS	0.939252	0.963691	0.00293	25.531640
	IHS	0.869487	0.929827	0.006592	26.040642
	CIELab	0.982436	0.998617	5.19E-08	17.345011
	NNDiffuse	0.944022	0.967511	0.003237	24.580730
F5	Ehlers	0.926486	0.92948	0.006241	26.658944
	GS	0.989994	0.98932	0.003323	25.503829
	GIHS	0.990875	0.989946	0.004153	25.881689
	HCS	0.869709	0.914225	0.00467	26.457399
	IHS	0.927104	0.989560	0.006261	26.460988
	CIELab	0.996861	0.994414	2.22E-08	4.9495726
	NNDiffuse	0.971652	0.989410	0.004303	25.740781
F6	Ehlers	0.891415	0.929872	0.006123	29.084361
	GS	0.989992	0.989450	0.002473	26.047215
	GIHS	0.989997	0.989974	0.004425	28.173902
	HCS	0.912241	0.972654	0.002903	26.946861
	IHS	0.95928	0.969886	0.006141	29.301514
	CIELab	0.997273	0.996194	3.40E-08	11.968807
	NNDiffuse	0.969999	0.979946	0.003062	27.901263
F10	Ehlers	0.923442	0.919671	0.014214	28.001920
	GS	0.989619	0.989061	0.008828	26.488351
	GIHS	0.971429	0.969959	0.008738	26.263696
	HCS	0.799403	0.694767	0.012855	28.756548
	IHS	0.914264	0.929730	0.014267	27.019798
	CIELab	0.997258	0.996956	2.69E-08	8.0781533
	NNDiffuse	0.929999	0.929405	0.0067663	26.362095

Figure A5

Result and comparison of the proposed Pan sharpening method for the F2 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A6

Result and comparison of the proposed Pan sharpening method for the F4 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A7

Result and comparison of the proposed Pan sharpening method for the F9 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Table A3

Numeric results of spectral quality metrics of the Pan-sharpened images produced by select algorithms for urban test sites (blue: highest accuracy; red: lowest accuracy).

		ERGAS	QAVG	RASE	RMSE	SAM	PSNR	SSIM
F2	Ehlers	11.86921	0.53991	47.33854	0.00401	4.71053	47.94083	0.97725
	GS	5.23770	0.58263	22.61594	0.00195	2.64855	55.16107	0.99458
	GIHS	7.34063	0.57639	33.87054	0.00287	2.46540	50.84869	0.98449
	HCS	15.71881	0.48541	62.43752	0.00529	2.73988	45.53622	0.97461
	IHS	11.04969	0.54971	51.03739	0.00432	3.46087	47.28736	0.95942
	CIELab	4.22351	0.68629	20.75741	0.00176	1.36201	65.10167	1
	NNDiffuse	9.43220	0.56867	49.55085	0.00420	3.40818	47.54411	0.96330
F4	Ehlers	22.84143	0.52356	89.68352	0.00405	7.95294	47.84821	0.95598
	GS	12.57253	0.42386	65.56001	0.00296	6.13381	50.56968	0.96821
	GIHS	15.81385	0.63109	90.10525	0.00407	3.15646	47.80746	0.94837
	HCS	23.98628	0.28239	121.71057	0.00550	8.28181	45.19589	0.93994
	IHS	19.50133	0.38961	111.13086	0.00502	8.99373	45.98577	0.92838
	CIELab	6.86913	0.70144	26.93926	0.00122	1.89738	58.29474	1
	NNDiffuse	19.72400	0.63077	98.22609	0.00441	6.13952	50.53176	0.97519
F9	Ehlers	9.89595	0.51189	39.61836	0.00495	4.27519	46.10242	0.97015
	GS	4.96528	0.55533	18.29249	0.00256	2.20216	51.30320	0.99273
	GIHS	5.40401	0.53562	24.49466	0.00306	1.92496	50.27892	0.98486
	HCS	10.02627	0.47659	39.67119	0.00496	2.02550	46.09085	0.93648
	IHS	9.55051	0.51220	43.30832	0.00541	3.25101	45.32893	0.92951
	CIELab	4.2936	0.6402	17.16686	0.00215	1.12493	54.36654	1
	NNDiffuse	8.25412	0.58294	18.91789	0.01112	1.80691	39.08057	0.94984

Table A4

Numeric results of spatial quality metrics of the Pan-sharpened images produced by select algorithms for urban test sites (blue: highest accuracy; red: lowest accuracy).

		CC	Zhou’s SP	SRMSE	SP ERGAS
F2	Ehlers	0.923124	0.939587	0.027718	28.484374
	GS	0.985504	0.968697	0.008643	25.816466
	GIHS	0.986114	0.989954	0.012513	26.611553
	HCS	0.937058	0.949630	0.019802	21.018918
	IHS	0.913914	0.929653	0.027772	27.734556
	CIELab	0.996129	0.995455	7.17E-08	6.2709053
	NNDiffuse	0.960443	0.961942	0.012853	22.490210
F4	Ehlers	0.820014	0.939655	0.017949	31.953533
	GS	0.989999	0.989842	0.009369	26.949381
	GIHS	0.992001	0.989953	0.012168	28.452035
	HCS	0.775576	0.688826	0.018171	33.941348
	IHS	0.919941	0.989666	0.018000	31.259808
	CIELab	0.997280	0.996187	5.44E-08	5.8158174
	NNDiffuse	0.914590	0.888001	0.012253	27.737860
F9	Ehlers	0.924923	0.949622	0.024424	28.179705
	GS	0.989237	0.982593	0.014115	26.132656
	GIHS	0.978923	0.989960	0.015724	26.434713
	HCS	0.860205	0.794738	0.023803	28.880067
	IHS	0.912546	0.969662	0.024479	27.533331
	CIELab	0.99848	0.998002	5.26E-08	4.4340045
	NNDiffuse	0.929067	0.958606	0.021917	25.926934

Figure A8

Result and comparison of the proposed Pan sharpening method for the F3 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A9

Result and comparison of the proposed Pan sharpening method for the F7 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Figure A10

Result and comparison of the proposed Pan sharpening method for the F8 test frame, zoomed areas, which are represented in a true color (RGB) combination. (a) MS; (b) Pan; (c) Lab; (d) GIHS; (e) GS; (f) HCS; (g) IHS; (h) NNDiffuse; (i) Ehlers.

[Figure omitted. See PDF]

Table A5

Numeric results of spectral quality metrics of the Pan-sharpened images produced by select algorithms for suburban test sites.

		ERGAS	QAVG	RASE	RMSE	SAM	PSNR	SSIM
F3	Ehlers	10.98399	0.63217	43.08140	0.00301	4.48411	50.43809	0.98554
	GS	5.51194	0.66128	23.69334	0.00154	2.84298	56.80727	0.99542
	GIHS	6.97787	0.65746	33.45619	0.00233	2.40080	52.63436	0.98413
	HCS	9.58868	0.62070	38.29194	0.00267	2.67967	51.46174	0.99176
	IHS	10.61170	0.63204	50.97203	0.00356	3.52966	48.97725	0.97134
	CIELab	4.14506	0.77127	20.06662	0.00140	1.84331	67.07440	1
	NNDiffuse	6.93153	0.68589	30.69262	0.00214	2.67079	53.38321	0.99047
F7	Ehlers	9.66602	0.58842	38.60687	0.00252	5.34836	51.98700	0.98848
	GS	4.43979	0.64207	28.36296	0.00183	2.15986	58.92781	0.99695
	GIHS	4.86275	0.61769	19.90348	0.00130	2.39510	57.74171	0.99608
	HCS	6.21293	0.59860	24.68755	0.00161	6.57336	55.87073	0.99479
	IHS	9.99170	0.58440	40.96218	0.00267	5.54742	51.47263	0.98461
	CIELab	3.3780	0.72134	17.43123	0.00114	1.88890	68.89373	1
	NNDiffuse	6.00374	0.60227	28.30174	0.00184	2.87237	54.68403	0.98907
F8	Ehlers	7.61492	0.68151	29.46707	0.00217	3.58878	53.26597	0.99194
	GS	4.02694	0.70121	14.48671	0.00107	2.77359	59.43332	0.99737
	GIHS	5.76109	0.70009	27.24384	0.00201	2.20337	53.94734	0.98655
	HCS	6.82371	0.68186	27.87873	0.00205	1.32168	53.74725	0.99579
	IHS	8.13416	0.68201	38.54293	0.00284	2.97812	50.93381	0.97920
	CIELab	3.5084	0.7133	13.48278	0.00099	1.20401	60.05712	1
	NNDiffuse	5.70992	0.69899	14.95033	0.00110	1.31369	59.15970	0.99740

Table A6

Numeric results of spatial quality metrics of the Pan-sharpened images produced by select algorithms for suburban test sites.

		CC	Zhou’s SP	SRMSE	SP ERGAS
F3	Ehlers	0.922299	0.949657	0.012645	28.478359
	GS	0.989999	0.979878	0.007121	26.395064
	GIHS	0.988664	0.989955	0.00808	26.561586
	HCS	0.942661	0.979506	0.01034	26.580025
	IHS	0.920211	0.929723	0.012687	27.542710
	CIELab	0.996151	0.996340	3.22E-08	7.7739934
	NNDiffuse	0.940385	0.972273	0.00683	26.669404
F7	Ehlers	0.940410	0.929782	0.013524	30.304598
	GSc	0.989997	0.989956	0.008056	28.497749
	GIHS	0.973639	0.978996	0.008367	29.121990
	HCS	0.968484	0.962481	0.009486	29.539885
	IHS	0.939801	0.964973	0.013591	30.416119
	CIELab	0.998630	0.996223	1.66E-08	3.1299142
	NNDiffuse	0.996187	0.989929	0.008685	28.273530
F8	Ehlers	0.918887	0.919399	0.009181	26.935703
	GS	0.989998	0.989753	0.005542	25.948199
	GIHS	0.972152	0.989942	0.005759	25.943732
	HCS	0.800374	0.700073	0.006954	27.084421
	IHS	0.921523	0.995543	0.009236	26.447253
	CIELab	0.997412	0.997655	2.26E-08	8.2907935
	NNDiffuse	0.989999	0.969353	0.004198	26.011647

Table A7

Band-by-band calculation results of spectral quality metrics belonging to test site F3.

Method	Band	Ehlers	GC	GIHS	HCS	IHS	CIELab	NNDIF
ERGAS	Band 1	13.23542	6.28494	7.53986	11.51714	14.68644	6.19922	9.43229
	Band 2	10.86379	5.49824	6.73039	10.62880	10.07883	4.07797	7.66444
	Band 3	6.52447	4.08371	4.97720	5.58341	7.75358	2.07075	2.81246
	Average	10.98399	5.51194	6.97787	9.58868	10.6117	4.14506	6.93153
QAVG	Band 1	0.86547	1.02826	0.95928	1.02387	0.87788	1.0607	1.03723
	Band 2	0.57607	0.65720	0.69449	0.52406	0.56826	0.70848	0.60802
	Band 3	0.48901	0.30277	0.36560	0.31617	0.45130	0.54621	0.41064
	Average	0.63217	0.66128	0.65746	0.62070	0.63204	0.77127	0.68589
RASE	Band 1	49.91752	30.04186	39.39898	42.79105	54.69452	25.33830	34.82911
	Band 2	46.79656	22.35778	35.26309	37.59033	52.26805	21.10608	31.65890
	Band 3	34.19874	18.59234	25.44349	36.19742	45.77826	13.75556	25.58183
	Average	43.08140	23.69334	33.45619	38.29194	50.97203	20.06662	30.69262
RMSE	Band 1	0.00833	0.00450	0.00525	0.00625	0.01033	0.00413	0.00525
	Band 2	0.00040	0.00010	0.00186	0.00156	0.00042	0.00008	0.00106
	Band 3	0.00031	0.00002	0.00001	0.00009	0.00001	0.00001	0.00009
	Average	0.00301	0.00154	0.00233	0.00267	0.00356	0.00140	0.00214
SAM	Band 1	4.86237	3.61186	3.40619	3.10669	4.08712	2.22633	3.13166
	Band 2	4.42003	2.42682	2.50814	2.87424	3.16997	1.82082	2.55769
	Band 3	3.95193	2.35714	1.78239	2.20364	3.05983	1.53200	2.21219
	Average	4.48411	2.84298	2.4008	2.67967	3.52966	1.84331	2.67079
PSNR	Band 1	56.42695	60.52591	54.16646	54.15527	51.81649	69.90153	57.27461
	Band 2	48.08904	59.45342	51.82179	51.08538	47.59585	67.71639	57.20947
	Band 3	47.47481	50.38406	50.63450	49.28219	46.97046	65.36066	45.98168
	Average	50.43809	56.80727	52.63436	51.46174	48.97725	67.07440	53.383210
SSIM	Band 1	0.99888	0.99702	0.99417	0.99251	0.98071	1	0.99051
	Band 2	0.98443	0.99539	0.98407	0.99175	0.97231	1	0.99048
	Band 3	0.97331	0.99385	0.97415	0.99108	0.96142	0.9999	0.99042
	Average	0.98554	0.99542	0.98413	0.99176	0.97134	1	0.99047

List A1. Definition of Terms in Table 2

RMSE: MN is the image size, $\mathrm{PS}(i,j)$ and $\mathrm{MS}(i,j)$ represent pixel digital number (DN) at $(i,j)$’th position of Pan-sharpened and MS image.

ERGAS:$\frac{dh}{dl}$ represents the ratio between the pixel size of high resolution and low resolution images; e.g., ¼ for Pléiades data, and $n$ number of bands. The RMSE represents root mean square error of band $i$.

SAM: The spectral vector ${\mathrm{V}=\text{{}\mathrm{V}}_1{,\mathrm{V}}_2{,\dots ,\mathrm{V}}_{\mathrm{n}}\text{}}$ stands for reference MS image pixels and ${\hat{\mathrm{V}}=\text{{}\hat{\mathrm{V}}}_1{,\hat{\mathrm{V}}}_2{,\dots ,\hat{\mathrm{V}}}_{\mathrm{n}}\text{}}$ stands for Pan-sharpened image pixels rep reference and both have L components.

RASE: The $\mu$ represnts the mean of b^th band; b is the number of bands and $\mathrm{RMSE}$ represents root mean square error.

PSNR: The $\mathrm{L}$ represents the number of gray levels in the image; MN is the image size, ${\mathrm{I}}_{\mathrm{r}}(\mathrm{i},\mathrm{j})$ is pixel value of reference image and ${\mathrm{I}}_{\mathrm{p}}(\mathrm{i},\mathrm{j})$ is the pixel value of Pan-sharpened image. A higher PSNR value indicates more similarity between the reference MS and Pan-sharpened images.

QAVG: The $\overline{x}$ and $\overline{y}$ are the means of reference and Pan-sharpened images, respectively; ${\sigma }_{xy}$ is the covariance and ${\sigma }_x^2$ and ${\sigma }_y^2$ are variances. As QI can only be applied to one band, the average value of three or more bands (QAVG) is used for calculating a global spectral quality index for multi-bands images. QI values range between −1 and 1. A higher value indicates more similarity between reference and Pan-sharpened image.

SSIM: The $\mu$ stands for mean, $\sigma$ stands for standard deviation; $I_r$ and $I_p$ represent reference and Pan-sharpened image respectively. The C1 and C2 are two necessary constants to avoid the index from a division by zero. These constants depend on the dynamic range of the pixel values. A higher value of the measured index shows the better quality of Pan-sharpened algorithm.

CC: C_r,f is the cross-correlation between reference and fused images, while C_r and C_f are the correlation coefficients belonging to reference and fused images respectively.

SRMSE: Edge magnitude (M) is calculated via spectral distance of horizontal and vertical ($M_x$ and $M_y$) edge intensities.

Sp-ERGAS:$\frac{dh}{dl}$ represents the ratio between the pixel size of MS and Pan images, and $n$ is the number of bands. Spatial RMSE is represented as below:

(A1)

$\mathrm{Spatial}\mathrm{RMSE}=\sqrt{\frac{1}{\mathrm{MN}}{{\displaystyle \sum }}_{i=1}^M{{\displaystyle \sum }}_{j=1}^N{(\mathrm{Pan}(i,j)-PS(i,j))}^2},$

where MN is the image size, $\mathrm{PS}(i,j)$ and $\mathrm{Pan}(i,j)$ represents the pixel digital number (DN) at $(i,j)$’th position of Pan-sharpened and Pan image.

Figures and Tables

Figure 1. CIELab image Pan sharpening flowchart.

Figure 2. CIELab color space [41].

Figure 3. The locations of the Pléiades image scenes overlaid on Google Earth©.

Figure 4. Comparison of visible atmospherically resistant index (VARI) index results extracted from multispectral (MS) and CIELab pansharpened images of test site F3. (a) Original MS, (b) threshold applied VARI of MS, (c) zoomed region from VARI of MS, (d) original CIELab, (e) threshold applied VARI of CIELab and (f) zoomed region from VARI of CIELab.

Table 1

The brief review of state of the art Pan sharpening methods.

Method	Description	References
IHS Method	In this system, the total amount of the brightness in one color is represented through intensity channel. The wavelength property of the color and the purity of the color are represented by hue and saturation respectively.	[30,31]
EHLERS (FFT-Enhanced IHS Transform) Method	The fundamental idea of this method is modifying the Pan image in the way that it looks more similar to the intensity component of MS image. This method uses FFT (fast fourier transform) filtering for partial replacement instead of entire replacement of the intensity component.	[32]
NNDiffuse (Nearest-neighbor diffusion-based) Method	This method, considers each pixel spectrum as a weighted linear combination of spectra of its sideward neighboring super pixels in the Pan sharpened image. Algorithm uses various factors like intensity smoothness (σ), spatial smoothness (σ_s) and pixel size ratio for conducting the Pan sharpening	[33,34]
GIHS (Generalized IHS) Method	Directly applying of IHS method needs many multiplication and addition operations, which makes the Pan sharpening operation computationally inefficient. GIHS method develops a computationally efficient Pan sharpening method, which does not require coordinate transformation	[35]
Gram-Schmidt Method	This method uses the Gram-Schmidt orthogonalization for converting the original low-resolution MS bands, which are linearly independent vectors, into a set of orthogonal vectors. The first vector in the orthogonal space is considered as simulated Pan image, which produced by weighted aggregation of the consecutive original MS bands.	[21,22]
Hyperspherical Color Sharpening Method	The Hyperspherical Color Sharpening method (HCS) is a Pan sharpening method designed for WorldView-2 sensor imagery and can be applied to any MS data containing 3bands or more. HCS approach is based on transforming original color space to hyperspherical color space.	[36,37]

Table 2

The description of all accuracy indices (definition of terms provided in List A1).

Quality Metric	Description	Formula	What Value to Look for (Higher/Lower)	Reference
RMSE	Root Mean Square Error (RMSE) is used to calculate the variation in DN values for checking the difference between the Pan sharpened and reference image.	$\mathrm{RMSE}=\sqrt{\frac{1}{\mathrm{MN}}{\displaystyle \sum }_{i=1}^M{\displaystyle \sum }_{j=1}^N{(\mathrm{MS}(i,j)-\mathrm{PS}(i,j))}^2}$	Lower (near to zero)	[45]
ERGAS	Relative dimensionless global error synthesis (ERGAS) is used to calculate the accuracy of Pan sharpened image considering normalized average error of each band of the result image.	$\mathrm{ERGAS}=100\frac{dh}{dl}\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{(\frac{\mathrm{RMSE}(i)}{\mathrm{Mean}(i)})}^2}$	Lower (near to zero)	[45]
SAM	Spectral Angle Mapper (SAM) represents the spectral similarity between the Pan sharpened and reference MS image using the average spectral angle		Lower (near to zero)	[36]
RASE	Relative average spectral error (RASE) is an error index to calculate average performance of Pan sharpening algorithm into spectral bands.	$\mathrm{RASE}=\frac{100}{\mu }\sqrt{\frac{1}{b}{\displaystyle \sum }_{i=1}^{b}RMSE}$	Lower (near to zero)	[46]
PSNR	Peak signal-to-noise ratio is widely used metric for comparison of distorted (Pan sharpened) and original (reference) image.	$\mathrm{PSNR}=20{\log }_{10}\left[\frac{{\mathrm{L}}^2}{\frac{1}{\mathrm{MN}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{M}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{N}}{({\mathrm{I}}_{\mathrm{r}}(\mathrm{i},\mathrm{j})-{\mathrm{I}}_{\mathrm{p}}(\mathrm{i},\mathrm{j}))}^2}\right]$	Higher Value	[47]
QAVG	The Average Quality index based on quality index is used to model the difference between reference and Pan sharpened images as a combination of three different factors: loss of correlation, luminance distortions and contrast distortion. As QI can only be applied to one band, the average value of three or more bands (QAVG) is used for calculating a global spectral quality index for multiband images.	$QI=\frac{4{\sigma }_{xy}\overline{x}\overline{y}}{({\sigma }_x^2+{\sigma }_y^2)\left[{(\overline{x})}^2+{(\overline{y})}^2\right]}$	Higher Value (Close to 1)	[48]
SSIM	Structural Similarity index (SSIM) is a method for measuring the structural similarity between reference and Pan sharpened images. This method compares the local patterns (luminance, contrast and structure) using means and standard deviations of two images.	$SSIM=\frac{(2{\mu }_{I_r}{\mu }_{I_p}+C_1)(2{\sigma }_{I_r{I}_p}+C_2)}{({\mu }^2{}_{I_r}+{\mu }^2{}_{I_p}+C_1)({\sigma }^2{}_{I_r}+{\sigma }^2{}_{I_p}+C_2)}$	Higher Value	[2]
CC	To assess the spatial quality of Pan sharpened images, the correlation coefficient between the Pan image and the intensity component of the Pan sharpened image is used.	$CC= \frac{2C_{r_f}}{C_r+C_f}$	Higher Value (Close to 1)	[49]
ZHOU index	Zhou’s spatial index uses a high frequency Laplacian filter for extracting high frequency information from both Pan and Pan sharpened images. Correlation coefficient is then calculated between filtered Pan image and each band of Pan sharpened image. The average of calculated cc is considered as spatial quality index.	$\mathrm{Laplacian}\mathrm{Kernel}=\left[\begin{array}{ccc}-1& -1& -1\\ -1& 8& -1\\ -1& -1& -1\end{array}\right]$	Higher Value (Close to 1)	[50]
SRMSE	Sobel based RMSE (SRMSE) is an index for spatial accuracy assessment that uses absolute edge magnitude difference of Pan and Pan sharpened image. This index utilizes 3 × 3 vertical and horizontal Sobel filter kernels for calculating the gradient of edge intensities. RMSE then calculated between Pan and Pan sharpened edge magnitude images.	$M=\sqrt{M_x^2+M_y^2}$ Where $M_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]\times$ image And $M_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]\times$ image	Lower (near to zero)	[51]
Sp-ERGAS	Spatial ERGAS (Sp-ERGAS) is an index for spatial quality assessment of Pan sharpened image, which uses spatial RMSE for assessment procedure	$\mathrm{Spatial}\mathrm{ERGAS}=100\frac{dh}{dl}\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{(\frac{\mathrm{Spatial}\text{\_}\mathrm{RMSE}(i)}{\mathrm{Pan}(i)})}^2}$	Lower (near to zero)	[52]

Table 3

The description of three datasets.

No	Location	Acquisition Data	Platform
D1	Istanbul	2017-04-09	PHR 1A
D2	Izmir	2015-12-04	PHR 1B
D3	Aydin	2018-04-10	PHR 1B

Table 4

The description of ten selected data frames.

Image	Location	Type	Objects
F1	Istanbul	Rural	Trees, Vegetation, Bare roads, Bare soil
F2	Istanbul	Urban	Buildings, Roads, Squares, Trees, Bare soil
F3	Istanbul	Suburban	Vegetation, Roads and highways, Buildings, Bare soil, Industry buildings, Green fields.
F4	Izmir	Urban	Buildings, Roads, Vegetation, Bare soil
F5	Izmir	Rural	Agricultural fields, Bare roads
F6	Izmir	Rural	Trees, Mountain, Bare roads
F7	Izmir	Suburban	Industry buildings, Agricultural fields, Roads, Bare soil
F8	Aydin	Suburban	Agricultural fields, Roads, Bare roads, Trees, Buildings, Bare soil
F9	Aydin	Urban	Buildings, Roads, Trees, Bare soil, Green fields
F10	Aydin	Rural	Trees, vegetation, agricultural fields, buildings, roads, Bare roads

Table 5

The overall relative ranking of the methods evaluated.

Method	Rural		Urban		Suburban
	Spectral	Spatial	Spectral	Spatial	Spectral	Spatial
CIELab	Good	Good	Good	Good	Good	Moderate
GIHS	Moderate	Moderate	Moderate	Moderate	Poor	Poor
GS	Good	Good	Moderate	Moderate	Poor	Moderate
HCS	Poor	Moderate	Poor	Poor	Poor	Poor
HIS	Poor	Poor	Poor	Poor	Moderate	Poor
NNDiffuse	Moderate	Moderate	Moderate	Poor	Poor	Moderate
Ehlers	Poor	Poor	Poor	Poor	Moderate	Poor