AIR-POLSAR-CR1.0: A Benchmark Dataset for Cloud Removal in High-Resolution Optical Remote Sensing Images with Fully Polarized SAR

Abstract

Due to the all-time and all-weather characteristics of synthetic aperture radar (SAR) data, they have become an important input for optical image restoration, and various cloud removal datasets based on SAR-optical have been proposed. Currently, the construction of multi-source cloud removal datasets typically employs single-polarization or dual-polarization backscatter SAR feature images, lacking a comprehensive description of target scattering information and polarization characteristics. This paper constructs a high-resolution remote sensing dataset, AIR-POLSAR-CR1.0, based on optical images, backscatter feature images, and polarization feature images using the fully polarimetric synthetic aperture radar (PolSAR) data. The dataset has been manually annotated to provide a foundation for subsequent analyses and processing. Finally, this study performs a performance analysis of typical cloud removal deep learning algorithms based on different categories and cloud coverage on the proposed standard dataset, serving as baseline results for this benchmark. The results of the ablation experiment also demonstrate the effectiveness of the PolSAR data. In summary, AIR-POLSAR-CR1.0 fills the gap in polarization feature images and demonstrates good adaptability for the development of deep learning algorithms.

1. Introduction

Satellite remote sensing data are widely utilized in various applications involving natural resources, the ecological environment, and disaster emergency response due to their wide coverage [1,2,3,4]. However, optical remote sensing images are inevitably affected by cloud cover, which can obscure ground information due to weather constraints. Cloud coverage leads to data loss, rendering the acquired remote sensing imagery insufficient for the requirements of remote sensing research and applications, particularly in quantitative remote sensing and the generation of high-quality surface products [5].

To advance the research on cloud removal, some optical-image-based datasets have first been proposed in recent years. For example, the RICEII dataset [6] includes 450 cloud and cloud-free images, also 512 × 512 pixels, derived from the Landsat-8 OLI/TIRS dataset, with the cloud-free reference images acquired within 15 days of the cloud images. The WHU2-CR dataset [7] generates 17,182 pairs of cloudy and cloud-free Sentinel-2 images globally based on three primary land cover types: urban, vegetation, and bare land, with a spatial resolution of 10 m. The acquisition times for the cloudy and cloud-free images are within 10 days of each other. Huang et al. constructed a new Sentinel-2 satellite image dataset, Sen2_MTC, for training multitemporal cloud removal models [8], where three cloudy images at the same location correspond to one cloud-free image. This dataset contains approximately 50 non-overlapping blocks, with each block consisting of about 70 pairs of cropped 256 × 256 pixel patches, totaling 13,669 images. The spatial resolution is 10 m.

Unlike optical imaging, synthetic aperture radar (SAR), due to its use of longer-wavelength bands, is less affected by atmospheric conditions and can penetrate cloud cover, enabling the capture of ground information regardless of the weather conditions. Consequently, SAR images, as critical prior knowledge, have become focal points of research and are widely utilized in various fields [9,10]. Accordingly, a series of datasets have been produced in deep learning. The SEN1-2 dataset [11] consists of 282,384 pairs of C-band Sentinel-1 VV- and VH-polarized SAR images and Sentinel-2 optical image patches for image fusion. The image in this dataset is cropped to 256 × 256 pixel patches, and the spatial resolution for both Sentinel-1 and Sentinel-2 data has been normalized to 10 m. The SEN12MS dataset [12] contains 180,748 images comprising dual-polarization SAR data from Sentinel-1, multispectral images from Sentinel-2, and MODIS land cover maps. The spatial resolution is also 10 m, while the MODIS land cover image resolution is 500 m, also covering a spatial extent of 256 × 256 pixels. This dataset is widely used in fields such as scene classification. The QXS-SAROPT dataset [13] consists of 20,000 GF-3 single-polarization SAR images and optical images from Google Earth, with a fixed size of 256 × 256 pixels and a spatial resolution of 1 m for image matching. Li et al. introduced a large-scale dataset, WHU-OPT-SAR [14], which comprises 29,400 GF-1 optical images and GF-3 single-polarization SAR images, with a resolution of 5 m, and the dataset combines 30 land cover types in the study area into seven major categories: farmland, forest, urban, rural, water bodies, roads, and others, which can be used for image classification tasks. The dataset is key to training well-integrated models in the field of deep learning. Based on the proposed multi-source dataset, a large number of deep-learning-based algorithms for SAR and optical image fusion [15,16], image matching [17], and semantic segmentation [18] have been proposed, further validating the feasibility of the combined application of optical and SAR data.

Additionally, several multi-source datasets have been proposed in the field of cloud removal. For instance, Gao et al. proposed an airborne dataset [19], which contains 1428 optical RGB and X-band VV-polarized SAR images, with a size of 2813 pixels × 2289 pixels, provided by the 2001 IEEE Geoscience and Remote Sensing Symposium (GRSS) data fusion competition, with a spatial resolution of 1 m. The land cover types mainly include urban areas, roads, and farmland, exhibiting rich textures. Ebel et al. introduced the SEN12MS-CR dataset [20], which contains 122,218 paired Sentinel-1 SAR images, along with cloudy and cloud-free multispectral optical images from Sentinel-2, each with a size of 256 × 256 pixels. This dataset covers global region data across all four seasons of 2018, with approximately 48% of the data affected by cloud cover, making it suitable for multi-modal cloud removal tasks. Czerkawski et al. constructed a cloud removal dataset containing a total of 445 paired Sentinel-1 and Sentinel-2 images from Scotland and India [21], with a resolution of 10 m. The cloudy images in this dataset were synthesized by overlaying cloud masks on the clear images. The SEN12MS-CR-TS dataset is a cloud removal dataset containing 15,578 multimodal and multitemporal global remote sensing images for every season [22]. It consists of 30 registered Sentinel-1 SAR images and Sentinel-2 multispectral images collected at uniform intervals throughout 2018. The spatial resolution of the dataset is 10 m, with an approximate cloud coverage of 50%. The AllClear dataset includes multitemporal data for cloud removal provided by the Google Earth Engine platform [23], featuring Sentinel-2, Sentinel-1, and Landsat 8/9 data. The original images in the dataset have all been resampled to a resolution of 10 m. The dataset encompasses 23,742 global regions of interest collected in 2022, representing various land cover types, and contains a total of 4,354,652 images. Xu et al. sampled 780 spatial locations distributed across continents and meteorological seasons and proposed the M3R-CR dataset for cloud removal [24]. This dataset includes 63,000 pairs of cloudy and clear PlanetScope data, Sentinel-1 images, and corresponding land cover maps, with a resolution of 3 m. These datasets are based on Sentinel-1 and Sentinel-2 images. Therefore, most existing multi-source datasets use single-polarization or dual-polarization backscatter feature images from Sentinel-1 as auxiliary data for Sentinel-2 image restoration. The spatial resolution of these images is 10 m, while the dataset with the highest resolution is shown in Table 1, with a maximum spatial resolution of 3 m.

However, single- or dual-polarization SAR images use a single polarization channel to transmit and receive electromagnetic waves, enabling them to obtain only the scattering characteristics of ground objects for a specific single-polarization electromagnetic wave while losing other components of information related to the ground object that are contained in the scattering wave’s polarization characteristics. The polarization of radar signals can be primarily categorized into linear polarization, circular polarization, and elliptical polarization. The fully polarimetric synthetic aperture radar (PolSAR) is able to transmit and receive electromagnetic waves in both horizontal and vertical directions at the same time, so there are four different polarization modes of HH, HV, VH, and VV, and the complex scattering matrix of the measured target can be obtained, including amplitude, phase, and polarization. By using the scattering matrix, coherence matrix, and covariance matrix provided by the polarization data, the scattering characteristics of ground objects in SAR images, such as entropy, alpha angle, anisotropy, etc., can be further extracted. Therefore, PolSAR has emerged as a new research focus in the field of microwave imaging systems [25] and helps to improve the accuracy of applications such as image reconstruction and image classification [26]. Numerous studies have demonstrated that PolSAR data outperform single-polarization and dual-polarization data in areas like target recognition and land cover classification [27,28,29].

The polarization characteristics of PolSAR images can be derived through polarization decomposition, utilizing polarized scattering information to enhance classification accuracy [30]. These techniques extract various scattering features from the coherent or covariance matrix of PolSAR images, including Pauli decomposition [31], Cloude decomposition [32], Freeman decomposition [33], Krogager decomposition [34], and Yamaguchi four-component decomposition [35,36]. Recently, extensive research has begun to employ deep learning algorithms for land cover classification of PolSAR data, enabling improving the classification accuracy while yielding more stable results.

For example, Zhou et al. [37] applied convolutional networks to image classification, representing PolSAR data with the amplitude of coherence matrix elements as standardized six-dimensional feature vectors. However, the phase information in the covariance/coherence matrix is crucial for distinguishing between different types of scatterers. Consequently, Zhang et al. [38] represented the real and imaginary parts of the diagonal and non-diagonal elements of the polarization coherence matrix as a nine-dimensional vector for input data, achieving an overall accuracy (OA) of 95.29% on the Flevoland dataset. Wang et al. [39] proposed a vision transformer (ViT)-based approach for PolSAR classification. When PolSAR data were represented as a nine-dimensional vector in the coherent matrix form, the OA value reached 98.16% on the Flevoland dataset.

Additionally, numerous studies have been proposed based on the fusion of PolSAR data and optical data. Chu et al. proposed a novel spatial dense channel attention fusion network (SDCAFNet) [40], which integrates feature fusion and semantic segmentation within a neural network utilizing GF-2 and backscattering coefficient images derived from four polarization modes of GF-3. To effectively learn the channel correlations between optical and PolSAR images, they introduced a spatial dense channel attention module in the network. The experimental results demonstrated that the fusion method combining optical and PolSAR images outperformed the single-modal image fusion methods across every metric, with the classification accuracy improving to 74.9%. La et al. [41] conducted experiments based on PALSAR-2 PolSAR data and Sentinel-2 MSI data. They represented the PolSAR data as intensity data for four polarizations, Cloude three-component, Yamaguchi four-component, and nine-dimensional vector data from the coherence matrix. The experimental results indicated that the classification accuracy increased to 72.8% with the inclusion of PALSAR-2 imagery. Thus, based on the PolSAR images, more complete target scattering information and polarization features regarding terrestrial objects can be provided. These indicate that PolSAR images can offer more effective complementary information for missing optical images and improving the accuracy of the restoration.

In conclusion, several limitations are associated with the currently popular cloud removal datasets:

(1) Single- or dual-polarization data can only transmit and receive electromagnetic waves of a fixed polarization mode, limiting their ability to capture complete scattering information regarding ground targets. Consequently, they lack descriptions of target polarization characteristics, such as phase, making it challenging to distinguish between features that exhibit identical backscatter signals [42]. However, most existing datasets predominantly consist of single-polarized or dual-polarized backscattering feature images combined with optical data. PolSAR data transmit and receive electromagnetic waves with different polarization modes to form a complete polarization basis and obtain a polarization scattering matrix. The polarization scattering matrix contains the amplitude and relative phase information of the backscattered signals from the target, enabling in-depth analysis and extraction of the physical properties of the target, thus providing a basis for further research on the scattering characteristics of the target. Consequently, the development of datasets that incorporate both PolSAR and optical images carries significant research value.

(2) Currently, high-spatial-resolution optical and SAR images have become increasingly accessible [43]. The spatial resolutions of optical remote sensing satellites have reached the sub-meter level; for example, the spatial resolutions of WorldView-3 and SuperView-1 are 0.31 m and 0.5 m. These high-spatial-resolution optical images may also experience issues with cloud coverage. Additionally, China’s GF-3 SAR satellite can reach a resolution of 1 m, which makes cloud removal possible based on high-resolution SAR data. Compared with low-resolution datasets, high-resolution images can provide more texture and structure information of ground objects, and more effective features can be extracted for image restoration, thus improving the restoration performance. However, most existing datasets are based on Sentinel-1 images with a resolution of 10 m, which provides limited textural details and makes it difficult to meet the needs for high-resolution optical image restoration. Therefore, to effectively address cloud removal at high spatial scales, there is a pressing demand for the development of high-resolution cloud removal datasets.

Therefore, to make full use of the advantages of PolSAR and promote research in high-resolution cloud removal tasks, we have constructed the first high-resolution remote sensing dataset based on PolSAR data, named AIR-POLSAR-CR1.0. Specifically, the dataset is based on the aerial PolSAR and optical images acquired from Zhangye, Gansu, and through preprocessing techniques such as radiometric correction, geometric registration, image mosaicking, and resampling. The spatial resolution of optical and PolSAR data is 0.5 m. Based on the existing research, PolSAR data can be represented as backscattering feature (BCFSAR) images for three polarizations and polarization feature (PFSAR) images containing phase information extracted based on a coherence matrix to facilitate subsequent image restoration. Additionally, we have also annotated the dataset by category and generated simulated cloudy images. Therefore, this dataset consists of cloudy and cloudless optical images, BCFSAR images, and PFSAR images specifically designed for cloud removal research. Several typical deep learning cloud removal algorithms were tested based on different feature categories and cloud cover to validate the stability of the proposed dataset, establishing corresponding baselines for reference by researchers in the field. The results of ablation experiments also prove that PolSAR can provide more comprehensive scattering information of ground objects for image restoration than single-polarized or dual-polarized backscattering feature images. The main contributions of this paper are as follows:

• We constructed a new multi-source dataset based on PolSAR data, namely AIR-POLSAR-CR1.0. It consists of RGB, near-infrared (NIR) optical images, and their corresponding fully PolSAR images. The fully PolSAR primarily includes BCFSAR images and PFSAR images. Compared to datasets constructed using single- or dual-polarization data, this dataset utilizes the polarization coherence matrix extracted from fully PolSAR data as the PFSAR image, incorporating inter-channel phase information. This provides more comprehensive scattering information and polarization features, aiding in the analysis of the geometric structure and scattering characteristics of objects while also offering effective information for image restoration in areas where optical images are missing;

• The AIR-POLSAR-CR1.0 dataset has a high spatial resolution of 0.5 m, making it the highest-spatial-resolution multi-source cloud removal dataset available. It can address the issue of lacking data sources for “sub-meter-level” optical data restoration and also provide better data resources for the research community, enabling the assessment and advancement of state-of-the-art image restoration algorithms through high-resolution images with significant structures and rich textural features;

• Based on the proposed dataset, we also performed manual annotations and cloud cover simulations. In subsequent research, analyses and processing can be conducted based on category information and varying percentages of missing data to explore the applicability of different models under various scenarios and missing conditions;

• We evaluated a set of representative deep-learning-based cloud removal methods using different experimental setups, establishing relevant baselines for assessing this dataset. The experimental results validated the stability of the dataset and its suitability for the existing algorithms, and the ablation study results also demonstrated the effectiveness of incorporating polarized feature images. This dataset will be released at https://github.com/WenjuanZhang-aircas/AIR-POLSAR-CR1.0 (accessed on 11 November 2024).

The remainder of this paper is organized as follows: Section 2 provides essential information about the dataset. Section 3 describes the construction of the dataset. Section 4 presents the experiments results, establishing baselines to analyze dataset characteristics. Finally, the conclusions and future directions are presented in Section 5.

2. Description of Dataset

The AIR-POLSAR-CR1.0 dataset is a high-resolution multi-source dataset constructed from aerial optical data and PolSAR images. Specifically, the dataset contains 2030 pairs of cloudy images, cloud-free optical images, BCFSAR images, and corresponding PFSAR image patches from the same region. Detailed information about the dataset is presented in Table 2. Similar to the bands of general high-spatial-resolution optical satellites, the airborne optical data include RGB and near-infrared channels, with a ground resolution of 0.5 m. Additionally, the airborne C-band 560 MHz full-polarization single-look complex SAR data encompass BCFSAR images with three HH, HV, and VV polarization bands, alongside PFSAR images with nine bands with a ground resolution of 0.5 m. Each image has a size of 256 × 256 pixels. Cloudy images were generated by overlaying cloud masks of varying proportions over clear optical images. Additionally, the dataset was annotated manually. Typical images from the dataset are illustrated in Figure 1, showcasing various land cover types, including forest, grassland, cropland, buildings, transportation routes, bare soil, sparse vegetation, and water bodies. The BCFSAR and PFSAR images are displayed as grayscale images. Furthermore, for the first five categories, cloud coverage ranges from 0–20% to 80–100%, while the examples for the transportation routes and water bodies exhibit cloud coverage of 40–60%.

3. Dataset Construction

The basic workflow for constructing the AIR-POLSAR-CR1.0 dataset is illustrated in Figure 2. This process primarily includes the acquisition of aerial remote sensing images, preprocessing of optical images and PolSAR data, representation of the polarization features of the PolSAR data, registration and cropping of multi-source images, classification annotation, and generation of cloudy images. Below, we will describe the specific details of this dataset generation process.

3.1. Aerial Remote Sensing Image Acquisition

Under the support of the national major scientific and engineering project for the aerospace remote sensing system, the Aerospace Information Research Institute under the Chinese Academy of Sciences has modified two Modern Ark 60 (MA60) remote sensing aircraft. The aircraft shown in Figure 3a are equipped with various observation windows, capable of simultaneously operating multiple optical and microwave observation devices and ensuring the consistency of the acquired optical and SAR data in terms of time and weather conditions.

The wide-field digital aerial photography camera can obtain high-precision, high-resolution, and wide-coverage optical image data synchronized with radar data, as shown in Figure 3b. The camera captures optical images in four spectral bands: red, green, blue, and near-infrared, making it applicable in emergency response, surveying, land resource management, urban planning, and other fields, with a spatial resolution of 0.2 m.

The high-resolution aerial multidimensional SAR system is a sensor with the capability of obtaining multidimensional data, including frequency, polarization, angle, and time phase. It can capture high-resolution PolSAR images in six bands: P, L, S, C, X, and Ku, as shown in Figure 3c [44]. The experiment is based on C-band airborne PolSAR data acquired from the multidimensional radar, including four polarization combinations: HH, HV, VH, and VV, with a spatial resolution of 0.5 m.

The Gansu Province’s Zhangye area was selected as the experimental site. The acquisition time of optical images and PolSAR images was 12 September 2021, and the experiment was carried out under cloud-free conditions.

3.2. Aerial Remote Sensing Image Preprocessing

The preprocessing of aerial image data consists of two main components: the preprocessing of aerial optical data and the preprocessing of aerial PolSAR data. The specific steps are outlined as follows:

3.2.1. Preprocessing of Aerial Optical Data

The processing flow for aerial optical data primarily includes the following steps: geometric and radiometric correction, adjustment calculation of aerial triangulation, and ultimately producing a digital orthoimage map (DOM), specifically including the following:

(1) Geometric and radiometric correction: The collected optical data from the measurement area undergo preliminary geometric and radiometric correction based on the original POS information.

(2) Adjustment calculation of aerial triangulation: The images are processed using ground control points to perform aerial triangulation, facilitated by regional network adjustments, producing images, orientation elements, and key point information.

(3) DOM generation: Based on the results of aerial triangulation, the images undergo dense matching to generate digital surface model (DSM) data. This DSM is then semi-automatically filtered to produce digital elevation model (DEM) data. Using the DEM data, images are orthorectified, followed by operations such as image mosaicking, color adjustment, and local corrections, ultimately outputting a standard DOM for the observation area. The generated optical orthorectified data are resampled to a spatial resolution of 0.5 m, ensuring consistency with the PolSAR data for subsequent image registration and cropping.

3.2.2. Preprocessing of Aerial PolSAR Data

The data processing flow of PolSAR data acquired by the multidimensional synthetic aperture radar system is illustrated, ultimately resulting in a DOM image through processes such as radiometric correction and geometric correction, specifically including the following:

(1) Format arrangement and data segmentation: Initially, the collected data from the surveyed area are divided into strip data and auxiliary data, generating raw data products organized by scene, along with auxiliary data, including product data files, product browsing images, product metadata files, and auxiliary data files, among others.

(2) Imaging processing and image correction: The raw data undergo imaging processing, radiometric correction, polarization correction, and channel registration to produce single-look complex images.

(3) DOM generation: Based on this, the polarimetric features are extracted from the single-look complex images. The single-look complex images and the polarimetric feature images undergo geometric correction to generate the DOM product for the regional PolSAR images. First, a control grid is established to solve for the geometric parameters. Then, a regional bundle adjustment is conducted on multiple SAR imagery within the photogrammetric area to optimize the geometric model parameters. Finally, the multi-scene PolSAR images are mosaicked to produce the DOM product.

3.3. Representation of Polarization Feature PolSAR Images

For the extraction and representation of polarization features in the dataset, feature extraction is based on the polarization coherence matrix, with specific details as follows. The PolSAR data provide a more comprehensive reflection of the scattering characteristics of the target objects based on the four polarization combinations of HH, HV, VH, and VV. Different polarization combinations exhibit varying sensitivities to target structure, material, texture, and shape. Each pixel in a PolSAR image contains information on polarized backscattering, which can be represented by a complex polarization scattering matrix S [39]:

(1)$S=\left[\begin{array}{cc}{S}_{HH}& S_{HV}\\ S_{VH}& S_{VV}\end{array}\right]$

where the real part of the complex number represents the radar signal intensity, while the imaginary part conveys phase information. $S_{ij}$ represents the complex scattering amplitude, while $S_{HV}$ denotes the scattering coefficient for horizontal (H) transmitted polarization and vertical (V) received polarization. The diagonal elements of the scattering matrix indicate that the incident wave and the scattered wave share the same polarization mode, referred to as co-polarization. The off-diagonal elements are known as cross-polarization, which indicates that the incident wave and the scattered wave possess different polarization modes. The polarization scattering matrix can be expressed in the form of a four-dimensional polarization target vector $k_L$:

(2)$k_L={[S_{HH},S_{HV},S_{VH},S_{VV}]}^T$

According to the principle of reciprocity, it satisfies

(3)$S_{HV}=S_{VH}$

In the Pauli basis, the scattering vector $k_p$ can be expressed as

(4)$k_p=1/\sqrt{2}{[S_{HH}+S_{VV},S_{HH}-S_{VV},2S_{HV}]}^T=1/\sqrt{2}[k_a,k_b,k_c]$

By multiplying the polarization scattering matrix $k_P$ with its conjugate transpose $k_P^{*H}$, we obtain the polarization coherence matrix T:

(5)$T=k_p\times k_p^{*H}=\left[\begin{array}{ccc}{T}_{11}& T_{12}& T_{13}\\ T_{21}& T_{22}& T_{23}\\ T_{31}& T_{32}& T_{33}\end{array}\right]$

The elements of the upper triangular matrix T can serve as inputs to the network. Therefore, the polarization characteristics can be represented as a nine-dimensional real vector as follows:

(6)$vv=[T_{11},T_{22},T_{33},Re\left(T_{12}\right),Im\left(T_{12}\right),Re\left(T_{13}\right),Im\left(T_{13}\right),Re\left(T_{23}\right),Im\left(T_{23}\right)]$

where $Re$(·) and $Im$(·) represent the real and imaginary components of a complex value, respectively. In subsequent processing, the polarization characteristics of each pixel of the PolSAR image are represented as a nine-dimensional real vector as input to the PFSAR image.

3.4. Image Registration and Cropping

Due to the consistency of the backscattering bands between HV polarization and VH polarization, only the backscattering band of VH polarization is utilized. The PolSAR data are represented as BCFSAR images with three polarization bands and PFSAR images with nine bands. After preprocessing, image registration is performed on the optical images, BCFSAR images, and PFSAR images, with registration error controlled within two pixels. The registered optical data, BCFSAR data, and PFSAR data are cropped into smaller regions of size 256 pixels by 256 pixels, with a stride of 128 pixels. The cropped dataset contains 2030 image pairs.

3.5. Category Annotation

Based on the classification system of mainstream land cover products and landscape styles, the dataset is classified according to the SinoLC-1 classification system [45], which unifies the ESA_GLC10 [46], ESRI_GLC10 [47], and FROM_GLC10 classification systems [48], as shown in Table 3. The primary land cover categories defined for the AIR-POLSAR-CR1.0 dataset include seven categories: “Tree cover”, “Grassland”, “Cropland”, “Buildings”, “Traffic route”, “Barren and sparse vegetation”, and “Water”.

The entire dataset was manually labeled to classify the categories. The category that occupies more than 50% of the area in the image is designated as the label for that image based on visual discrimination, with the number of samples in each category listed in Table 4. The sample amount varies significantly across different scenes, ranging from 85 to 395 images. Overall, the proposed dataset contains a total of 2030 images across the seven categories. Additionally, Figure 4 presents sample images of typical regions from these seven categories, including BCFSAR images, PFSAR images, and cloud-free optical images.

3.6. Cloud Image Generation

To evaluate the performance of image inpainting for missing regions of various sizes across different land types, masks with different cloud cover percentages were generated through random rotation or scaling transformations. The cloud cover ratios for each image were set at 0–20%, 20–40%, 40–60%, 60–80%, and 80–100%. Within each category, the cloud masks were randomly distributed in the same proportion and overlaid on cloud-free optical images to create simulated missing optical images. Figure 5 presents sample images of five different cloud cover percentages for the cropland category.

4. Experimental Studies

To validate the dataset’s applicability for cloud removal, we selected five deep-learning-based cloud removal algorithms for experimentation. Furthermore, to explore the optimal input formats for each model, ablation experiments were conducted on the dataset to demonstrate the effectiveness of BCFSAR and PFSAR feature images. Specifically, the experiment does not use auxiliary data but only utilizes the VV and VH dual-polarization BCSAR images, using only BCFSAR images or PolSAR images, and both images together as auxiliary data for comparative analysis. This section will first introduce various metrics for evaluating the restoration results. Then, ablation analysis will be performed based on different input formats for each model to evaluate the restoration performance. Finally, a comparative analysis of these advanced deep learning algorithms will be presented, along with an evaluation of their restoration results across different categories and cloud cover conditions.

4.1. Experimental Setup and Evaluation Metrics

The experiments were conducted on an NVIDIA GeForce RTX 3090 GPU using the PyTorch deep learning framework. Optimization was performed using the Adaptive Moment Estimation (ADAM) algorithm, with hyperparameters set to $\beta 1=0.5$, $\beta 2=0.999$, and the initial learning rate set to $7\times {10}^{-5}$. The number of training epochs was configured to 200, and the optimal weights were saved for testing.

The representative metrics for evaluating the restoration results include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Spectral Angle Mapper (SAM), and Correlation Coefficient (CC). PSNR is an indicator of image quality that assesses the pixel-level differences between the reconstructed image and the ground truth image. SSIM quantifies the structural differences between the target and predicted images by capturing perceptual changes in structural information, brightness, and contrast. The SAM metric measures the spectral similarity of two images by quantifying the spectral angle between them. The CC metric is used to assess the degree of correlation between two images.

When the input images are considered as vectors $x\in R^L$ and $y\in R^L$, the calculation formulas for these metrics are as follows:

(7)$PSNR(x,y)=20·lg{\displaystyle \frac{1}{RMSE(x,y)}}$

(8)$SSIM(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+{\theta }_1)(2{\sigma }_{xy}+{\theta }_2)}{({\mu }_x+{\mu }_y+{\theta }_1)({\sigma }_x+{\sigma }_y+{\theta }_2)}}$

(9)$SAM(x,y)=co{s}^{-1}·({\displaystyle \frac{{\sum }_{i=1}^L{x}_i{y}_i}{\sqrt{{\sum }_{i=1}^L{x}_i^2{\sum }_{i=1}^L{y}_i^2}}})$

(10)$CC(x,y)={\displaystyle \frac{{\sum }_{i=1}^L(x_i-{\mu }_x)(y_i-{\mu }_y)}{\sqrt{({\sum }_{i=1}^L{(x_i-{\mu }_x)}^2}\sqrt{{\sum }_{i=1}^L{(y_i-{\mu }_y)}^2}}}$

where ${\mu }_x$ and ${\mu }_y$ are the mean values of images x and y, ${\sigma }_x$ and ${\sigma }_y$ are the standard deviations of images x and y, and ${\sigma }_x$ and ${\sigma }_y$ represent the covariance between the two images. Constants ${\theta }_1$ and ${\theta }_2$ are used to stabilize the division with a small constant. The calculation formulas for the metrics are as follows:

(11)$RMSE(x,y)=\sqrt{{\displaystyle \frac{1}{L}}\sum _{i=1}^L{(x_i-y_i)}^2}$

4.2. Baseline Methods

Generally, cloud removal is a process of missing information reconstruction. Five deep-learning-based cloud removal algorithms were selected for experiments on this dataset. The comparative algorithms include the Pix2Pix model [49], SAR-Opt-cGAN model [19], DSen2-CR model [50], USSRN-CR model [51], and GLF-CR model [52].

(1) Pix2Pix Model: This is a deep learning model that is utilized for image translation. It employs a conditional generative adversarial network (cGAN) [53] structure with a U-Net architecture as the generator to directly convert single-polarized or dual-polarized backscattering feature SAR images into cloud-free optical images.

(2) SAR-Opt-cGAN Model: This model also uses a CGAN framework, but it modifies and further develops the U-Net architecture to integrate multimodal remote sensing data. To enhance its spectral mapping capabilities, it incorporates 1024 feature mapping layers in both the encoder and decoder. This model generates cloud-free optical data from simulated optical images by leveraging single-polarized or dual-polarized backscattering feature SAR images as auxiliary data.

(3) DSen2-CR Model: This model introduces SAR images as priors into the network. It simply concatenates the cloudy optical images with SAR data and utilizes a deep residual neural network architecture (ResNet) [54] to generate cloud-free optical images. The highly nonlinear transformation from SAR to optical, along with cloud detection and processing, are implicitly learned and executed within the network. Long skip connections facilitate the transmission of the network’s inputs to the output layer, enabling the entire network to learn to predict residual maps, effectively preserving surface information in cloud-free areas. Additionally, a novel cloud-adaptive loss function is proposed to maximize the retention of original information.

(4) USSRN-CR Model: This model is based on a residual architecture for missing image reconstruction, which utilizes SAR images as auxiliary data to effectively remove clouds. It consists of two modules: a feature extraction and fusion module and a reconstruction module. The feature extraction and fusion module introduces gated convolutional layers to distinguish between cloudy and clear pixels to better extract relevant area information, thus avoiding errors in feature extraction from cloudy regions. In the reconstruction module, spatial and channel attention mechanisms are integrated to acquire global spatial and spectral information to implement better features.

(5) GLF-CR Model: This is a cloud removal algorithm based on global–local fusion that effectively reconstructs cloud-covered regions using SAR data as auxiliary input. From a global perspective, it proposes a global context interaction module, employing a transformer mechanism to guide global interactions of contextual information using SAR features, ensuring consistency between the reconstructed information of cloudy and cloud-free areas. From a local perspective, it introduces a local feature compensation module that enhances the transmission of complementary information embedded in the SAR images, along with a filtering module that mitigates the effects of speckle noise, resulting in more reliable texture details.

4.3. Data Preprocessing

To eliminate outliers, the four bands of the optical data are cropped to within the range of [0, 64349]. Secondly, the three bands of BCFSAR feature images are cropped to within the range of [−30 dB, 10 dB]. After data statistics of PFSAR feature images, the average and standard deviation values are calculated to perform data culling. Ninety-five percent of the data are retained, and the data from nine bands are cropped within the range of [−98 dB, −15 dB]. In order to ensure the stability of model training, the optical data and PolSAR data are normalized in the input model to ensure that the data are in the same range.

4.4. Experimental Results

4.4.1. Ablation Experiments Based on Standard Datasets

Considering that the previous models only used single-polarized or dual-polarized backscattering feature SAR images for image inpainting, in order to verify the effectiveness of the proposed dataset, we conducted an ablation experiment based on the input data in this section using different PolSAR feature images as auxiliary data to evaluate the repair performances of each input. In the experiments, we denote the model AA that inputs without any PolSAR images as the AA without PolSAR model, the model that inputs only the VV and VH bands of the BCFSAR images as the AA with TBBCFSAR model, the model that inputs only BCFSAR images as the AA with BCFSAR model, and the model that solely uses PFSAR images as the AA with PFSAR model. When both images are used in conjunction, it is designated as the AA with PolSAR model.

(1) Pix2pix Model: Figure 6a–h illustrate the missing optical images, BCFSAR images, PFSAR images, result images from the Pix2pix with TBBCFSAR model, the Pix2pix with BCFSAR model, the Pix2pix with PFSAR model, the Pix2pix with PolSAR model, and ground truth. Table 5 presents quantitative evaluation results, showing a quantitative assessment of the Pix2pix method on the AIR-POLSAR-CR1.0 dataset. Since the Pix2pix model uses only PolSAR images as input, no ablation experiments without PolSAR data were performed.

Figure 6 displays the reconstruction results for cropland and buildings. Notably, in the second row of the figure, the Pix2pix with PolSAR model exhibits superior reconstruction results concerning the edge structural information of the cropland when compared to the Pix2pix with TBBCFSAR model, Pix2pix with BCFSAR model, and Pix2pix with PFSAR model. The highest PSNR index was also recorded in the quantitative evaluation results in Table 6. In terms of spectral reconstruction, all three models display high SAM indices, indicating significant spectral discrepancies among them. Overall, the Pix2pix model demonstrates greater restoration accuracy when utilizing both BCFSAR and PFSAR images as auxiliary input sources.

(2) SAR-opt-cGAN Model: Similar to the Pix2pix model, Figure 7a–c depict the simulated optical images, BCFSAR images, and PFSAR images, respectively. Figure 7d–h illustrate the restoration results from the SAR-opt-cGAN without PolSAR model, the SAR-opt-cGAN with TBBCFSAR model, SAR-opt-cGAN with BCFSAR model, SAR-opt-cGAN with PFSAR model, and SAR-opt-cGAN with PolSAR model, while Figure 7i presents the ground truth. Table 6 provides the quantitative evaluation results.

Overall, Figure 7 includes the reconstruction outcomes for features such as barren and sparse vegetation and traffic route. In the second row of the figure, the SAR-opt-cGAN with PolSAR model successfully reconstructs the outline of the route, maintaining continuity in its structure. In contrast, the SAR-opt-cGAN without PolSAR model, the SAR-opt-cGAN with TBBCFSAR model, the SAR-opt-cGAN with BCFSAR, and SAR-opt-cGAN with PFSAR models fail to accurately reconstruct the road. As shown in Table 6, the accuracy of the SAR-opt-cGAN with PFSAR model surpasses the other models and achieves the highest accuracy across various metrics, with a PSNR value of 28.000. This indicates that the SAR-opt-cGAN model is also more suitable for the combined use of BCFSAR and PFSAR images as input.

(3) DSen2-CR Model: Figure 8a represents the missing optical image, while Figure 8b and Figure 8c show the BCFSAR and PFSAR images, respectively. Figure 8d–i depict the restoration results from the DSen2-CR without PolSAR model, the DSen2-CR with TBBCFSAR model, DSen2-CR with BCFSAR model, DSen2-CR with PFSAR model, and DSen2-CR with PolSAR model, along with the ground truth. Table 7 provides the quantitative evaluation results.

As illustrated in the first row of Figure 8, the reconstruction of the cropland area appears somewhat blurred. Compared to the other two models, the DSen2-CR with PFSAR model yields better visual results, effectively reconstructing detailed information. In Table 7, when only the PFSAR feature map is used as auxiliary data, this model achieves the highest reconstruction accuracy. It is comparable to the DSen2-CR with PolSAR model in terms of the SSIM and CC metrics but outperforms the other models in PSNR and SAM metrics. Considering that the DSen2-CR model reconstructs the missing regions based on learned residuals, introducing joint BCFSAR images and PFSAR images as auxiliary information during the repair process may introduce noise and redundant information. This can make it difficult for the model to accurately learn the residual information of the data, adversely affecting the generation results and reducing the overall accuracy of the image restoration.

(4) GLF-CR Model: Figure 9a–i illustrate the missing optical image, BCFSAR image, PFSAR image, and the restoration results from the GLF-CR without PolSAR model, GLF-CR with TBBCFSAR model, GLF-CR with BCFSAR model, GLF-CR with PFSAR model, and GLF-CR with PolSAR model, along with the ground truth. Table 8 presents the quantitative evaluation results of the GLF-CR model on the standard dataset.

As depicted in Figure 9, both the GLF-CR without PolSAR model, the GLF-CR with TBBCFSAR model, GLF-CR with BCFSAR model, and GLF-CR with PolSAR model exhibit noticeable artifacts in the reconstructed route areas, with the hue of the reconstructed images showing significant discrepancies compared to the ground truth image. In contrast, the structure of the GLF-CR with PFSAR model is more consistent. Additionally, as shown in Table 8, the GLF-CR with PolSAR model achieves the best quantitative results across every metric. Consequently, the GLF-CR model also attains the highest reconstruction accuracy when only the PFSAR image is used as auxiliary input.

(5) USSRN-CR Model: Figure 10a–h represent the missing optical image, BCFSAR image, PFSAR image, and restoration results from the USSRN-CR without PolSAR model, the USSRN-CR with TBBCFSAR model, USSRN-CR with BCFSAR model, USSRN-CR with PFSAR model, and USSRN-CR with PolSAR model. Figure 10i is the ground truth image. Table 9 provides the quantitative evaluation results of the USSRN-CR model on the standard dataset.

As shown in Figure 10, the USSRN-CR with PFSAR model produces images that exhibit clear edges and maintain consistency and continuity in structure, with more pronounced textural details reconstructed in buildings and tree cover areas. In contrast, the other models fail to accurately reconstruct the structural outlines of buildings, resulting in excessively smooth textural details in vegetative surfaces. Furthermore, in Table 9, the USSRN-CR with PFSAR model achieves the best quantitative results, outperforming the others across three metrics and showing comparable performance to the USSRN-CR with PolSAR model on the SSIM metric. The USSRN-CR with PFSAR model reaches the highest reconstruction accuracy.

In summary, different cloud removal models achieve optimal performance under varying input data conditions. When PolSAR data are not used as auxiliary data, the restoration results of all the models deteriorate, and the PSNR metrics decrease. The Pix2pix2 model and the SAR-Opt-cGAN model demonstrate the best network performance when both the BCFSAR feature image and the PFSAR feature image are used in conjunction. In contrast, the DSen2-CR model, GLF-CR model, and USSRN-CR model attain the highest reconstruction accuracy when only the PFSAR image is used as auxiliary input. Among the various models, using PFSAR images or BCFSAR images as auxiliary data results in higher accuracy than TBBCFSAR images in terms of PSNR, SSIM, CC, and SAM metrics. All the models exhibit improved reconstruction accuracy when utilizing the PFSAR image and perform poorly in terms of accuracy when only using TBBCFSAR data. Notably, in the Pix2pix2 model, which directly converts PolSAR images into optical images, the PSNR metric increased by 1.12 dB. Based on the qualitative and quantitative evaluation results, BCFSAR images and PFSAR images introduce new scattering features and polarization information to the model. Therefore, compared with the TBBCFSAR images used in previous datasets, incorporating PolSAR data or PFSAR data auxiliary information in deep-learning-based cloud removal models significantly enhances the accuracy of image restoration results.

4.4.2. Category Analysis and Cloud Cover Analysis of Cloud Removal Methods Based on Standard Datasets

Table 10 shows the PSNR metrics of each baseline model across seven categories. Generally, all the models achieved higher reconstruction accuracy for uniform features such as barren and sparse vegetation and water, while their performance was lower for complex features like buildings and traffic route. Among the algorithms, the USSRN-CR and GLF-CR models show significant improvements in the metrics for the water and traffic route categories. The USSRN-CR model outperforms the GLF-CR model in reconstructing barren and sparse vegetation, buildings, cropland, and water, while, in tree cover, grassland, and traffic routes, the PSNR metrics of the GLF-CR model are comparable to those of the USSRN-CR model. Table 11 shows the PSNR indicators of each baseline model regarding different cloud cover. In general, with the increase in cloud cover, the performance of each model decreases.

Figure 11a–h display the missing optical image, BCFSAR image, PFSAR image, reconstruction results from five baseline models, and the ground truth image. Table 12 provides the quantitative evaluation results.

Specifically, Figure 11 illustrates the reconstruction performance of these five baseline algorithms across various categories. The Pix2pix model, which utilized only the PolSAR image as the auxiliary input, generated reconstruction results that showed significant changes in cloud-free areas, with the texture details failing to be reconstructed. In barren and sparse vegetation, the texture of cloud-free regions was also reconstructed, resulting in considerable discrepancies with the ground truth. The spectral information in the reconstructed images for building areas also displayed significant deviations. In contrast, the SAR-Opt-cGAN model, which incorporated cloudy optical images as the input, preserved the color characteristics and texture details of cloud-free areas but struggled to reconstruct the texture details and edge structural information of complex features such as buildings. The DSen2-CR model produced noticeable artifacts in the cropland and water categories. The GLF-CR model’s reconstruction results were relatively blurry, making it challenging to accurately reconstruct the structural characteristics of objects. Comparatively, the USSRN-CR model effectively restored the structural information of linear features, and the spectral color information resembled that of the ground truth image. The quantitative results presented in Table 12 further demonstrate the superiority of the USSRN-CR model over the other models.

5. Conclusions

In this paper, we introduce the composition and construction methods of the AIR-POLSAR-CR1.0 dataset. The dataset contains 2030 optical images (both cloudy and cloud-free), as well as BCFSAR and PFSAR images, all with a spatial resolution of 0.5 m. In comparison to the existing single-polarization and dual-polarization datasets, this dataset uses BCFSAR images for three polarizations and PFSAR images with full polarization characteristics and introduces phase information between channels, thereby providing more comprehensive scattering and polarization information regarding ground objects. The results of ablation experiments also prove that the model based on PolSAR images or PFSAR images as auxiliary data can obtain higher accuracy than TBBCFSAR images in image restoration.

In addition, the dataset has been manually annotated, and clouds of different sizes have been simulated to facilitate more complex research and analysis of PolSAR and optical data. This provides valuable guidance for further research and practice in the field of image restoration. Such studies contribute to selecting the appropriate combination of network models and auxiliary feature information to achieve better restoration results, enabling a comprehensive understanding of their applicability in different scenarios and cloud cover conditions.

Author Contributions

Conceptualization, Y.W., W.Z. and B.Z. (Bing Zhang); methodology, Y.W., W.Z. and B.Z. (Bing Zhang); formal analysis, Y.W.; investigation, Y.W., W.Z. and B.Z. (Bing Zhang); resources, W.Z. and B.Z. (Bing Zhang); data curation, Y.W., W.Z., J.P., W.J., F.Y., B.Z. (Bo Zhang), X.Y. and B.Z. (Bing Zhang); writing—original draft preparation, Y.W., W.J. and F.Y.; writing—review and editing, Y.W., W.Z. and B.Z. (Bing Zhang); visualization, Y.W.; supervision, W.Z. and B.Z. (Bing Zhang); project administration, W.Z. and B.Z. (Bing Zhang); funding acquisition, W.Z. and B.Z. (Bing Zhang) All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Acknowledgments

We sincerely appreciate the constructive comments and suggestions of the anonymous reviewers, which have greatly helped to improve this paper.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Examples of the AIR-POLSAR-CR1.0 dataset.

Figure 2. The basic flowchart of the AIR-POLSAR-CR1.0 dataset construction.

Figure 3. Flight platform and measuring equipment. (a) MA60 aircraft. (b) The wide-field digital aerial camera. (c) The multidimensional SAR system.

Figure 4. Examples of the main categories of the AIR-POLSAR-CR1.0 dataset.

Figure 5. Examples of the different cloud coverage of the AIR-POLSAR-CR1.0 dataset.

Figure 6. Experimental results of the Pix2pix model on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the Pix2pix with TBBCFSAR model. (e) Results of the Pix2pix with BCFSAR model. (f) Results of the Pix2pix with PFSAR model. (g) Results of the Pix2pix with PolSAR model. (h) Ground truth.

Figure 7. Experimental results of the SAR-Opt-cGAN model on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the SAR-Opt-cGAN without PolSAR model. (e) Results of the SAR-opt-cGAN with TBBCFSAR model. (f) Results of the SAR-Opt-cGAN with BCFSAR model. (g) Results of the SAR-Opt-cGAN with PFSAR model. (h) Results of the SAR-Opt-cGAN with PolSAR model. (i) Ground truth.

Figure 8. Experimental results of the DSen2-CR model on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the DSen2-CR without PolSAR model. (e) Results of the DSen2-CR with TBBCFSAR model. (f) Results of the DSen2-CR with BCFSAR model. (g) Results of the DSen2-CR with PFSAR model. (h) Results of the DSen2-CR with PolSAR model. (i) Ground truth.

Figure 9. Experimental results of the GLF-CR model on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the GLF-CR without PolSAR model. (e) Results of the GLF-CR with TBBCFSAR model. (f) Results of the GLF-CR with BCFSAR model. (g) Results of the GLF-CR with PFSAR model. (h) Results of the GLF-CR with PolSAR model. (i) Ground truth.

Figure 10. Experimental results of the USSRN-CR model on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the USSRN-CR without PolSAR model. (e) Results of the DSen2-CR with TBBCFSAR model. (f) Results of the USSRN-CR with BCFSAR model. (g) Results of the USSRN-CR with PFSAR model. (h) Results of the USSRN-CR with PolSAR model. (i) Ground truth.

Figure 11. Experimental results of the baseline models on the AIR-POLSAR-CR1.0 dataset. (a) Simulated cloudy images. (b) BCFSAR images. (c) PFSAR images. (d) Results of the Pix2pix model. (e) Results of the SAR-Opt-cGAN model. (f) Results of the DSen2-CR model. (g) Results of the GLF-CR model. (h) Results of the USSRN-CR model. (i) Ground truth.

Table 1

Introduction to cloud removal datasets.

Datasets		w/SAR	Source	Resolution/m	Band and Range Bandwidth of SAR Data	Polarization Mode	Size
Simulated datasets	Airborne	✓	Airborne SAR data and airborne optical data	1	X/\	VV	128
	Scotland and India	✓	Sentinel-1 and Sentinel-2	10	C/10.7–14.1 MHz	VV and VH	256
Real datasets	RICEII	×	Landsat-8	30	\	\	512
	WHU2-CR	×	Sentinel-2	10	\	\	256
	Sen2_MTC	×	Sentinel-2 of the time series	10	\	\	256
	SEN12MS-CR	✓	Sentinel-1 and Sentinel-2	10	C/10.7–14.1 MHz	VV and VH	256
	SEN12MS-CR-TS	✓	Sentinel-1 and Sentinel-2 images of the time series	10	C/10.7–14.1 MHz	VV and VH	256
	AllClear	✓	Landsat 8/9, Sentinel-1, and Sentinel-2 images of the time series	10	C/10.7–14.1 MHz	VV and VH	256
	M3R-CR	✓	Sentinel-1, PlanetScope data, and WorldCover land cover maps	3	C/10.7–14.1 MHz	VV and VH	256

Table 2

Description of the AIR-POLSAR-CR1.0 dataset.

Datasets		w/SAR	Source	Resolution/m	Band and Range Bandwidth of SAR Data	Polarization Mode	Size
Simulated datasets	AIR-POLSAR-CR1.0	✓	Airborne PolSAR images and airborne optical data	0.5	C/560 MHz	HH, HV, VH, and VV	256

Table 3

Category relations between FROM_GLC10, ESA_GLC10, ESRI_GLC10, SinoLC-1, and AIR-POLSAR-CR1.0.

	FROM_GLC10	Esri_GLC10	ESA_GLC10	SinoLC-1	AIR-POLSAR-CR1.0
Land cover	Forest	Trees	Trees	Tree cover	Tree cover
	Grassland	Grass	Grassland	Grassland	Grassland
	Cropland	Crops	Cropland	Cropland	Cropland
	Impervious area	Built area	Built-up	Building Traffic route	Building Traffic route
	Bare land	Bare	Barren/sparse vegetation	Barren and sparse vegetation	Barren and sparse vegetation
	Water body	Water	Open water	Water	Water

Table 4

Numbers of different categories in the AIR-POLSAR-CR1.0 dataset.

Land Cover	Sample Number
Forest	230
Grassland	85
Cropland	380
Building	395
Traffic route	245
Barren and sparse vegetation	310
Water	385

Table 5

Quantitative evaluation of the Pix2pix model on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	Input		PSNR ↑	SSIM ↑	CC ↑	SAM ↓
	BCFSAR	PFSAR
Pix2pix	VV and VH	×	21.771	0.474	0.433	5.962
	✓	×	22.153	0.484	0.438	5.602
	×	✓	22.888	0.501	0.489	5.010
	✓	✓	23.166	0.506	0.505	4.955

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

Table 6

Quantitative evaluation of the SAR-Opt-cGAN model on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	Input		PSNR ↑	SSIM ↑	CC ↑	SAM ↓
	BCFSAR	PFSAR
SAR-Opt-cGAN	×	×	26.614	0.758	0.743	2.686
	VV and VH	×	27.065	0.757	0.769	2.626
	✓	×	27.247	0.766	0.789	2.772
	×	✓	27.514	0.765	0.783	2.508
	✓	✓	28.000	0.773	0.792	2.383

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

Table 7

Quantitative evaluation of the DSen2-CR model on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	Input		PSNR ↑	SSIM ↑	CC ↑	SAM ↓
	BCFSAR	PFSARs
DSen2-CR	×	×	30.350	0.837	0.821	1.278
	VV and VH	×	30.596	0.843	0.833	1.282
	✓	×	30.673	0.839	0.832	1.275
	×	✓	31.005	0.843	0.840	1.248
	✓	✓	30.757	0.846	0.839	1.273

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

Table 8

Quantitative evaluation of the GLF-CR model on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	Input		PSNR ↑	SSIM ↑	CC ↑	SAM ↓
	BCFSAR	PFSAR
GLF-CR	×	×	31.009	0.847	0.840	1.440
	VV and VH	×	31.535	0.845	0.854	1.305
	✓	×	31.902	0.852	0.863	1.294
	×	✓	32.383	0.856	0.873	1.231
	✓	✓	32.235	0.856	0.869	1.271

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

Table 9

Quantitative evaluation of the USSRN-CR model on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	Input		PSNR ↑	SSIM ↑	CC ↑	SAM ↓
	BCFSAR	PFSAR
USSRN-CR	×	×	31.541	0.850	0.849	1.210
	VV and VH	×	32.265	0.858	0.867	1.178
	✓	×	32.342	0.860	0.871	1.171
	×	✓	33.036	0.867	0.885	1.120
	✓	✓	32.626	0.869	0.877	1.198

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

Table 10

Quantitative evaluation results of different categories of PSNR indicators on the AIR-POLSAR-CR1.0 dataset by baseline models. The best performance is shown in bold.

Model	Barren and Sparse Vegetation	Building	Cropland	Tree Cover	Grassland	Traffic Route	Water
Pix2pix	22.935	20.407	23.759	21.824	21.046	20.762	28.532
SAR-Opt-cGAN	28.505	26.007	28.072	27.709	27.238	26.422	31.031
DSen2-CR	29.809	27.330	31.133	29.798	28.909	27.828	39.015
GLF-CR	31.499	28.275	31.925	30.924	30.261	29.484	41.162
USSRN-CR	31.817	28.310	33.644	30.883	30.232	29.476	42.669

Table 11

Quantitative evaluation results of different cloud cover of PSNR indicators on the AIR-POLSAR-CR1.0 dataset by baseline models. The best performance is shown in bold.

Model	0–20%	20–40%	40–60%	60–80%	80–100%
Pix2pix	23.133	22.933	23.413	23.278	23.073
SAR-Opt-cGAN	32.889	29.078	27.831	25.904	24.299
DSen2-CR	38.634	32.507	30.840	27.484	25.562
GLF-CR	39.970	33.691	31.929	28.998	27.326
USSRN-CR	41.405	34.387	32.615	29.386	27.386

Table 12

Quantitative evaluation results of comparison models on the AIR-POLSAR-CR1.0 dataset. The best performance is shown in bold.

Model	PSNR ↑	SSIM ↑	CC ↑	SAM ↓
Pix2pix	23.166	0.506	0.505	4.955
SAR-Opt-cGAN	28.000	0.773	0.792	2.383
DSen2-CR	31.005	0.843	0.840	1.248
GLF-CR	32.383	0.856	0.873	1.231
USSRN-CR	33.036	0.867	0.885	1.120

↑ indicates that a higher score means a better performance, and ↓ indicates that a lower score means a better performance.

References

1. Sharma, S.; Beslity, J.O.; Rustad, L.; Shelby, L.J.; Manos, P.T.; Khanal, P.; Reinmann, A.B.; Khanal, C. Remote sensing and GIS in natural resource management: Comparing tools and emphasizing the importance of in-situ data. Remote Sens.; 2024; 16, 4161. [DOI: https://dx.doi.org/10.3390/rs16224161]

2. Zhao, S.; Liu, M.; Tao, M.; Zhou, W.; Lu, X.; Xiong, Y.; Li, F.; Wang, Q. The role of satellite remote sensing in mitigating and adapting to global climate change. Sci. Total Environ.; 2023; 904, 166820. [DOI: https://dx.doi.org/10.1016/j.scitotenv.2023.166820]

3. Pettorelli, N.; Laurance, W.F.; O’Brien, T.G.; Wegmann, M.; Nagendra, H.; Turner, W. Satellite remote sensing for applied ecologists: Opportunities and challenges. J. Appl. Ecol.; 2014; 51, pp. 839-848. [DOI: https://dx.doi.org/10.1111/1365-2664.12261]

4. Kaku, K. Satellite remote sensing for disaster management support: A holistic and staged approach based on case studies in Sentinel Asia. Int. J. Disast Risk Reduct.; 2019; 33, pp. 417-432. [DOI: https://dx.doi.org/10.1016/j.ijdrr.2018.09.015]

5. Li, Y.; Chen, W.; Zhang, Y.; Tao, C.; Xiao, R.; Tan, Y. Accurate cloud detection in high-resolution remote sensing imagery by weakly supervised deep learning. Remote Sens. Environ.; 2020; 250, 112045. [DOI: https://dx.doi.org/10.1016/j.rse.2020.112045]

6. Lin, D.; Xu, G.; Wang, X.; Wang, Y.; Sun, X.; Fu, K. A remote sensing image dataset for cloud removal. arXiv; 2019; arXiv: 1901.00600

7. Li, J.; Wu, Z.; Hu, Z.; Li, Z.; Wang, Y.; Molinier, M. Deep learning based thin cloud removal fusing vegetation red edge and short wave infrared spectral information for Sentinel-2A imagery. Remote Sens.; 2021; 13, 157. [DOI: https://dx.doi.org/10.3390/rs13010157]

8. Huang, G.L.; Wu, P.Y. CTGAN: Cloud transformer generative adversarial network. Proceedings of the IEEE International Conference on Image Processing (ICIP); Bordeaux, France, 16–19 October 2022; pp. 511-515.

9. Bermudez, J.D.; Happ, P.N.; Oliveira, D.A.B.; Feitosa, R.Q. SAR to optical image synthesis for cloud removal with generative adversarial networks. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.; 2018; 4, pp. 5-11. [DOI: https://dx.doi.org/10.5194/isprs-annals-IV-1-5-2018]

10. Zhang, S.; Li, X.; Zhou, X.; Wang, Y.; Hu, Y. Cloud removal using SAR and optical images via attention mechanism-based GAN. Pattern Recognit. Lett.; 2023; 175, pp. 8-15. [DOI: https://dx.doi.org/10.1016/j.patrec.2023.09.014]

11. Schmitt, M.; Hughes, L.H.; Zhu, X.X. The SEN1-2 dataset for deep learning in SAR-optical data fusion. arXiv; 2018; arXiv: 1807.01569[DOI: https://dx.doi.org/10.5194/isprs-annals-IV-1-141-2018]

12. Schmitt, M.; Hughes, L.H.; Qiu, C.; Zhu, X.X. SEN12MS–A curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion. arXiv; 2019; arXiv: 1906.07789[DOI: https://dx.doi.org/10.5194/isprs-annals-IV-2-W7-153-2019]

13. Huang, M.; Xu, Y.; Qian, L.; Shi, W.; Zhang, Y.; Bao, W.; Wang, N.; Liu, X.; Xiang, X. The QXS-SAROPT dataset for deep learning in SAR-optical data fusion. arXiv; 2021; arXiv: 2103.08259

14. Li, X.; Zhang, G.; Cui, H.; Hou, S.; Wang, S.; Li, X.; Chen, Y.; Li, Z.; Zhang, L. MCANet: A joint semantic segmentation framework of optical and SAR images for land use classification. Int. J. Appl. Earth Obs.; 2022; 106, 102638. [DOI: https://dx.doi.org/10.1016/j.jag.2021.102638]

15. Gao, G.; Wang, M.; Zhang, X.; Li, G. DEN: A new method for SAR and optical image fusion and intelligent classification. IEEE Trans. Geosci. Remote Sens.; 2024; 63, 5201118. [DOI: https://dx.doi.org/10.1109/TGRS.2024.3500036]

16. Wang, S.; Cai, B.; Hou, D.; Ding, Q.; Wang, J.; Shao, Z. MF-BHNet: A hybrid multimodal fusion network for building height estimation using Sentinel-1 and Sentinel-2 imagery. IEEE Trans. Geosci. Remote Sens.; 2024; 62, pp. 1-19. [DOI: https://dx.doi.org/10.1109/TGRS.2024.3477588]

17. Xu, W.; Yuan, X.; Hu, Q.; Li, J. SAR-optical feature matching: A large-scale patch dataset and a deep local descriptor. Int. J. Appl. Earth Obs. Geoinf.; 2023; 122, 103433. [DOI: https://dx.doi.org/10.1016/j.jag.2023.103433]

18. Liu, C.; Sun, Y.; Xu, Y.; Sun, Z.; Zhang, X.; Lei, L.; Kuang, G. A review of optical and SAR image deep feature fusion in semantic segmentation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.; 2024; 17, pp. 12910-12930. [DOI: https://dx.doi.org/10.1109/JSTARS.2024.3424831]

19. Gao, J.; Yuan, Q.; Li, J.; Zhang, H.; Su, X. Cloud removal with fusion of high resolution optical and SAR images using generative adversarial networks. Remote Sens.; 2020; 12, 191. [DOI: https://dx.doi.org/10.3390/rs12010191]

20. Ebel, P.; Meraner, A.; Schmitt, M.; Zhu, X.X. Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery. IEEE Trans. Geosci. Remote Sens.; 2020; 59, pp. 5866-5878. [DOI: https://dx.doi.org/10.1109/TGRS.2020.3024744]

21. Czerkawski, M.; Upadhyay, P.; Davison, C.; Werkmeister, A.; Cardona, J.; Atkinson, R.; Michie, C.; Andonovic, I.; Macdonald, M.; Tachtatzis, C. Deep internal learning for inpainting of cloud-affected regions in satellite imagery. Remote Sens.; 2022; 14, 1342. [DOI: https://dx.doi.org/10.3390/rs14061342]

22. Ebel, P.; Xu, Y.; Schmitt, M.; Zhu, X.X. SEN12MS-CR-TS: A remote-sensing data set for multimodal multitemporal cloud removal. IEEE Trans. Geosci. Remote Sens.; 2022; 60, 5222414. [DOI: https://dx.doi.org/10.1109/TGRS.2022.3146246]

23. Zhou, H.; Kao, C.H.; Phoo, C.P.; Mall, U.; Hariharan, B.; Bala, K. AllClear: A comprehensive dataset and benchmark for cloud removal in satellite imagery. arXiv; 2024; arXiv: 2410.23891

24. Xu, F.; Shi, Y.; Ebel, P.; Yang, W.; Zhu, X.X. Multimodal and multiresolution data fusion for high-resolution coud removal: A novel baseline and benchmark. IEEE Trans. Geosci. Remote Sens.; 2023; 62, 5600615.

25. Shi, J.; Nie, M.; Ji, S.; Shi, C.; Liu, H.; Jin, H. Polarimetric synthetic aperture radar image classification based on double-channel convolution network and edge-preserving Markov random field. Remote Sens.; 2023; 15, 5458. [DOI: https://dx.doi.org/10.3390/rs15235458]

26. Wang, Z.; Wang, Z.; Qiu, X.; Zhang, Z. Global polarimetric synthetic aperture radar image segmentation with data augmentation and hybrid architecture model. Remote Sens.; 2024; 16, 380. [DOI: https://dx.doi.org/10.3390/rs16020380]

27. Wu, Y.H.; Ji, K.F.; Yu, W.X. Comparison of classification performance of full-, dual-and single-polarization SAR images using SVM. J. Remote Sens.; 2008; 12, 53.

28. Fukuda, S.; Hirosawa, H. Support vector machine classification of land cover: Application to polarimetric SAR data. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS); Sydney, NSW, Australia, 9–13 July 2001; Volume 1, pp. 187-189.

29. Moreira, A.; Prats-Iraola, P.; Younis, M.; Krieger, G.; Hajnsek, I.; Papathanassiou, K.P. A tutorial on synthetic aperture radar. IEEE Geosci. Remote Sens. Mag.; 2013; 1, pp. 6-43. [DOI: https://dx.doi.org/10.1109/MGRS.2013.2248301]

30. Chiang, C.Y.; Chen, K.S.; Chu, C.Y.; Chang, Y.L.; Fan, K.C. Color enhancement for four-component decomposed polarimetric SAR image based on a CIE-Lab encoding. Remote Sens.; 2018; 10, 545. [DOI: https://dx.doi.org/10.3390/rs10040545]

31. Erith, M.; Alfonso, Z.; Erik, L. A multi-sensor approach to separate palm oil plantations from forest cover using ndfi and a modified pauli decomposition technique. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS); Waikoloa, HI, USA, 26 September–2 October 2020; pp. 4481-4484.

32. Cloude, S.R.; Pottier, E. An entropy based classification scheme for land applications of polarimetric SAR. IEEE Trans. Geosci. Remote Sens.; 1997; 35, pp. 68-78. [DOI: https://dx.doi.org/10.1109/36.551935]

33. Ballester-Berman, J.D.; Lopez-Sanchez, J.M. Applying the Freeman–Durden decomposition concept to polarimetric SAR interferometry. IEEE Trans. Geosci. Remote Sens.; 2009; 48, pp. 466-479. [DOI: https://dx.doi.org/10.1109/TGRS.2009.2024304]

34. Krogager, E. New decomposition of the radar target scattering matrix. Electron. Lett.; 1990; 26, pp. 1525-1527. [DOI: https://dx.doi.org/10.1049/el:19900979]

35. Yamaguchi, Y.; Moriyama, T.; Ishido, M.; Yamada, H. Four-component scattering model for polarimetric SAR image decomposition. IEEE Trans. Geosci. Remote Sens.; 2005; 43, pp. 1699-1706. [DOI: https://dx.doi.org/10.1109/TGRS.2005.852084]

36. Yamaguchi, Y.; Sato, A.; Boerner, W.M.; Sato, R.; Yamada, H. Four-component scattering power decomposition with rotation of coherency matrix. IEEE Trans. Geosci. Remote Sens.; 2011; 49, pp. 2251-2258. [DOI: https://dx.doi.org/10.1109/TGRS.2010.2099124]

37. Zhou, Y.; Wang, H.; Xu, F.; Jin, Y.Q. Polarimetric SAR image classification using deep convolutional neural networks. IEEE Geosci. Remote Sens. Lett.; 2016; 13, pp. 1935-1939. [DOI: https://dx.doi.org/10.1109/LGRS.2016.2618840]

38. Zhang, L.; Chen, Z.; Zou, B.; Gao, Y. Polarimetric SAR terrain classification using 3D convolutional neural network. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS); Valencia, Spain, 22–27 July 2018; pp. 4551-4554.

39. Wang, H.; Xing, C.; Yin, J.; Yang, J. Land cover classification for polarimetric SAR images based on vision transformer. Remote Sens.; 2022; 14, 4656. [DOI: https://dx.doi.org/10.3390/rs14184656]

40. Chu, B.; Chen, J.; Chen, J.; Pei, X.; Yang, W.; Gao, F.; Wang, S. SDCAFNet: A deep convolutional neural network for land-cover semantic segmentation with the fusion of PolSAR and optical images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.; 2022; 15, pp. 8928-8942. [DOI: https://dx.doi.org/10.1109/JSTARS.2022.3213601]

41. La, Y.; Bagan, H.; Yamagata, Y. Urban land cover mapping under the Local Climate Zone scheme using Sentinel-2 and PALSAR-2 data. Urban Clim.; 2020; 33, 100661. [DOI: https://dx.doi.org/10.1016/j.uclim.2020.100661]

42. Lee, J.S.; Pottier, E. Polarimetric Radar Imaging: From Basics to Applications; CRC Press: Boca Raton, FL, USA, 2017.

43. Cornebise, J.; Oršolić, I.; Kalaitzis, F. Open high-resolution satellite imagery: The worldstrat dataset–with application to super-resolution. Adv. Neural Inf. Process. Syst.; 2022; 35, pp. 25979-25991.

44. Yan, J.; Xiaolan, Q.; Jie, P.; Songtao, S.G.; Zezhong, W.; Wei, W.; Hong, Y. MPOLSAR-1.0: Multidimensional SAR multiband fully polarized fine classification dataset. J. Radars.; 2024; 13, pp. 525-538.

45. Li, Z.; He, W.; Cheng, M.; Hu, J.; Yang, G.; Zhang, H. SinoLC-1: The first 1-meter resolution national-scale land-cover map of China created with the deep learning framework and open-access data. Earth Syst. Sci. Data; 2023; 15, pp. 4749-4780. [DOI: https://dx.doi.org/10.5194/essd-15-4749-2023]

46. Van De Kerchove, R.; Zanaga, D.; Keersmaecker, W.; Souverijns, N.; Wevers, J.; Brockmann, C.; Grosu, A.; Paccini, A.; Cartus, O.; Santoro, M. et al. ESA WorldCover: Global land cover mapping at 10 m resolution for 2020 based on Sentinel-1 and 2 data. Proceedings of the American Geophysical Union Fall Meeting (AGU); New Orleans, LA, USA, 13–17 December 2021; Volume 2021, GC45I-0915.

47. Karra, K.; Kontgis, C.; Statman-Weil, Z.; Mazzariello, J.C.; Mathis, M.; Brumby, S.P. Global land use/land cover with Sentinel 2 and deep learning. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS); Brussels, Belgium, 11–16 July 2021; pp. 4704-4707.

48. Gong, P.; Liu, H.; Zhang, M.; Li, C.; Wang, J.; Huang, H.; Clinton, N.; Ji, L.; Li, W.; Bai, Y. et al. Stable classification with limited sample: Transferring a 30-m resolution sample set collected in 2015 to mapping 10-m resolution global land cover in 2017. Sci. Bull.; 2019; 64, pp. 370-373. [DOI: https://dx.doi.org/10.1016/j.scib.2019.03.002]

49. Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Honolulu, HI, USA, 21–26 July 2017; pp. 1125-1134.

50. Meraner, A.; Ebel, P.; Zhu, X.X.; Schmitt, M. Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS J. Photogramm. Remote Sens.; 2020; 166, pp. 333-346. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2020.05.013] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32747852]

51. Wang, Y.; Zhang, B.; Zhang, W.; Hong, D.; Zhao, B.; Li, Z. Cloud removal with SAR-Optical data fusion using a unified spatial-spectral residual network. ISPRS J. Photogramm. Remote Sens.; 2023; 62, 5600820. [DOI: https://dx.doi.org/10.1109/TGRS.2023.3339210]

52. Xu, F.; Shi, Y.; Ebel, P.; Yu, L.; Xia, G.S.; Yang, W.; Zhu, X.X. GLF-CR: SAR-enhanced cloud removal with global–local fusion. ISPRS J. Photogramm. Remote Sens.; 2022; 192, pp. 268-278. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2022.08.002]

53. Mirza, M. Conditional generative adversarial nets. arXiv; 2014; arXiv: 1411.1784

54. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Las Vegas, NV, USA, 27–30 June 2016; pp. 770-778.