SAR Ship Detection Dataset (SSDD): Official Release and Comprehensive Data Analysis

Abstract

SAR Ship Detection Dataset (SSDD) is the first open dataset that is widely used to research state-of-the-art technology of ship detection from Synthetic Aperture Radar (SAR) imagery based on deep learning (DL). According to our investigation, up to 46.59% of the total 161 public reports confidently select SSDD to study DL-based SAR ship detection. Undoubtedly, this situation reveals the popularity and great influence of SSDD in the SAR remote sensing community. Nevertheless, the coarse annotations and ambiguous standards of use of its initial version both hinder fair methodological comparisons and effective academic exchanges. Additionally, its single-function horizontal-vertical rectangle bounding box (BBox) labels can no longer satisfy the current research needs of the rotatable bounding box (RBox) task and the pixel-level polygon segmentation task. Therefore, to address the above two dilemmas, in this review, advocated by the publisher of SSDD, we will make an official release of SSDD based on its initial version. SSDD’s official release version will cover three types: (1) a bounding box SSDD (BBox-SSDD), (2) a rotatable bounding box SSDD (RBox-SSDD), and (3) a polygon segmentation SSDD (PSeg-SSDD). We relabel ships in SSDD more carefully and finely, and then explicitly formulate some strict using standards, e.g., (1) the training-test division determination, (2) the inshore-offshore protocol, (3) the ship-size reasonable definition, (4) the determination of the densely distributed small ship samples, and (5) the determination of the densely parallel berthing at ports ship samples. These using standards are all formulated objectively based on the using differences of existing 75 (161 × 46.59%) public reports. They will be beneficial for fair method comparison and effective academic exchanges in the future. Most notably, we conduct a comprehensive data analysis on BBox-SSDD, RBox-SSDD, and PSeg-SSDD. Our analysis results can provide some valuable suggestions for possible future scholars to further elaborately design DL-based SAR ship detectors with higher accuracy and stronger robustness when using SSDD.

1. Introduction

Detecting ships in the vast ocean, trade-bustling rivers, and important ports is conducive to traffic control, trade activity monitoring, fishery surveillance, and defense deployment. Many scholars have proposed various ship detection methods using different sensors, e.g., optical satellites [1], multispectral satellites [2], video surveillance systems [3], and Synthetic Aperture Radar (SAR) satellites [4]. In contrast to the first three types, SAR can provide all-day and all-weather monitoring services, which is more suitable for monitoring ships due to the changeable marine climate on Earth.

Since the United States launched the first SAR satellite on 28 June 1978 [5], a variety of SAR ship detection methods have emerged [6] based, e.g., on constant false alarm rate (CFAR) [7], generalized likelihood ratio test (GLRT) [8], visual saliency [9], super-pixel segmentation [10], polarization decomposition [11], and some auxiliary features (e.g., oil spill clues and ships’ wake) [12,13]. At present, these methods are all attributed to traditional ones because they always need to design ship features manually. In general, there are four basic steps involved in them [14], i.e., (1) land masking, (2) preprocessing, (3) prescreening, and (4) discrimination. These four steps are often executed in turn in practical applications, troublesomely. However, for SAR images with different resolutions, sea states (e.g., winds and waves), and satellites, these traditional methods are usually not robust enough and are accompanied by troublesome model parameter adjustments. Worse still, their detection speed is also not fast enough to meet real-time application requirements.

With the rise of artificial intelligence (AI), more novel solutions begin to appear. Since AlexNet, a novel deep convolutional neural network (CNN), achieved victory in the 2012 ImageNet image classification competition, deep learning (DL) has been a new research direction in the field of machine learning (ML). It is leading ML to be closer to its original goal―AI. Nowadays, DL provides an increasing number of elegant solutions for various communities, e.g., computer vision (image classification and object detection), speech recognition, natural language processing, audio recognition, bioinformatics, and medical science. It is no exception for the SAR remote sensing community. For object detection in images, there have been many well-known CNN-based detectors that have end-to-end training and inference, including (1) two-stage detectors, e.g., the region-convolutional neural network (R-CNN) proposed by Girshick et al. [15] in 2014; the fast region-convolutional neural network (Fast R-CNN) proposed by Girshick et al. [16] in 2015; the faster region-convolutional neural network (Faster R-CNN) proposed by Ren et al. [17] in 2015; and the Cascade R-CNN proposed by Cai et al. [18] in 2018. Also (2) One-stage detectors, e.g., you only look once (YOLO) proposed by Redmon et al. [19] in 2016; single shot multibox detector (SSD) proposed by Liu et al. [20] in 2016; the RetinaNet proposed by Lin et al. [21] in 2017; and the anchor-free CenterNet proposed by Duan et al. [22] in 2019.

1.1. Deep Learning for SAR Ship Detection before SSDD

Taking SAR ship detection as an example, since the first public report was presented at the conference of IEEE International Geoscience and Remote Sensing Symposium (IGARSS) in Beijing, China on 3 November 2016 by C.P. Schwegmann et al. [23], a large number of studies based on DL have emerged for SAR ship detection. According to our statistics, before 1 December 2017, there were four public reports in the SAR remote sensing community that apply DL to SAR ship detection, including (1) three conference papers from the IGARSS and the International Workshop on Remote Sensing with Intelligent Processing (RSIP) and (2) one journal paper from Remote Sensing. See their list summary in Table 1.

In the report of C.P. Schwegmann et al. [23], inspired by the work of Srivastava et al. [27] from the computer vision community, a very deep High-Way CNN was established to achieve SAR ship discrimination, i.e., ruling out false alarms. This work was presented at the conference of IGARSS in Beijing, China. Their work applied DL to the last step of the traditional SAR ship detection workflow. Similarly, on 26 Junuary 2017, Liu et al. [24] designed a CNN to achieve ship discrimination. Their network contains two convolution layers, two maximum pooling layers, and three fully connected layers. Their workflow is inspired by R-CNN, proposed by Girshick et al. [15] in 2014. Although the above two reports drew support from the DL technology, they both still designed their detection methods according to the standard four-step process. That is, CNN is only used to accomplish the ship-background binary classification task. From their reports, the binary classification accuracy of the proposed CNN achieved a huge process compared with several traditional classifiers, e.g., decision trees, AdaBoost, and support vector machine (SVM). Nevertheless, end-to-end detection was not completed in their work. Their methods are still troublesome.

In the report of Kang et al. [25], the famous Faster R-CNN [17] was first applied to SAR ship detection. Moreover, the detection outputs of Faster R-CNN were modified according to their corresponding classification confidences or scores. Here, the bounding box with a score lower than 0.2 was transferred into CFAR to detect again to avoid the missed detection. They hold the view that Faster R-CNN always missed small ships because its deeper layers offered little small-ship information due to multiple max-pooling operations. Without exaggeration, this work poses a significant impact on the follow-up scholars’ researches. If not considering the post-processing of CFAR, their work can achieve the full end-to-end training and test, avoiding the traditional four-step process. In particular, the exemption of the sea-land segmentation step can greatly reflect the greatest advantage of DL. Later, Kang et al. [26] also further improved the network structure of Faster R-CNN, i.e., adding contextual information in the region of interest (ROI) and pooling and generating proposals in multiple layers with different depths. As a result, their experimental results on the Sentinel-1 SAR images revealed that their modified version could detect more ships and remove more false alarms. This work reaches the goal of end-to-end SAR ship detection.

Regretfully, the four reports in Table 1 do not all provide public training datasets for later possible scholars. This hinders the development of DL in the field of SAR ship detection. It has been extensively shown that a huge amount of data is needed for DL to be effective. Only big training data can ensure DL networks learn target features deeply and accurately. Before 1 December 2017, there were no open datasets dedicated to SAR ship detection based on DL available.

1.2. Initial Release of SSDD

Fortunately, on 1 December 2017, Jianwei Li, a co-author of this article, made his own collected dataset, i.e., SAR Ship Detection Dataset (SSDD), publicly available to everyone at the conference of SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA) in Beijing, China [28]. SSDD is the first open dataset in this community. See the list summary of SSDD in Table 2. Detailed descriptions of SSDD are shown in Table 3.

In his report [28], he applied the classical and famous two-stage detector Faster R-CNN to complete the SAR ship detection task. In addition to this, he also proposed four strategies to further improve the performance of the standard Faster RCNN algorithm when it is used for SAR ship detection, including feature fusion, transfer learning, hard negative mining, and other optimized implementation details. Finally, his proposed Improved-Faster R-CNN enhanced the detection performance by ~8% mean average precision (mAP) on the SSDD dataset.

In fact, the report of Li et al. [28] merely released an initial coarse version of SSDD. Surprisingly, this first open dataset SSDD gained unprecedented attention from quite a number of scholars, which was beyond the author’s imagination. According to our investigation from 2016 to 25 August 2021 (the completion time of this manuscript), after the release of SSDD, 46.59% of the total 161 public reports confidently choose SSDD to study DL-based SAR ship detection. Obviously, this situation reveals the popularity and great influence of SSDD in the SAR remote sensing community.

These 75 (161 × 46.59%) public reports [28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102] that used SSDD will be shown in Section 2. Here, only reports in English are recorded. See the pie chart in Figure 1. In Figure 1, the others include (1) the self-collected databases and (2) the other five open datasets in Table 4, i.e., SAR-Ship-Dataset released by Wang et al. in 2019 [103], AIR-SARShip-1.0 released by Sun et al. in 2019 [104], HRSID released by Wei et al. in 2020 [63], LS-SSDD-v1.0 released by Zhang et al. in 2020 [105], and AIR-SARShip-2.0 [106].

1.3. Success of SSDD

We hold the view that the tremendous success of SSDD is due to the following seven factors:

• The public time of SSDD is the earliest. It is older than the second open dataset SAR-Ship-Dataset by ~1.5 years. When no other datasets are available, SSDD becomes the only option.

• Many countries or organizations have launched various SAR satellites. Several frequently used satellites for SAR ship detection include Sentinel-1 from the European Space Agency (ESA) [107], Gaofen-3 from China [108], TerraX-SAR from Germany [109], COSMO-SkyMed from Italy [110], ALOS from Japan [111], and Kompsat-5 from South Korea [112]. They are all commercial satellites. Except for Sentinel-1, users need to pay to download data, increasing research and development costs. However, the resolutions of Sentinel-1 are modest. In Sentinel-1 SAR images, ships are universally small, with unclear geometric features. However, the emergence of SSDD can solve the above dilemma.

• The publisher of SSDD is active in the SAR remote sensing research community. Some public media platforms promote the dissemination of this dataset.

• The SAR image samples in SSDD are various with different resolutions from 1m to 15 m, different sensors from RadarSat-2, TerraSAR-X, and Sentinel-1, different polarizations (HH, VV, VH, and HV), different sea conditions, different ship scenes, including inshore and offshore, and different ship sizes. Data diversity is one of the major issues in building reliable regressive/predictive detection models. See Table 3.

• When several reports using SSDD appeared, follow-up scholars chose to experiment on this SSDD dataset in order to facilitate the comparison of methodologies with previous scholars. As a result, there are gradually increasing public reports using this SSDD dataset.

• In the early stage, the GPU computing power of computers used by most scholars in the SAR remote sensing community was limited. The sample number of SSDD is relatively moderate, i.e., 1160, compared with large-scale datasets in the computer vision community, e.g., ~9000 images in the PASCAL VOC dataset [113] and ~20 w images in the COCO dataset [114]. This reduces the equipment cost of studies. This enables ordinary researchers equipped with general performance GPUs to carry out research and development. This point enables the community, using the SSDD dataset to study DL-based SAR ship detection, to become rather active. As a result, the increase of researchers may greatly lead to the increase of research results. Moreover, a relatively moderate sample number also facilitates the debugging of models, improving work efficiency, rather than a long time of training waiting. Of course, when using the SSDD dataset, some few-shot strategies, e.g., data augmentation and transfer learning, should be considered so as to avoid overfitting.

• There are typical hard-detected samples in SSDD. These samples all need special consideration in the practical application of SAR ship detection, e.g., (1) small ships with inconspicuous features, (2) densely parallel ships berthing at ports with overlapping ship hulls, (3) large scale-difference ships, (4) ship detection under severe speckle noise, (5) ship detection under complex, and (6) multiple types of sea clutters. Ship detection in these difficult samples is a research hotspot, regardless of traditional hand-crafted methods or modern DL-based methods. Therefore, SSDD can provide a possible data source to study these focus issues.

1.4. Motivations of This Review

Nevertheless, the coarse annotations and ambiguous using standards of SSDD’s initial version have hindered fair methodological comparisons and effective academic exchanges. Firstly, there are some coarse annotations in the initial version, e.g., missed ship annotations, false ship annotations, and not compact bounding boxes. Therefore, the initial version is “dirty”. The phenomenon of dirty data is widespread in the field of computer vision. For huge data, deep networks can reduce the negative influence of dirty data through batch training to improve the generalization ability of models. However, for the few-shot SAR data, a training oscillation may occur in deep networks, which will lower the detection performance. Therefore, it is necessary to correct them. Some scholars [51,52] have corrected them partly, but their revised labels are not publicly available. Secondly, in the original conference report of SSDD [28], the using standards are ambiguous, even unreasonable. For example, the training-test division is random, but the cardinality of the test set affects the resulting accuracy due to too few samples [90]. This results in unfair methodological comparisons with different scholars. Moreover, the inshore–offshore protocol was not provided in his raw report, leading to an unfair detection accuracy comparison between inshore ships and offshore ones by later scholars. More importantly, at the moment, there is still a lack of comprehensive data analysis of this dataset. This is not conducive to further research by other scholars.

Moreover, the single-function horizontal-vertical rectangle bounding box (BBox) labels of SSDD’s initial version can no longer satisfy the current research needs of both the rotatable bounding box (RBox) task and the pixel-level polygon segmentation (PSeg) one. Horizontal-vertical BBox is not suitable for ships with large aspect ratios and arbitrary orientations. Furthermore, ships at ports are too closely packed to be effectively distinguished, thereby resulting in missing detections [62]. There are lots of background clutters in the BBox, reducing the ship feature learning benefits. On the contrary, RBox can better describe the true shape of the target while providing better accuracy in ship detection. Therefore, some scholars [31,36,46,56,59,62,79,80,91] have employed RBox to detect ships in SSDD. Here, the rotatable bounding box ground truths are labeled by themselves, but these RBox labels are not publicly available. Moreover, PSeg is the highest-level task because it is based on pixel-level. Obviously, SAR ship detection using PSeg is the most ideal because PSeg can almost completely suppress background clutter. Up to now, several scholars [52,54,98,99] have drawn support from it to achieve SAR ship detection. PSeg ground truths are labeled by themselves, but they are not publicly available, too.

Therefore, to address the above two dilemmas, advocated by the publisher of SSDD, i.e., a co-author of this review, Jianwei Li, we will make an official release of SSDD based on its initial version [28]. The official release version of SSDD will cover three types―(1) bounding box SSDD (BBox-SSDD), (2) rotatable bounding box SSDD (RBox-SSDD), and (3) polygon segmentation SSDD (PSeg-SSDD). These re-released three types of ship ground truths labels will be convenient for future scholars to use, according to different task requirements. With the participation of many researchers, we re-label SAR ships in SSDD more carefully and finely. Furthermore, we explicitly formulate some strict using standards for the sake of fair methodological comparisons and effective academic exchanges, including (1) the training-test division determination, (2) the inshore–offshore protocol, (3) the ship-size reasonable definition, (4) the determination of the densely distributed small ship samples, and (5) the determination of the densely parallel berthing at ports ship samples. To be clear, the determinations of these using standards are based on the using differences of the existing 75 public reports [28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102].

Most notably, we also conduct a comprehensive data analysis on BBox-SSDD, RBox-SSDD, and PSeg-SSDD, which is missing in its initial release version. Our analysis results will be able to provide some valuable suggestions for possible future scholars to further elaborately design DL-based SAR ship detectors with higher accuracy and stronger robustness when using SSDD. We expect that this review will be useful for relevant scholars who are studying SAR ship detection based on DL.

The main contributions of this review are summarized as follows:

• The official version of SSDD is released, including three types: BBox-SSDD, RBox-SSDD, and PSeg-SSDD. It will be convenient for future scholars to use according to different task requirements.

• A comprehensive data analysis on BBox-SSDD, RBox-SSDD, and PSeg-SSDD is conducted. It can provide some valuable suggestions for possible future scholars to further design DL-based SAR ship detectors with higher accuracy and stronger robustness when using SSDD.

• Some more reasonable and stricter using standards are formulated objectively based on the using differences of existing 75 (161 × 46.59%) public reports. We also provide some potential solutions to improve the detection performance of difficult-detected ship samples.

The rest of this review is arranged as follows. Section 2 provides a summary of public reports using SSDD. This is convenient for scholars to consult, analyze and summarize the regularity. Section 3 shows the official ground truth ship labels in SSDD, including BBox, RBox, and PSeg. This can provide potential scholars with a comprehensive browse without querying in the data directory. Section 4 introduces the data directory of SSDD. In the data directory, we also provide some useful tools in Python language for user-friendly service. Data analysis is presented in Section 5. In this section, we provide some valuable suggestions when using SSDD. Using standards are listed in Section 6. This will contribute to the comparison of reasonable methods in the future. Finally, Section 7 summarizes the whole article.

Last but not least, the official release version of SSDD (i.e., BBox-SSDD, RBox-SSDD, and PSeg-SSDD) is available at https://github.com/TianwenZhang0825/Official-SSDD (accessed on 25 August 2021).

2. Summary of Public Reports Using SSDD

Table 5 shows the summary list of 75 public reports using SSDD. In Table 5, all public reports are sorted according to the order of their publication time. In order to facilitate readers’ browsing, the public reports of the same publication year are marked as uniform color blocks. The number of public publications has increased year by year.

Moreover, we provide some statistical analysis in Figure 2. From Figure 2a, the number of public publications has increased year by year. The highest growth rate was in 2019. The growth rate slows down slightly in 2020, which may be because of the influence of the successive releases of other datasets. By 25 August 2021 (the completion time of this manuscript), there have been 24 public reports using SSDD. As one can imagine, there must be >28 public reports from 2020 that will appear throughout 2021. From Figure 2b, the studies using SSDD are active in both peer-review journals and conferences, showing SSDD’s great influence. From Figure 2c, except for one conference report from the Indian scholars’ Anil Raj et al. [87], all reports are from Chinese scholars. This may be because (1) the publisher of SSDD Jianwei Li is from China; (2) SSDD was firstly released at the conference of the BIGSARDATA in Beijing, China; (3) it was in Beijing, China, at the 2016 IGARSS conference where the first report from C.P. Schwegmann et al. [23] applied DL to SAR ship detection, although they [23] did not use SSDD. Therefore, China has become the most active country or region in research.

From Figure 2d, SSDD appears in a variety of mainstream remote sensing journals, e.g., MDPI Remote Sensing (RS), IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JSTARS), IEEE Transactions on Geoscience and Remote Sensing (TGRS), and IEEE Geoscience and Remote Sensing Letters (GRSL). This shows that SSSD is widely accepted by academics in the remote sensing community.

Finally, we also counted the label types used among these 75 public reports in Figure 3, i.e., BBox, RBox, and PSeg. As shown in Figure 3, there are 62 papers using Bbox and only 9 using RBox; there are only four papers using PSeg. The initial release version of SSDD only provided BBox labels. This makes the number using BBox account for the majority (82.7%). However, Bbox is not compact enough for ship detection, and there is a lot of background clutter in it, resulting in insufficient ship feature extraction. Therefore, later, Wang et al. [31], An et al. [36,80], Chen et al. [46], Pan et al. [56], Yang et al. [59,79], Chen et al. [62], and He et al. [91] used RBox to detect ships in SAR ships. Their methods were inspired by the rotational text detection in the deep learning community, e.g., R2CNN [115] and EAST [116]. Compared with RBox, PSeg is better because it is based on the pixel level. Obviously, SAR ship detection using PSeg is the most ideal because PSeg can almost thoroughly suppress the background clutter. So far, there are only four papers adopting these kinds of labels, including Su et al. [52], Mao et al. [54], Sun et al. [98], and Wu et al. [99]. It should be noted that only one of the four realizes the ship segmentation authentically, i.e., Su et al. [52]. The other three merely used the PSeg labels to guide the ship detection of BBox. Specifically, Mao et al. [54] generated semantic feature maps to generate the attention score so as to obtain better ship BBoxs. Sun et al. [98] designed a semantic attention-based network for inshore ship detection. In this network, PSeg labels were used to suppress cruciform sidelobes. Wu et al. [99] established the interaction between instance segmentation and object detection. Finally, the BBox detection accuracy on inshore scenes is improved greatly.

The fact that there are rather few scholars using RBox and PSeg labels for SAR ship detection comes from the lack of ground truths. The initial release version of SSDD did not provide the two. This defect hinders more research by relevant scholars when using RBox and PSeg. Motivated by this, therefore, this article will handle this defect.

3. SSDD

In this section, we will display the ship ground truths and descript label formats of BBox-SSDD, RBox-SSDD, and PSeg-SSDD in Section 3.1, Section 3.2, and Section 3.3, respectively. In order to facilitate the reader’s overall browsing, all images in the test set (i.e., images with the name suffix of 1 and 9) are displayed.

3.1. BBox-SSDD

Figure A3 in Appendix A shows the bounding box ground truths in the test set of BBox-SSDD. The inshore samples are marked in magenta.

Figure 4 shows the label format of BBox-SSDD, where we use the PASCAL VOC format to explain. The initial release version of SSDD adopted this format. One BBox is described by two points in the x-y image coordinates, i.e., A(xmin, ymin) and B(xmax, ymax). Thus, the width (w) and height (h) of the BBox are

(1)$w=xmax-xmin$

(2)$h=ymax-ymin$

Some CNN-based detectors adopt some other descriptions of BBox, e.g., (xmin, ymin, w, h) of YOLO [19]. Users can flexibly convert them according to the labels we provide. Of course, we also provide conversion tools in Python language. In fact, this review can be used as a detailed tutorial of SSDD for beginners. To be clear, in Figure 4a, the image channel number is the default 3, where 3 refers to RGB. Considering that SAR images only have a gray-level channel, we copy this channel twice to obtain RGB 3-channel images. This is to facilitate the use of detectors in the field of computer vision. Moreover, the current ImageNet pre-training weights are all based on 3-channel inputs. According to experience, using the pre-training weights can accelerate the convergence speed of networks and reduce the risk of overfitting with fine-tuning. Of course, one can extract the data of one channel to use for designing the “specialized” SAR ship detectors from scratch. This practice is worth advocating since it can eliminate learning bias between different data domains.

3.2. RBox-SSDD

Figure A1 in Appendix A shows the rotational bounding box ship ground truths in the test set of RBox-SSDD. The inshore samples are marked in magenta. In Figure A1, boxes are rectangles, not parallelograms. Some do not look like right angles because images are scaled (the aspect ratio is not maintained) for tidiness and beauty.

Figure 5 shows the label format of RBox-SSDD. The initial release version of SSDD did not provide such labels for scholars. One RBox can be described by four vertices, i.e., A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4). We call this scheme four-point type. Here, the four corners of a quadrilateral are right angles, i.e., ∠A = ∠B = ∠C = ∠D = 90°. Thus, the w and h of RBox are

(3)$w=\sqrt{{\left(x2-x3\right)}^2+{\left(y2-x3\right)}^2}$

(4)$h=\sqrt{{\left(x1-x2\right)}^2+{\left(y1-x2\right)}^2}$

The above representation may be the most intuitive, but some detectors adopt different coding modes. Therefore, we also provide another description mode of one RBox, i.e., $\left(x, y,w,h, \theta \right)$ in Figure 5a, where x, y is the coordinate of the center P, and $\theta$ is the direction angle. We call this scheme center-angle type. In fact, this center-angle type is equivalent to four-point one.

In RBox-SSDD, $\theta \in \left[0°,90°\right]$ is based on the angle system rather than the radian system. Of course, one can convert it easily into the radian system according to application requirements. Furthermore, $\theta$ is the angle between the principal ship axis and negative y-axis. It is more in line with human visual habits, based on the counter-clockwise direction. One can also use $90°-\theta$ to represent the ship direction angle.

It should be noted that Wang et al. [31], An et al. [36,80], Chen et al. [46], Pan et al. [56], Yang et al. [59,79], Chen et al. [62], and He et al. [91] used RBox to detect ships in SAR ships, but their angle estimation results are discrete, e.g., 30°, 60°, 90°, and so on. Their angular clearance is too large to describe the direction of the ship finely. However, the angle labels we provide are the continuous float-type rather than the discrete int-type, which will lead to better direction estimation accuracy. Obviously, RBox-SSDD has the potential to promote greater progress for ship direction estimation in the future. Scholars need to design stronger regression models to achieve such fine angle regression.

Given the above, according to the labels we provide, scholars can use RBox-SSDD more flexibly.

3.3. PSeg-SSDD

Figure A2 in Appendix A shows the ship polygon segmentation labels in the test set of PSeg-SSDD. The inshore samples are marked in magenta. In Figure 3, different ships are covered with different colors for clarity. The outline of the ship polygon is marked in green. From Figure A2, obviously, it is the most satisfactory to adopt polygon segmentation labels to detect ships. This task is at the pixel level, similar to traditional CFAR detections.

Figure 6 shows the label format of PSeg-SSDD. The initial release version of SSDD did not provide such labels for scholars. From Figure 6, there are a series of points used to describe the outline of the ship. These points can be connected into a closed polygon. We use a famous and open annotation tool, LabelMe [117], to obtain these points. To be clear, due to scale differences of ships in SSDD, ships with different scales or sizes must result in different numbers of outline points. Large ships offer more points while small ones offer fewer points. Moreover, in Figure 6b, the order of these points is counterclockwise.

Based on PSeg-SSDD, one can study the semantic segmentation task by using a fully convolutional network (FCN) [118] or U-Net [119]. Semantic segmentation refers to classify each pixel in the image. For SAR ship detection, that will be a ship-background binary pixel-level classification task. If one pixel is predicted with a “1” label, then it will be regarded as one ship pixel. On the contrary, for a “0” label, it is the background. Taking the PASCAL VOC dataset format as an example, we provide the semantic segmentation mask in Figure 7. In Figure 7b, the black regions in the mask belong to background pixels meanwhile the green ones belong to ship pixels. Because there is only one ship category, the colors in the mask are only black and green. This is different from the 20 colors in the PASCAL VOC dataset.

Furthermore, based on PSeg-SSDD, one can study the instance segmentation task by using Mask R-CNN [120] or PANet [121]. Instance segmentation means that the machine automatically frames different instances from the image with the object detection method and then marks them pixel by pixel in different instance areas with the semantic segmentation method.

In fact, the labels of polygon segmentation can act as labels for BBox and RBox. Suppose the labels of polygon segmentation are denoted as the set L_PSeg,

(5)$L_{PSeg}=\left\{\left(x_1,y_1\right),\left(x_2,y_2\right),\cdots ,\left(x_n,y_n\right)\right\}$

where there are n points make up this set L to describe this ship. Therefore, the labels of BBox L_BBox can be obtained by

(6)$L_{BBox}=\left\{\left(x_{min},y_{min},x_{max,}{y}_{max}\right) | x_{min}=\underset{i=1,2,\cdots n}{\min }\left\{x_i\right\},y_{min}=\underset{i=1,2,\cdots n}{\min }\left\{y_i\right\},x_{max}=\underset{i=1,2,\cdots n}{\max }\left\{x_i\right\},y_{max}=\underset{i=1,2,\cdots n}{\max }\left\{y_i\right\}\right\}$

The above formula is obvious because a BBox is indeed the smallest circumscribed horizontal rectangle of a PSeg. Therefore, for the instance segmentation task, the training label is shown in Figure 8.

In Figure 8, each ship is given a polygon outline label and a horizontal rectangular box at the same time. For the same ship, the color of the polygon outline label and the horizontal rectangular box is the same. This shows that the two types of labels represent the same ship. Moreover, the colors of different ships are different because they are different instances, determined by the definition of the instance segmentation.

Finally, the labels of RBox L_RBox can be obtained from the PSeg labels L_PSeg. The former is the smallest circumscribed rectangle of the latter, i.e.,

(7)$$\begin{gathered} L_{RBox}=\left(x,y,w,h,\theta \right) \\ {\displaystyle where \left(x,y\right),\left(w,h\right),\theta =cv2.minAreaRect\left(L_{PSeg}\right)} \end{gathered}$$

where cv2 is the python-opencv package, and minAreaRect is the function to calculate the smallest circumscribed rectangle. The input of minAreaRect is a point set, and the output is a list. The first element of this list is a tuple (x, y), i.e., the center coordinate of the box. The second element is a tuple (w, h), i.e., the width and height of the box. The last one is a float-type number $\theta$, i.e., the ship direction angle ranging from 0° to 90°.

4. Data Directory

Figure 9 is the diagrammatic sketch of the data directory. In the root directory (Official-SSDD), there are four folders―BBox-SSDD (labels of BBox), RBox-SSDD (labels of RBox), PSeg-SSDD (labels of PSeg), and BBox-RBox-PSeg-SSDD (labels of BBox, RBox, and PSeg) provided to study the “all-in-one” ship detection. We provide two types of annotations―the PASCAL VOC format (voc_type) [113] and the COCO format (coco_type) [114].

In Figure 9, due to limited pages, we only expanded the specific contents of the first folder (BBox-SSDD) to display. From Figure 9, we also provide the SAR images with ship ground truths in the JPEGImages_BBox_GT. Moreover, it should be noted that after further update and maintenance in the future, the data directory may change slightly.

5. Data Analysis

The initial reports of SSDD lack comprehensive data analysis. This is not conducive to the further research of follow-up scholars. Therefore, in this review, it will be complemented. Frist, we will analyze the image size (width and height) distribution in the following parts. Afterward, we will make a specific and detailed data analysis on BBox-SSDD, RBox-SSDD, and PSeg-SSDD, respectively, in Section 5.1, Section 5.2 and Section 5.3.

The size of the sample image in the dataset has a significant impact on the final detection performance. Generally, a deep network often needs a fixed image input size to maintain the unity of feature dimensions, e.g., SSD-300’s input dimension is 300 × 300, SSD-512 is 512 × 512 [20], and YOLO is 416 × 416 or other multiples of 32 [19]. When the images in the dataset are not uniform, one usually needs to use some image interpolation methods to resize the original image. Therefore, the dimension of the resized scale is rather important. This size should preferably be determined according to the image size distribution of the dataset. For example, in the computer vision community, the default input size of the COCO dataset is 1333 × 800, which is an empirical optimal value adopted by a wide range of scholars. If this resized scale is too small, the information of many ships will be lost after a series of convolutions, resulting in their missed detections. If this resized scale is too large, the parameters of the network will increase rapidly, resulting in slow convergence speed, longer training time, and sacrificing the computational cost.

Nowadays, when using SSDD, many scholars in the SAR ship detection community adopt different input sizes in their different networks, e.g., 160 × 160 in Zhang et al. [65], 300 × 300 in Wang et al. [31], 500 × 500 in Jian et al. [89], 512 × 512 in Zhang et al. [73], 600 × 600 in Yu et al. [97], 600 × 1000 in Wei et al. [51], and so on. This leads to an unreasonable comparison of methods in accuracy. Generally, for the same network, the accuracy of 300 × 300 is often inferior to 500 × 500 or larger sizes. Therefore, it is necessary to make a detailed analysis of the image size. Moreover, an appropriate input size is also conducive to the design of better data enhancement methods, e.g., the work of Yang et al. [122].

Figure 10 shows the SAR image sample size statistics in SSDD. The three types of datasets share the same analysis results. In Figure 10, the test set is an image set with their suffix is 1 or 9. See the samples marked in magenta. The rest constitutes the training set. As a result, the training-test ratio is 8:2, i.e., 928 training samples and 232 samples. In Figure 10, we investigated the width-height distribution of images in Figure 10a and the ratio between width and height in Figure 10b.

From Figure 10a, the following conclusions can be drawn:

• The sample sizes in the SSDD dataset are quite different. Taking the entire dataset as an example, the smallest width is 214 pixels, while the largest one is 668 pixels. Their difference has tripled. The smallest height is 160 pixels, while the largest one is 526 pixels. Their difference has tripled too.

• The widths of images are widely larger than the heights from the green lines. Therefore, we hold the view that it may be unreasonable to directly stretch the height of the image to its width size. Otherwise, the aspect ratio of the original ship in the SAR image must change, which is a typical violation of the SAR imaging mechanism.

• Many samples share the same 500-pixel width, i.e., a strong cluster width = 500, but they do not share the same height. Their heights range from ~200-pixel to ~500-pixel. It is noteworthy that the mean, median, and mode values are all located at width = 500.

From Figure 10b, the following conclusions can be drawn:

• The ratio between the image width and height reflects a normal distribution. The ratio with the highest frequency was ~1.4. Therefore, we hold the view that it is better to maintain this ratio during image pre-processing because this can minimize the information loss caused by pre-processing.

• The aspect ratio of images has an extreme tailing effect at both ends of the histogram. Therefore, scholars can consider clipping the image with an extremely differentiated aspect ratio so as to realize the normalization of network input. In fact, this practice can also serve as data enhancement.

Furthermore, from Figure 10, the image scale distribution on the entire dataset, the training set, and the test set remains roughly unchanged. This shows that the dataset partition mechanism we customized is reasonable. In essence, DL is indeed to fit the distribution of samples on the training set. Consequently, it can be generalized to the test set, which shares a similar distribution to the training set.

Finally, we suggest the resized image scale as 500 × 350 pixels because it is closer to the median values of the width and the height Figure 10a. In addition, the ratio is ~1.4 (500/350), which is in line with the mean value of Gaussian distribution in Figure 10b.

5.1. Data Analysis on BBox-SSDD

Figure 11 shows the data statistics results on the BBox-SSDD dataset. We have investigated the distribution of the BBox width and height shown in Figure 11a, the distribution of the ratio between the BBox width and height in Figure 11b, the distribution of the area of the BBox in Figure 11c, and the distribution of the BBox center coordinates (x, y) in Figure 11d where x = (xmin + xmax)/2 and y = (ymin + ymax)/2 from Figure 4a.

From Figure 11a, the following conclusions can be drawn:

• Ships in SSDD are universally small. The width-height distribution of BBox presents a symmetrical funnel shape. There are more ships at the top of the funnel and fewer ships in the center of the funnel. This shows that SAR ships are rarely square (the green line), which is also reasonable because ships are always flat. The average size of ships is only ~ 35 × 35 pixels. It is extremely difficult to detect such small ships. Thus, scholars should pay special attention to this phenomenon.

• The reason why the ship size distribution presents a symmetrical structure based on the diagonal is that the breadth and length of the ship are not completely distinguished in the image coordinate system. Sometimes, the BBox width and height are confused.

From Figure 11b, the aspect ratio of BBox ships is seriously unbalanced. The aspect ratio of most ships is less than 1. Some ships with extreme aspect ratios are easy to miss detection because of the scarcity of training data. Scholars can use the 90° rotation data enhancement to alleviate this problem. Moreover, one can also selectively only enhance the data of ships with a large length-width ratio so as to realize the learning balance of the network.

From Figure 11c, we can more clearly find that the SAR ship is small. However, the upper limit of the BBox areas is rather large. In other words, some ships are extremely large in size, but their number is extremely small. For example, the number of areas > 50,000 is only 4 in the entire dataset. It is possible that these ships have the highest resolution because the area of BBox is calculated according to the number of pixels. The area of the same ship in high-resolution images is larger than that in low-resolution images. This huge cross-scale ship detection is a challenging task. In order to realize the multi-scale ship detection under the condition of multi-resolution, scholars can appropriately down-sample the large ships to improve the benefit of multi-scale feature learning of the network. Moreover, for too small ships, it can be improved by up-sampling interpolation to avoid their information loss in the network. A potential solution to small ship detection is to design a network to super-resolution reconstruct small ships under low resolution so as to expand the characteristics of small ships.

From Figure 11c, the positions of ships in images are completely random, so scholars need to be careful when using random slicing to realize data enhancement.

5.2. Data Analysis on RBox-SSDD

Figure 12 shows the data statistics results on the RBox-SSDD dataset. We have investigated the distribution of the RBox width and height shown in Figure 12a, the distribution of the ratio between the RBox width and height in Figure 12b, the distribution of the area of the RBox in Figure 12c, the distribution of the RBox center coordinates (x, y) in Figure 12d (see Figure 5a), and the distribution of the RBox angle $\theta$ in Figure 12e.

.

From Figure 12a, compared with the BBox size distribution in Figure 11a, the size distribution of the ship is closer to the edge of the funnel. In other words, almost no ships are square, which is in line with common sense. BBox ships appear diagonally because the direction of the ship is around 45°, so the width of BBox is equal to the height. This indicates that RBox is more advanced than BBox, and the former can better depict the real ship.

From Figure 12b, the distribution rule of the width-height ratio of RBox is different from that of BBox. When the aspect ratio is about 3, a small crest appears. This is because RBox can successfully identify the long and short sides of ships, so the aspect ratio is mostly greater than 1. However, RBox is still powerless for very small ships because the total number of pixels of the ship is too small. From Figure 12c, similar conclusions can be drawn to Figure 11c. From Figure 12d, similar conclusions can be drawn to Figure 12d. Because of the limited pages, we will not repeat the introduction.

Only RBox provided the direction theta $\theta$ distribution in Figure 12e. From Figure 12e, the angle distribution of ships presents a bowl shape with high ends and low middle. In other words, most ships maintain the relevant vertical and horizontal state in SAR images. However, some ships parking at ports may enable large direction angles. Based on this phenomenon, we can roughly identify inshore ships. Then, we can contrapuntally design the network to improve the detection accuracy of inshore ships. Moreover, if scholars want to improve the multi-directional detection performance of detectors, we hold the view that ships with strong angle identification (the bowl bottom in Figure 12e) should be rotated to generate more samples so as to avoid the angle imbalance learning.

5.3. Data Analysis on PSeg-SSDD

Figure 13 shows the data statistics results on the PSeg-SSDD dataset. We have investigated the distribution of the PSeg area shown in Figure 12a, the distribution of the PSeg perimeter in Figure 12b, and the distribution of the proportion of the PSeg area among the whole image in Figure 12c.

In Figure 13a, in line with previous statements, the number of large ships is far smaller than that of small ships. This quantitative imbalance will inevitably make it difficult for networks to learn the characteristics of large ships effectively. As a result, the detection accuracy of large ships is lower than that of small ships. Quantitative comparison results can be found in the work of Mao et al. [54]. This phenomenon seems contrary to common sense because, in fact, small ships are harder to detect usually. One can reduce the resolution of the large ship image by down-sampling to improve the detection performance of large ships. This is because, in this way, the rare large ships may become medium-sized ships so as to avoid the huge scale imbalance learning of the network.

In Figure 13b, the distribution of the PSeg perimeter is a little different from that of the PSeg area based on the visual observation from the highest value. In other words, the larger the area of a ship, it does not necessarily mean the larger its perimeter. This is caused by the flat shape of the ship. Moreover, the existence of speckle noise may affect the statistics of the actual perimeter of the ship because the noise will make the edge of the ship more uneven, thus increasing the contact area between the ship and the ocean. Therefore, when using SSDD, one should better consider suppressing speckle noise. Some previous work in Zhang et al. [55] and Chen et al. [39] considered this problem.

From Figure 13c, SAR ships always account for a very small proportion in the whole image, most <4%. This vividly shows the characteristics of the “bird’s-eye” view of SAR. Therefore, SAR remote sensing images are different from the optical images of natural scenes that have a “person’s-eye” view. This shows that it is not feasible to directly apply deep learning detectors from the field of computer vision to SAR ship detection. Scholars should design the network according to the characteristics of SAR so as to achieve purposeful ship detection rather than generic object detection [123].

6. Using Standards

So far, 75 public reports have used SSDD to study DL-based SAR ship detection, but there are no unified standards for the way they use SSDD because the initial open report of SSDD did not provide them. This situation will hinder fair methodological comparison and effective academic exchanges. Therefore, we will explicitly formulate some strict using standards, e.g., (1) the training-test division determination in Section 6.1, (2) the inshore–offshore protocol in Section 6.2, (3) the ship-size reasonable definition in Section 6.3, (4) the determination of the densely distributed ship samples in Section 6.4, and (5) the determination of the densely parallel berthing at ports ship samples in Section 6.5.

6.1. Training-Test Division Determination

The original report of SSDD in [28] adopted the random ratio of 7:1:2 to divide the dataset into a training set, a validation set, and a test set. However, this random division mechanism leads to great uncertainty in the samples in the test set. For example, using the same detector for multiple training and testing by a random division leads to different accuracy results due to the great uncertainty of training samples. This is because the number of samples in SSDD is too small, only 1160. In this case, the random partition may destroy the distribution consistency between the training set and the test set.

Later, some scholars also adopted other ratios for training, validation, and testing, e.g., 7:2:1 in the work of Chen et al. [62], about 5:1 in the work of Yu et al. [97], 7:3 in the work of Wu et al. [99], 8:2 in the work of Chen et al. [90], and so on. Obviously, these diverse dataset division mechanisms will lead to unfair methodological comparison, which is not conducive to academic exchanges. This problem was also revealed by Zhang et al. [57,73] and Chen et al. [90].

In fact, in the field of computer vision, two well-known datasets for object detection, i.e., the PASCAL VOC [113] and COCO datasets [114], both provide the only determined training set, verification set, and test set, which is also to ensure the fairness of their competition. Later, inspired by this practice, the publisher of the AIR-SARShip-1.0 dataset [104] provided the unique training set and test set files. Therefore, inspired by these works, we make strict regulations on the division of training set and test set of SSDD, as in Table 6.

From Table 6, the images with the last digits of the file number 1 and 9 are uniquely determined as the test set, and the rest are regarded as the training set. Such a rule can also maintain the distribution consistency of the training set and test set, which is conducive to network feature learning. More information on distribution consistency can be found in the work of Han et al. [66,67,68]. Moreover, the official released SSDD does not provide the unique validation set. Scholars can extract some images from the training set to form a verification set according to their own needs. We only care about the fair accuracy comparison when the test set is exactly the same.

Finally, according to our experience, we suggest that scholars do not set up a verification set because it sacrifices the learning gain of the network. In short, the number of samples in SSDD is very small, so we should cherish each sample to ensure that the training gradient of each test sample is reduced. However, the verification set does not participate in the training gradient descent, which will inevitably lead to insufficient ship feature learning. Of course, if researchers want to monitor whether the model has been overfitted in the training process, they can set up multiple overlapping cross-verification sets to achieve the purpose.

6.2. Inshore-Offshore Protocol

Many previous reports focused on inshore ship detection, e.g., Wei et al. [51], Su et al. [52], Yang et al. [59], Zhang et al. [73], and so on. Ships landing on the shore are easily interfered with by port facilities, and the land backgrounds in the image are more complex. Nowadays, inshore SAR ship detection is a research hotspot. In order to ensure the fairness of accuracy in these two different scenarios, we also uniquely determined the inshore and offshore files tested. Inshore images are marked in magenta in Figure A1.

According to statistics, among the 232 test images, there are 186 offshore scene images, while there are only 46 inshore scene images. The proportion between offshore and inshore scenes is shown in Figure 14.

Similar to HRSID [63] and LS-SSDD-v1.0 [105], we regard images containing land as inshore samples and others as offshore samples. From Figure 14, the numbers of inshore samples and offshore ones are hugely imbalanced (19.8% VS. 80.2%). This phenomenon seems to accord with the fact that the ocean area of the Earth is much larger than land. However, DL needs a lot of data to learn features. More data often brings better learning benefits; less data is bound to cause inadequate learning. Thus, the sample number imbalance between the offshore scene and inshore one will bring about a huge imbalance of models’ learning representation capacity between offshore and inshore. Networks will be trapped in many easy offshore samples. The detection performance of inshore ships will become poor due to fewer training samples; meanwhile, that of offshore ships will become excellent due to more samples.

In the future, scholars should pay special attention to the above problem when designing detectors. Several reports, e.g., the balance scene learning mechanism in the work of Zhang et al. [73] and the visual attention mechanism in the work of Chen et al. [46], can provide some valuable suggestions. We hold the view that one can design a classifier to realize scene recognition and then carry out selective-scene data enhancement to achieve balanced scene learning. Moreover, an interesting report from Chen et al. [90] proposed to mix-up stitches multiple rotating ships into one image and mosaic combines four original images into one image, which can improve the detection performance of inshore scenes. This work is inspired by YOLOv4 [124]. This method is rather useful and can also avoid network training falling into a large number of useless pure background negative samples, i.e., no ships in the image [105].

6.3. Ship-Size Definition

Multi-scale ship detection is a challenging task because different types of ships have different sizes, and if the same ship has different resolutions, it will also lead to a change in the total number of pixels in the image. However, so far, there is still no clear definition of which ships are small ships and which ships are large ships in SAR images. Some scholars believe that ships with less than 40 pixels are small ships, but they do not take into account the actual resolution of the image. Moreover, it is not consistent with the consensus in the computer vision community to determine the size definition of ships simply according to the number of pixels.

In the SAR ship detection community, Wei et al. [51], Su et al. [52], and Mao et al. [54] followed the standard of the COCO dataset to classify ship sizes, i.e., the area of BBox < 32² means a small ship, the area of BBox < 96² but >32² means a medium ship, and the area of BBox > 96² means a large ship. However, this definition is tailored only for the COCO dataset, and it may be problematic to use it on the SSDD dataset. It does not match the area distribution of BBox very well, as shown in Figure 11c, Figure 12c and Figure 13a.

Therefore, it is better to specify the ship size definition according to the SSDD dataset we use. Moreover, we should also define the ship size according to different label types.

Finally, according to Figure 15, we define the ship size standard in Table 7. Here, according to the statistical histogram of the label area, we respectively distribute the large, medium, and small ships.

6.4. Densely Distributed Small Ship Samples

Densely distributed small ships are difficult to detect because the characteristics are not clear. Many scholars have focused on this issue. In order to facilitate the accuracy evaluation specifically in this specific scenario, we specify the samples of densely distributed small ships in the test set as in Figure 16. Among the 232 test images, there are 10 images that have densely distributed small ships. In Figure 16, according to our experience, the ships in the 001119.jpg are the most difficult to detect. Scholars should pay more attention to it.

Moreover, for these kinds of difficult samples, we provide several potential solutions:

• One can use random crop data enhancement to increase the proportion of small ships in the whole image.

• One can detect small ships in the shallow layer of the deep network with low feature loss.

• One can combine CFAR into the deep network because CFAR is more pixel-sensitive.

• One can combine visual saliency theory to generate a saliency map to guide deep network learning features because these small ships are very significant in human-eye observation.

• One can design a deep network to super-resolution reconstruct the small ship. In this way, the features of small ships will become richer.

6.5. Densely Parallel Berthing at Ports Ship Samples

Ships densely parallel berthing at ports are also rather difficult to detect. On the one hand, the very complex land background will reduce the training efficiency because there will be a large number of negative samples generated during training. On the other hand, ships moored side by side will produce hull overlap effects because of SAR’s special imaging mechanism and limited resolution. In order to facilitate the accuracy evaluation specifically in this specific scenario, we specify the samples of parallel berthing at ports ships in the test set as in Figure 17.

Moreover, for these kinds of difficult samples, we provide several potential solutions:

• One can use the attention mechanism to suppress the land interference so as to focus on the ship region.

• One can use a segmentation mask to assist ship detection, e.g., the reports of Mao et al. [54], Hou et al. [84], and Wu et al. [99].

• One can use generative adversarial networks (GAN) [125] to generate more samples of such scenes so as to improve the learning proportion of these ships, e.g., the work of Jiang et al. [95].

• One can use the soft-NMS post-processing algorithm [126] to avoid missed detections, e.g., the reports of Wei et al. [51] and Zhang et al. [101].

7. Conclusions

This article reviews the current usage status of the first open SSDD dataset in the SAR ship detection community. We release the official version of SSDD, covering three types: BBox-SSDD, RBox-SSDD, and PSeg-SSDD. We have made a detailed analysis of the three datasets when applied to different tasks. We comprehensively summarize the differences between ship detection in the SAR remote sensing community and general object detection in the computer vision community, which will help future scholars to design more purposeful detectors combined with the characteristics of SAR. Furthermore, we explicitly formulate some strict using standards for the sake of fair methodological comparisons and effective academic exchanges, including (1) the training-test division determination, (2) the inshore-offshore protocol, (3) the ship-size reasonable definition, (4) the determination of the densely distributed small ship samples, and (5) the determination of the densely parallel berthing at ports ship samples. These using standards can provide a fair methodological comparison. We also put forward many valuable suggestions to improve the detection accuracy of difficult samples for possible future scholars. We expect that this review will be useful for relevant scholars who are studying DL-based SAR ship detection. We hope this review could also serve as a careful and useful introductory tutorial for beginners who are preparing to study SAR ship detection based on DL.

Finally, we will develop an online evaluation system for benchmarks on SSDD. Researchers of this field can submit their results for a completely fair evaluation.

Author Contributions

Conceptualization, J.L. and H.S.; methodology, T.Z. (Tianwen Zhang); software, X.X. and C.L.; validation, B.W.; formal analysis, X.Z. (Xu Zhan); investigation, Y.X. and Y.Z.; resources, X.K. and D.P.; data curation, T.Z. (Tianjiao Zeng); writing—original draft preparation, T.Z. (Tianwen Zhang); writing—review and editing, X.Z. (Xiaoling Zhang) and I.A.; visualization, H.S. and Y.Z.; supervision, J.S.; project administration, S.W.; funding acquisition, X.Z. (Xiaoling Zhang). All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National Natural Science Foundation of China under Grants 61571099.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Acknowledgments

The authors would like to thank the editors and anonymous reviewers for their valuable comments that can greatly improve our manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Figure A1

Rotatable bounding boxes or ground truths in the test set of RBox-SSDD. The inshore samples are marked in magenta.

[Figure omitted. See PDF]

[Figure omitted. See PDF]

[Figure omitted. See PDF]

Figure A2

Polygon segmentation labels in the test set of PSeg-SSDD. The inshore samples are marked in magenta.

[Figure omitted. See PDF]

[Figure omitted. See PDF]

Figure A3

Bounding box ground truths in the test set of BBox-SSDD. The inshore samples are marked in magenta.

[Figure omitted. See PDF]

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Pie chart of the used proportion of SSDD among all DL-based SAR ship detection public reports. There are 161 public reports using DL for SAR ship detection. Among them, there are 75 that used SSDD as their study data source, i.e., 46.6%.

Figure 2. Statistical analysis of 75 public reports using SSDD. (a) Number of public reports according to publication year; (b) distribution proportion between journal reports and conference ones; (c) national and regional distribution of researchers; (d) distribution of peer-reviewed journals.

Figure 3. Statistical distribution of label types (i.e., BBox, RBox, and PSeg) used in existing 75 public reports.

Figure 4. The label of BBox-SSDD where we take the PASCAL VOC format to introduce. (a) BBox in a SAR image; (b) xml annotation details.

Figure 5. The label of RBox-SSDD where we take the PASCAL VOC format to introduce. (a) RBox in a SAR image; (b) xml annotation details.

Figure 6. The label of PSeg-SSDD where we take the PASCAL VOC format to introduce. (a) PSeg in a SAR image; (b) xml annotation details.

Figure 7. Semantic segmentation of PSeg-SSDD. (a) An original SAR image; (b) its semantic segmentation mask.

Figure 8. Instance segmentation of PSeg-SSDD. (a) An original SAR image; (b) its instance segmentation annotation.

Figure 9. Diagrammatic sketch of the data directory. We provide two types of annotations, i.e., the PASCAL VOC format (voc_type) [113] and the COCO format (coco_type) [114]. It should be noted that after further update and maintenance in the future, the data directory may change slightly.

Figure 10. SAR image sample size statistics in SSDD. (a) Distribution of the image width and height; (b) distribution of the ratio between the image width and height.

Figure 10. SAR image sample size statistics in SSDD. (a) Distribution of the image width and height; (b) distribution of the ratio between the image width and height.

Figure 11. Data statistics results on the BBox-SSDD dataset. (a) Distribution of the BBox width and height; (b) distribution of the ratio between the BBox width and height; (c) distribution of the BBox area; (d) distribution of the BBox ship center x and center y.

Figure 12. Data statistics results on the RBox-SSDD dataset. (a) Distribution of the RBox width and height; (b) distribution of the ratio between the RBox width and height; (c) distribution of the RBox area; (d) distribution of the RBox ship center x and center y. (e) distribution of the RBox theta θ.

Figure 12. Data statistics results on the RBox-SSDD dataset. (a) Distribution of the RBox width and height; (b) distribution of the ratio between the RBox width and height; (c) distribution of the RBox area; (d) distribution of the RBox ship center x and center y. (e) distribution of the RBox theta θ.

Figure 13. Data statistics results on the PSeg-SSDD dataset. (a) Distribution of the PSeg area; (b) distribution of the PSeg perimeter; (c) distribution of area ratio of the PSeg ship among the whole image.

Figure 14. Proportion of offshore and inshore scenes.

Figure 15. Area distribution of different label types. (a) BBox; (b) RBox; (c) PSeg.

Figure 16. Densely distributed small ship samples.

Figure 17. Densely parallel berthing at ports ship samples.

Table 1

List summary of four public reports using deep learning (DL) before 1 December 2017.

No.	Publication Date †	Authors	Title	Journal/Conference ††
1	2016-11-03	C.P. Schwegmann et al. [23]	Very deep learning for ship discrimination in Synthetic Aperture Radar imagery	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
2	2017-06-26	Liu et al. [24]	SAR ship detection using sea-land segmentation-based convolutional neural network	International Workshop on Remote Sensing with Intelligent Processing (RSIP)
3	2017-06-26	Kang et al. [25]	A modified Faster R-CNN based on CFAR algorithm for SAR ship detection	International Workshop on Remote Sensing with Intelligent Processing (RSIP)
4	2017-08-20	Kang et al. [26]	Contextual region-based convolutional neural network with multilayer fusion for SAR ship detection	Remote Sensing

^† First public time online. ^†† Conferences are italicized, and journals are not italicized. Note that different colors mean different years.

Table 2

List summary of SSDD.

No.	Publication Date †	Authors	Title	Journal/Conference ††
1	2017-12-01	Li et al. [28]	Ship detection in SAR images based on an improved Faster R-CNN	SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA)

^† First public time online. ^†† Conferences are italicized, and journals are not italicized.

Table 3

Detailed descriptions of SSDD [28].

Sensors	RadarSat-2, TerraSAR-X, Sentinel-1
Polarization	HH, VV, VH, HV
Resolution	1 m–15 m
Places	Yantai, China; Visakhapatnam, India
Scale	1:1, 1:2, 2:1
Ship	Different sizes and materials
Sea condition	Good and bad conditions
Scenes	Inshore and offshore
Image number	1160
Ship number	2456
These descriptions are in line with the initial presentation of SSDD.

Table 4

Summary list of six open datasets for SAR ship detection.

No.	Dataset	Publication Date †	Authors	Title	Journal/Conference/Website ††
1	SSDD	2017-12-01	Li et al. [28]	Ship detection in SAR images based on an improved Faster R-CNN	SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA)
2	SAR-Ship-Dataset 1	2019-03-29	Want et al. [103]	A SAR dataset of ship detection for deep learning under complex backgrounds	Remote Sensing
3	AIR-SARShip-1.0 2	2019-12-01	Sun et al. [104]	AIR-SARShip-1.0: High-resolution SAR Ship Detection Dataset	Journal of Radars
4	HRSID 3	2020-06-29	Wei et al. [63]	HRSID: A high-resolution SAR images dataset for ship detection and instance segmentation	IEEE Access
5	LS-SSDD-v1.0 4	2020-09-15	Zhang et al. [105]	LS-SSDD-v1.0: A deep learning dataset dedicated to small ship detection from large-scale Sentinel-1 SAR images	Remote Sensing
6	AIR-SARShip-2.0 5	--	Sun et al. [106]	--	http://radars.ie.ac.cn/web/data/getData?dataType=SARDataset_en&pageType=en (accessed on 25 August 2021)

^† First public time online. ^†† Conferences and websites are italicized, and journals are not italicized. ¹ https://github.com/CAESAR-Radi/SAR-Ship-Dataset (accessed on 25 August 2021). ² http://radars.ie.ac.cn/web/data/getData?newsColumnId=d25c94d7-8fe8-415f-a897-cb88657a8141&pageType=en (accessed on 25 August 2021). ³ https://github.com/CAESAR-Radi/SAR-Ship-Dataset (accessed on 25 August 2021). ⁴ https://github.com/TianwenZhang0825/LS-SSDD-v1.0-OPEN (accessed on 25 August 2021). ⁵ http://radars.ie.ac.cn/web/data/getData?newsColumnId=74fe223a-0b01-4830-8d99-1ba276e67ad8&pageType=en (accessed on 25 August 2021).

Table 5

Summary list of 75 public reports using SSDD.

No.	Publication Date †	Authors	Title	Journal/Conference ††
1	2017-12-01	Li et al. [28]	Ship detection in SAR images based on an improved Faster R-CNN	SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA)
2	2018-03-09	Chen et al. [29]	Robust single stage detector based on two-stage regression for SAR ship detection	International Conference on Innovation in Artificial Intelligence (ICIAI)
3	2018-04-10	Jiao et al. [30]	A densely connected end-to-end neural network for multiscale and multiscene SAR ship detection	IEEE Access
4 ■	2018-08-29	Wang et al. [31]	Simultaneous ship detection and orientation estimation in SAR images based on attention module and angle regression	Sensors
5	2019-03-05	Liu et al. [32]	Multi-scale proposal generation for ship detection in SAR images	Remote Sensing
6	2019-03-05	Gui et al. [33]	A multilayer fusion light-head detector for SAR ship detection	Sensors
7	2019-04-02	Chang et al. [34]	Ship detection based on YOLOv2 for SAR imagery	Remote Sensing
8	2019-05-21	Zhang et al. [35]	High-speed ship detection in SAR images based on a grid convolutional neural network	Remote Sensing
9 ■	2019-06-26	An et al. [36]	Drbox-v2: An improved detector with rotatable boxes for target detection in SAR images	IEEE Transactions on Geoscience and Remote Sensing
10	2019-07-15	Cui et al. [37]	Dense attention pyramid networks for multi-scale ship detection in SAR images	IEEE Transactions on Geoscience and Remote Sensing
11	2019-07-20	Chen et al. [38]	SAR ship detection under complex background based on attention mechanism	Image and Graphics Technologies and Applications (IGTA)
12	2019-07-24	Chen et al. [39]	A deep neural network based on an attention mechanism for SAR ship detection in multiscale and complex scenarios	IEEE Access
13	2019-08-05	Gui et al. [40]	A scale transfer convolution network for small ship detection in SAR images	IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC)
14	2019-08-21	Li et al. [41]	Small targets recognition in SAR ship image based on improved SSD	IEEE International Conference on Signal, Information and Data Processing (ICSIDP)
15	2019-09-23	Zhang et al. [42]	A lightweight feature optimizing network for ship detection in SAR image	IEEE Access
16	2019-10-07	Yang et al. [43]	SAR image target detection and recognition based on deep network	SAR in Big Data Era: Models, Methods and Applications (BIGSARDATA)
17	2019-10-24	Zhang et al. [44]	Depthwise separable convolution neural network for high-speed SAR ship detection	Remote Sensing
18	2019-10-24	Chen et al. [45]	Regional attention-based single shot detector for SAR ship detection	IET International Radar Conference (IRC)
19 ■	2019-11-04	Chen et al. [46]	MSARN: A deep neural network based on an adaptive recalibration mechanism for multiscale and arbitrary-oriented SAR ship detection	IEEE Access
20	2019-11-14	Wang et al. [47]	An improved Faster R-CNN based on MSER decision criterion for SAR image ship detection in harbors	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
21	2019-11-14	Li et al. [48]	SAR ship detection based on Resnet and transfer learning	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
22	2019-11-14	Li et al. [49]	Multiscale ship detection based on dense attention pyramid network in SAR images	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
23	2019-11-14	Liu et al. [50]	Scale-transferrable pyramid network for multi-scale ship detection in SAR images	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
24	2020-01-02	Wei et al. [51]	Precise and robust ship detection for high-resolution SAR imagery based on HR-SDNet	Remote Sensing
25 ■■	2020-03-19	Su et al. [52]	HQ-ISNet: High-quality instance segmentation for remote sensing imagery	Remote Sensing
26	2020-03-30	Su et al. [53]	Ship detection based on RetinaNet-plus for high-resolution SAR imagery	Asia-Pacific Conference on Synthetic Aperture Radar (APSAR)
27 ■■	2020-04-03	Mao et al. [54]	Efficient low-cost ship detection for SAR imagery based on simplified U-Net	IEEE Access
28	2020-04-16	Zhang et al. [55]	High-speed ship detection in SAR images by improved YOLOv3	International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP)
29 ■	2020-04-20	Pan et al. [56]	MSR2N: Multi-stage rotational region-based network for arbitrary-oriented ship detection in SAR images	Sensors
30	2020-04-22	Zhang et al. [57]	ShipDeNet-20: An only 20 convolution layers and <1-MB lightweight SAR ship detector	IEEE Geoscience and Remote Sensing Letters
31	2020-04-30	Dai et al. [58]	A novel detector based on convolution neural networks for multiscale SAR ship detection in complex background	Sensors
32 ■	2020-05-25	Yang et al. [59]	A novel false alarm suppression method for CNN-based SAR ship detector	IEEE Geoscience and Remote Sensing Letters
33	2020-05-28	Zhao et al. [60]	Attention receptive pyramid network for ship detection in SAR images	IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
34	2020-06-23	Han et al. [61]	Analysis of detection preference to CNN based SAR ship detectors	Information Communication Technologies Conference (ICTC)
35 ■	2020-06-24	Chen et al. [62]	R2FA-Det: Delving into high-quality rotatable boxes for ship detection in SAR images	Remote Sensing
36	2020-06-29	Wei et al. [63]	HRSID: A high-resolution SAR images dataset for ship detection and instance segmentation	IEEE Access
37	2020-07-07	Fu et al. [64]	An anchor-free method based on feature balancing and refinement network for multiscale ship detection in SAR images	IEEE Transactions on Geoscience and Remote Sensing
38	2020-07-21	Zhang et al. [65]	HyperLi-Net: A hyper-light deep learning network for high-accurate and high-speed ship detection from Synthetic Aperture Radar imagery	ISPRS Journal of Photogrammetry and Remote Sensing
39	2020-08-21	Han et al. [66]	Small ship detection in SAR images based on modified SSD	IEEE International Conference on Signal, Information and Data Processing (ICSIDP)
40	2020-08-21	Han et al. [67]	Asymmetric and square convolutional neural network for SAR ship detection from scratch	International Conference on Biomedical Signal and Image Processing (ICBIP)
41	2020-08-31	Han et al. [68]	Multi-size convolution and deep learning network for SAR ship detection from scratch	IEEE Access
42	2020-10-06	Zhou et al. [69]	Lira-YOLO: A lightweight model for ship detection in radar images	Journal of Systems Engineering and Electronics
43	2020-10-14	Mao et al. [70]	An anchor-free SAR ship detector with only 1.17M parameters	International Conference on Aviation Safety and Information Technology (ICASIT)
44	2020-10-14	Mao et al. [71]	Network slimming method for SAR ship detection based on knowledge distillation	International Conference on Aviation Safety and Information Technology (ICASIT)
45	2020-10-30	Han et al. [72]	Asymmetric convolution-based neural network for SAR ship detection from scratch	International Conference on Computing and Pattern Recognition (ICCPR)
46	2020-11-16	Zhang et al. [73]	Balance scene learning mechanism for offshore and inshore ship detection in SAR images	IEEE Geoscience and Remote Sensing Letters
47	2020-12-02	Li et al. [74]	A lightweight Faster R-CNN for ship detection in SAR images	IEEE Geoscience and Remote Sensing Letters
48	2020-12-03	Zhu et al. [75]	Rapid ship detection in SAR images based on YOLOv3	International Conference on Communication, Image and Signal Processing (CCISP)
49	2020-12-04	Zhang et al. [76]	Balanced feature pyramid network for ship detection in Synthetic Aperture Radar images	IEEE Radar Conference (RadarConf)
50	2020-12-07	Chen et al. [77]	Learning slimming SAR ship object detector through network pruning and knowledge distillation	IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
51	2020-12-28	Guo et al. [78]	A Centernet++ model for ship detection in SAR images	Pattern Recognition
52 ■	2021-01-08	Yang et al. [79]	A novel CNN-based detector for ship detection based on rotatable bounding box in SAR images	IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
53 ■	2021-02-03	An et al. [80]	Transitive transfer learning-based anchor free rotatable detector for SAR target detection with few samples	IEEE Access
54	2021-02-03	Zhao et al. [81]	A dense connection based SAR ship detection network	IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC)
55	2021-02-03	Mao et al. [82]	Ship detection for SAR imagery based on deep learning: A benchmark	IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC)
56	2021-02-12	Wang et al. [83]	Soft thresholding attention network for adaptive feature denoising in SAR ship detection	IEEE Access
57	2021-02-17	Hou et al. [84]	SAR image ship detection based on scene interpretation	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
58	2021-02-17	Zhang et al. [85]	ShipDeNet-18: An only 1 MB with only 18 convolution layers light-weight deep learning network for SAR ship detection	IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
59	2021-03-01	Kun et al. [86]	SAR image ship detection based on deep learning	International Conference on Computer Engineering and Intelligent Control (ICCEIC)
60	2021-03-01	Anil Raj et al. [87]	A novel ship detection method from SAR image with reduced false alarm	International Conference on Computational Intelligence and Energy Advancements (ICCIEA)
61	2021-03-11	Li et al. [88]	A novel multidimensional domain deep learning network for SAR ship detection	IEEE Transactions on Geoscience and Remote Sensing
62	2021-03-17	Jin et al. [89]	An approach on image processing of deep learning based on improved SSD	Symmetry
63	2021-03-23	Chen et al. [90]	End-to-end ship detection in SAR images for complex scenes based on deep CNNs	Journal of Sensors
64 ■	2021-03-24	He et al. [91]	Learning polar encodings for arbitrary-oriented ship detection in SAR images	IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
65	2021-03-31	Tian et al. [92]	Image enhancement driven by object characteristics and dense feature reuse network for ship target detection in remote sensing imagery	Remote Sensing
66	2021-04-05	Li et al. [93]	SAR image nearshore ship target detection in complex environment	IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC)
67	2021-04-27	Zhao et al. [94]	Ship detection from scratch in Synthetic Aperture Radar (SAR) images	International Journal of Remote Sensing
68	2021-05-13	Jiang et al. [95]	High-speed lightweight ship detection algorithm based on YOLO-v4 for three-channels RGB SAR image	Remote Sensing
69	2021-06-09	Zhu et al. [96]	An effective ship detection method based on RefineDet in SAR images	International Conference on Communications, Information System and Computer Engineering (CISCE)
70	2021-06-30	Yu et al. [97]	TWC-Net: A SAR ship detection using two-way convolution and multiscale feature mapping	Remote Sensing
71 ■■	2021-06-30	Sun et al. [98]	Semantic attention-based network for inshore SAR ship detection	International Conference on Digital Image Processing (ICDIP)
72 ■■	2021-07-01	Wu et al. [99]	A deep detection network based on interaction of instance segmentation and object detection for SAR images	Remote Sensing
73	2021-07-13	Sun et al. [100]	DSDet: A lightweight, densely connected sparsely activated detector for ship target detection in high-resolution SAR images	Remote Sensing
74	2021-07-14	Zhang et al. [101]	Quad-FPN: A novel quad feature pyramid network for SAR ship detection	Remote Sensing
75	2021-07-26	Sun et al. [102]	An anchor-free detection method for ship targets in high-resolution SAR images	IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

^† First public time online. ^†† Conferences are italicized, and journals are not italicized. ^■ Rotatable bounding box (RBox) is used. ^■■ Polygon segmentation (PSeg) is used. Note that different colors mean different year.

Table 6

Training-test division.

	Training Set	Test Set
Image file name	* 0.jpg, * 2.jpg, * 3.jpg, * 4.jpg, * 5.jpg, * 6.jpg, * 7.jpg, * 8.jpg,	* 1.jpg, * 9.jpg
Number	928	232
Training-test ratio	8:2

* n.jpg means that the last digit of the document number is n, e.g., * 0.jpg means 000010.jpg, 000020.jpg, etc.

Table 7

Ship-size definition.

Label Type	Small	Medium	Large
BBox-SSDD	Area < 1000	1000 <= Area <= 8000	Area > 8000
RBox-SSDD	Area < 625	625 <= Area <= 7500	Area > 7500
PSeg-SSDD	Area < 625	625 <= Area <= 6875	Area > 6875