MMPW-Net: Detection of Tiny Objects in Aerial Imagery Using Mixed Minimum Point-Wasserstein Distance

Abstract

The detection of distant tiny objects in aerial imagery plays a pivotal role in early warning, localization, and recognition tasks. However, due to the scarcity of appearance information, minimal pixel representation, susceptibility to blending with the background, and the incompatibility of conventional metrics, the rapid and accurate detection of tiny objects poses significant challenges. To address these issues, a single-stage tiny object detector tailored for aerial imagery is proposed, comprising two primary components. Firstly, we introduce a light backbone-heavy neck architecture, named the Global Context Self-Attention and Dense Nested Connection Feature Extraction Network (GC-DN Network), which efficiently extracts and fuses multi-scale features of the target. Secondly, we propose a novel metric, MMPW, to replace the Intersection over Union (IoU) in label assignment strategies, Non-Maximum Suppression (NMS), and regression loss functions. Specifically, MMPW models bounding boxes as 2D Gaussian distributions and utilizes the Mixed Minimum Point-Wasserstein Distance to quantify the similarity between boxes. Experiments conducted on the latest aerial image tiny object datasets, AI-TOD and VisDrone-19, demonstrate that our method improves AP50 performance by 9.4% and 5%, respectively, and AP performance by 4.3% and 3.6%. This validates the efficacy of our approach for detecting tiny objects in aerial imagery.

1. Introduction

Tiny objects pervade numerous application domains in the real world, encompassing military surveillance, agricultural pest control, driving assistance, and maritime rescue, among others. Despite significant advances in object detection fueled by the evolution of deep-learning and neural networks [1], most efforts have been directed toward detecting objects of regular sizes in natural images. However, for aerial imagery captured by unmanned aerial vehicles (UAVs) [2], tiny objects often hold greater research significance and urgency, constituting the most noteworthy aspects within an image. In the recently proposed AI-TOD dataset [3] for aerial tiny object detection, tiny objects are defined as those smaller than 12 pixels, typically exhibiting blurred boundaries and extremely limited appearance information. This poses a formidable challenge in learning discriminative features, leading to a substantial number of failures in detecting these tiny objects.

Recent progress in tiny object detection has primarily focused on enhancing feature discrimination capabilities. Several endeavors have introduced super-resolution reconstruction techniques to elevate the resolution of small objects and their corresponding features, thereby addressing the scale issue. Generative Adversarial Networks [4] (GANs) have been extensively utilized to directly generate super-resolved representations of small objects. Additionally, Feature Pyramid Networks [5] (FPNs) have been proposed to learn multi-scale features, aiming to alleviate the poor performance of conventional object detectors on micro-objects. While these existing methods have indeed improved the performance of TOD to some extent, the enhancement in accuracy often comes at the cost of additional computational overhead, rendering them less suitable for applications requiring rapid responses to small objects. Furthermore, as the model depth increases, the features of small targets gradually diminish. Taking YOLOv5 [6] as an example, the receptive field of the feature map (80 × 80) corresponding to the original drone image (5630 × 4314) is 71, meaning that targets smaller than 71 pixels cannot learn effective features, as illustrated in Figure 1.

Beyond the learning of discriminative features, the quality of training sample selection plays a pivotal role in anchor-based micro-object detectors, where the assignment of positive and negative (pos/neg) labels is of utmost importance [7]. However, for tiny objects, the scarcity of feature pixels exacerbates the challenge of training sample selection. As depicted in Figure 1, we can observe a pronounced variation in IoU sensitivity across different object scales. Specifically, for a regular small object of 8 × 8 pixels, a positional deviation of just 2 pixels results in a drastic drop in IoU from 0.62 to 0.24, a reduction of 38.7%. For an extremely tiny object of 4 × 4 pixels, the same 2-pixel deviation leads to an even more precipitous decline in IoU from 0.391 to 0.032, representing an 8.1% retention, thereby causing inaccurate label assignment. Notably, the sensitivity of IoU stems from the inherent discreteness of bounding box position changes.

This phenomenon underscores that the IoU metric ceases to be scale-invariant for objects with discrete positional deviations, ultimately leading to two notable deficiencies in label assignment. Specifically, IoU thresholds (θp, θn) are employed in anchor-based detectors to allocate pos/neg training samples, with (0.7, 0.3) typically used in Region Proposal Networks [8] (RPNs). Firstly, the IoU’s heightened sensitivity to tiny objects means that minute positional deviations can flip anchor labels, fostering similarity between positive and negative sample features and hindering network convergence. Secondly, it has been widely observed that the average number of positive samples assigned to each ground truth (gt) in micro-object datasets is significantly lower than what would be expected using the IoU metric alone. This occurs because the IoU between some gts and any anchor falls below the minimum positive threshold. Consequently, the supervisory information for training micro-object detectors becomes inadequate. While several approaches have introduced novel ideas, such as dynamic allocation strategies like ATSS [9], which can adaptively derive IoU thresholds for pos/neg label assignment based on the statistical characteristics of objects, the sensitivity of IoU poses a challenge in finding optimal thresholds that can provide high-quality pos/neg samples for tiny object detectors. The intricacies of IoU sensitivity hinder the consistent delivery of sufficient and relevant training data, thereby limiting the performance of tiny object detection systems.

Given the ineffectiveness of IoU and its derivatives in accurately measuring tiny objects, this paper introduces a novel metric, termed the Mixed Minimum Point-Wasserstein Distance (MMPW), to assess the similarity between bounding boxes, superseding the standard IoU. Furthermore, we develop a lightweight, single-stage micro-object detection framework based on this novel metric. Specifically, we initially model bounding boxes as two-dimensional Gaussian distributions. Subsequently, we employ the proposed Normalized Mixed Minimum Point-Wasserstein Distance (MMPW) to quantify the similarity between these derived Gaussian distributions. The primary advantage of the MMPW distance lies in its comprehensive incorporation of all relevant factors considered in existing loss functions, namely overlap, non-overlap regions, center point distances, and aspect ratio deviations. Additionally, it simplifies the computational process while also enabling the measurement of distributional similarities. Importantly, the MMPW distance exhibits scale invariance, rendering it particularly suitable for measuring the similarity between tiny objects. Consequently, it addresses the limitations of IoU-based metrics, which struggle with the sensitivity issues associated with tiny objects. By leveraging this innovative metric, our framework offers an effective solution for detecting tiny objects with enhanced accuracy and robustness.

Motivated by the aforementioned challenges, we design MMPW-Net, a single-stage detector based on MMPW with Global Context Self-Attention and Dense Nested Connection. Notably, MMPW serves as a substitute for IoU not only in label assignment but also in NMS and loss functions. Extensive experiments on the new TOD datasets, AI-TOD and VisDrone-19, reveal that MMPW-Net achieves the highest and fastest detection performance with minimal parameters. In summary, the key contributions of our work are as follows:

• MMPW Metric: We analyze the sensitivity of IoU to position deviations in tiny objects and propose MMPW as a superior metric to measure the similarity between bounding boxes. MMPW is applied to label assignment, NMS, and loss functions.

• GC-DN Network: We devise a GC-DN network with a lightweight backbone (GC-Backbone), incorporating global self-attention and a dense connection neck (DN-Neck). This architecture fuses high-level semantic and low-level spatial features, mitigating the issue of missed detections due to the loss of small object features in deep networks.

• MMPW-Net Architecture: By integrating the aforementioned components into a generic single-stage object detector architecture and augmenting it with a tiny object detection layer, we form MMPW-Net. On the AI-TOD and VisDrone-19 datasets, MMPW-Net achieves mAP accuracies of 54.8 and 51.3, respectively.

2. Related Research

2.1. Airborne Object Detection Dataset

In recent years, numerous object detection datasets in aerial imagery have been introduced, such as NWPU VHR-10 [10], HRSC2016 [11], VEDAI [12], xView [13], DOTA [14], VisDrone-19 [15], UA-VDT [16], and DIOR [17], to advance research in geo-vision object detection. However, the diverse scales of objects in these datasets render them more suitable for evaluating detectors designed for multi-scale object detection rather than micro-object detection. While some works on micro-object detection in aerial images have utilized aerial datasets (e.g., VEDAI [12] and R²-CNN [18]), VEDAI focuses solely on vehicle detection, and the dataset used in R²-CNN remains unpublished. The method and experiments presented in this paper are both conducted on the new TOD datasets, AI-TOD [3] and VisDrone-19 [15].

The AI-TOD [3] dataset, tailored for evaluating the performance of multi-class tiny object detection, encompasses eight distinct object categories, with an impressive 86% of its instances measuring less than 16 pixels. This comprehensive dataset is meticulously constructed upon several prominent and publicly available large-scale aerial imagery datasets, including the DOTA-V.5 [14] training set, xView [13] training set, VisDrone [15] Det training set, Airbus Ship Training set, and the DIOR [17] Training + Test set.

VisDrone-19 [15], another widely utilized dataset for object detection in drone-captured aerial imagery, features images with dimensions of 2000 × 1500 pixels. On average, each image in this dataset contains 53 instances, with a significant proportion of these objects being smaller than 32 pixels. The specific details and usage of the above two datasets are elaborated in Section 4.1.

2.2. Object Detection in Aerial Images

Compared to traditional, handcrafted feature-based object detection methods, recent Convolutional Neural Network (CNN)-based object detection algorithms have witnessed remarkable advancements in both accuracy and speed. CNN-driven detectors can be broadly classified into two categories: anchor-based and anchor-free detectors. The former can be further subdivided into one-stage and two-stage detectors, while the latter encompass keypoint-based and center-based detectors. Although these methods have currently achieved good results, they also have their respective limitations.

Anchor-Based Detectors: In the realm of two-stage detectors, Faster R-CNN [19] stands as a seminal work, comprising a Region Proposal Network (RPN) and a Region-Based Convolutional Neural Network (R-CNN) for object detection. Subsequently, numerous detectors have been proposed to enhance its performance, including FPN, Cascade R-CNN [20], and TridentNet [21]. Classical two-stage detectors are highly instructive; yet, they are often constrained by significant time costs, making it challenging to balance accuracy and speed. On the other hand, one-stage detectors, such as SAFF-SSD [22], RetinaNet [23], and the YOLO [24] family, directly predict class probabilities and bounding box offsets, rendering them simpler and more efficient than their two-stage counterparts. Benefiting from its efficient feature extraction and fusion capabilities, coupled with various enhancement strategies, the YOLO algorithm has been proven to be a combination of high efficiency and high accuracy. However, it still struggles with detecting tiny targets in aerial imagery.

Anchor-Free Detectors: Within the keypoint-based paradigm, detectors first locate several predefined or self-learned keypoints and subsequently generate bounding boxes to detect objects. Examples include CornerNet [25], Grid R-CNN [26], CenterNet [27], and RepPoints [28]. Meanwhile, center-based detectors treat the center of an object as a foreground to define positives and then predict distances from these positives to the four sides of the object’s bounding box for detection. DenseBox, FCOS [29], and FoveaBox [30] are notable representatives of this approach. Due to the absence of predefined anchor box sizes and aspect ratios, the model tends to be more concise and adaptable. However, this is accompanied by slower convergence during the training process and a tendency to fall into local optima.

Inspired by the tremendous success of CNN-based object detectors in natural scenes, extensive research has recently been conducted on object detection in aerial imagery. Diverging from detectors tailored for natural scenes, the majority of studies on aerial object detection have leveraged anchor-based approaches, exemplified by R-P-Faster R-CNN [31], YOLT [32], RoI Transformer [33], R²-CNN [18], and Mask OBB. These advancements demonstrate the adaptability and effectiveness of CNN-based detection techniques in the challenging domain of aerial imagery analysis. However, limited by issues such as the lack of features in tiny targets, the success rate of detection still needs to be improved.

2.3. Tiny Object Detection in Aerial Images

The detection of tiny objects poses a formidable challenge in the field of computer vision, prompting researchers to devise innovative methodologies to tackle this intricate task. Currently, the prevalent approaches to tiny object detection can be broadly categorized into three streams: multi-scale feature learning, crafting superior training strategies, and GAN-based detection frameworks.

Multi-Scale Feature Learning: A straightforward yet classical approach involves resizing the input images to various scales and training distinct detectors, each optimized for peak performance within a specific scale range. To mitigate computational expenses, several endeavors have aimed to construct feature-level pyramids across different scales. For instance, SSD [22] leverages feature maps of varying resolutions to detect objects, while the Feature Pyramid Network (FPN) introduces a top-down architecture with lateral connections, integrating multi-scale feature information to enhance detection performance. However, its sensitivity to background noise and limitations in feature acquisition result in only moderate performance when dealing with small targets. Subsequent advancements have further refined FPN’s capabilities, including PANet, BiFPN, and Recursive FPN [34]. PSPNet presents a Pyramid Scene Parsing Network that harnesses contextual information to address the intricacies of tiny object detection. However, it struggles with densely distributed tiny objects and lacks real time performance. Additionally, TridentNet [21] constructs a parallel multi-branch architecture with varying receptive fields, generating feature maps tailored to specific scales. R²CNN [18] introduces TinyNet, a custom-designed backbone, specifically for detecting tiny objects in large-scale aerial imagery. Similarly, their performance on small objects is far inferior to that on medium and large objects, which demonstrates that as the number of feature extraction layers increases, the features of small objects gradually diminish during the downsampling process.

Better Training Strategies: Inspired by the challenge of simultaneously detecting both minute and large-scale objects with consistent performance, Singh et al. introduced SNIP, and later, Singh et al. further developed SNIPER [35], which selectively trains object detectors within specific scale ranges. This approach addresses the issue of scale variance in object detection. Additionally, Kim et al. proposed the Scale-Aware Network (SAN) [36], which maps features extracted from diverse spatial contexts onto a scale-invariant subspace, thereby enhancing the robustness of detectors against scale variations. To improve the detection accuracy of tiny objects in aerial imagery, Sig NMS presents a novel Non-Maximum Suppression (NMS) method. Undoubtedly, all the aforementioned methods strive to achieve multi-scale perception and enhance the recognition capability for small objects. However, limited by model generalization and computational complexity, they cannot be applied in scenarios requiring rapid responses. Meanwhile, Yu et al. devised a scale matching technique tailored for micro-object detection. This method aligns the object scales between two datasets, facilitating the detection of small objects by leveraging the scale consistency across datasets. In summary, these advancements demonstrate innovative strategies to tackle the intricate problem of scale invariance in object detection, particularly focusing on enhancing the performance for both extreme ends of the scale spectrum: minute and large objects.

GAN-Based Detectors: The Perceptual GAN, as the pioneering detector leveraging GAN technology for small object detection, aims to enhance the detection of diminutive objects by mitigating the representational disparity between small and large objects. Furthermore, Bai et al. introduced MT-GAN [37], a framework tailored for training image-level super-resolution models, specifically designed to bolster the feature representation of small regions of interest (ROIs). Additionally, the work presented explores a feature-level super-resolution approach, which is employed to elevate the small object detection performance of proposal-based detectors. However, due to the uncontrollable generation process of GANs, their strong dependence on training samples, and the issue of unstable training, they are difficult to apply in small object detection for airborne imagery.

2.4. Evaluation Metrics and Label Assignment Strategies in TOD

Numerous researchers contend that the difficulty in detecting tiny objects stems from the scarcity of object feature information and the loss of small object details due to excessively deep network layers. However, through extensive experimentation, we demonstrate that the primary challenge in tiny object detection lies in the absence of suitable evaluation and measurement criteria. In other words, many small objects, though detected, are erroneously eliminated during the Non-Maximum Suppression (NMS) and subsequent processing stages due to conventional object measurement standards, leading to a severe issue of missed detections for small objects [38]. Consequently, this paper proposes a novel measurement approach tailored specifically for small objects, which can serve as a replacement for the Intersection over Union (IoU) metric commonly used in tiny object detectors. Based on this new metric, we devised an efficient and rapid single-stage detector that is well suited for the new measurement paradigm.

IoU is the most widely utilized metric for measuring the similarity between bounding boxes. Nevertheless, IoU functions effectively only when there is an overlap between the bounding boxes. To address this limitation, Generalized IoU (GIoU) was introduced by incorporating a penalty term that transforms the bounding boxes toward their minimal enclosing box. However, GIoU degrades to IoU when one bounding box entirely encloses the other. Consequently, DIoU and CIoU were proposed to overcome the limitations of IoU and GIoU by considering three geometric properties: overlap area, central point distance, and aspect ratio. GIoU, CIoU, and DIoU are primarily applied in NMS and loss functions as replacements for IoU to enhance general object detection performance; yet, their utilization in label assignment remains under-explored. In a recent work, Yang et al. introduced the Gaussian–Wasserstein Distance (GWD) [39] loss for oriented object detection by measuring the positional relationship between oriented bounding boxes. However, GWD was motivated to address boundary discontinuities and square-like issues in object-oriented detection. Wang et al. introduced a Normalized Wasserstein Distance (NWD) [40], which provides a new perspective for the measurement of small objects by modeling the similarity between bounding boxes as the assessment of two distributions. This approach has achieved promising results. Our motivation stems from reducing IoU’s sensitivity to minor positional deviations in small objects, and our proposed method can substitute IoU in all components of anchor-based object detectors.

Allocating high-quality anchor points for ground truth (gt) boxes of tiny objects poses a formidable challenge. A straightforward approach is to reduce the Intersection over Union (IoU) threshold when selecting positive samples. While this can enable more tiny objects to match with anchors, it deteriorates the overall quality of training samples. Furthermore, numerous recent efforts have aimed to render the label assignment process more adaptive, thereby enhancing detection performance. For instance, Zhang et al. proposed an Adaptive Training Sample Selection (ATSS) [9] method, which automatically computes the positive/negative (pos/neg) thresholds for each gt based on the statistical IoU values from a set of anchors. Kang et al. introduced Probabilistic Anchor Assignment (PAA) by assuming that the joint loss distribution of pos/neg samples follows a Gaussian distribution. Additionally, Optimal Transport Assignment (OTA) [41] formulates the label assignment process from a global perspective as an optimal transport problem. However, these methods all utilize IoU metrics to quantify the similarity between two bounding boxes and primarily focus on threshold settings within label assignment, which are suboptimal for TOD.

3. Method

IoU, inherently a Jaccard similarity coefficient utilized for computing the similarity between two finite sample sets, has inspired Wang et al. [40] to propose NWD, achieving remarkable performance in metric evaluations. Building upon this foundation, we reassess the similarities and differences in assessing ordinary-sized and tiny targets. Drawing on the concept of minimum point distance and Wasserstein distance, we devise an optimized metric, MMPW, tailored for microscopic objects.

Furthermore, we introduce a novel end-to-end feature extraction and lightweight detection network, enabling high-precision and real-time target detection in airborne imagery. This network architecture, akin to a conventional one-stage detector, comprises the GC-DN Network (integrating GC-Backbone and DN-Neck) and a Detection Head leveraging MMPW.

The GC-Backbone incorporates a global self-attention mechanism, reducing computational complexity and enhancing detection speed. The DN-Neck, featuring dense connections across various layers, fuses high-level semantic and low-level spatial features, enriching the feature representation. Notably, the MMPW-based Detection Head provides a superior metric strategy for tiny objects, leading to significant accuracy improvements. An illustrative diagram of the overall structure is presented in Figure 2. Specifically, the input image first undergoes two Conv operations to rapidly reduce its scale. In the GC-Backbone section, it passes through four identical feature extraction modules equipped with a global contextual attention mechanism, which, respectively, output four feature maps at different scales. In the DN-Neck, we adopt a bottom-up and then top-down approach. Specifically, deep-level features are fused with shallow-level features progressively from bottom to top, and after reaching the top pathway, they are progressively downsampled to supplement information to the deep-level features. The fusion process is implemented through our designed CSF layer, ultimately outputting four feature maps at different scales in the neck section. In the MMPW Detection Head part, the four sets of feature maps, respectively, correspond to solving the prediction problems for targets of different scales. Within each set of detection heads, the target’s category, location, and confidence are predicted. During this process, the MMPW Distance is not only used for bounding box measurement but is also applied in the Non-Maximum Suppression (NMS) strategy. The final output is the required detection result.

3.1. GC-DN Network

3.1.1. GC-Backbone

This paper introduces a lightweight backbone network, GC-Backbone, equipped with a global self-attention mechanism. It consists of four identical global context extraction modules, termed GC modules, connected in series. Employing the non-local global self-attention approach for modeling, this architecture enables the model to capture global context relationships, thereby strengthening its feature extraction capabilities. Additionally, by incorporating a simplified non-local (SNL) block and a computationally lightweight squeeze-and-excitation (SE) block, the modules significantly reduce the computational cost of the model. Each individual GC module encompasses two parallel branches, where the input feature maps undergo separate processing through Global Context Blocks (GCBs) in each branch before being concatenated. A structural diagram illustrating this configuration is presented in Figure 3.

The design of the Global Context Block (GCB) is inspired by GCNet, leveraging the principle of global self-attention modeling to effectively capture global contextual relationships or features. It incorporates the advantages of both the simplified non-local (SNL) block and the lightweight squeeze-and-excitation (SE) block. Since non-local blocks inherently learn global contexts independent of queries, our GCB’s global attention pooling model aligns with that of NL blocks in capturing global contexts, yet with notably lower computational costs. By employing a bottleneck transformation to reduce redundancies in the global context features, the GC block further minimizes the number of parameters and FLOPs. Consequently, the FLOPs and parameter count of the GC block are significantly lower than those of NL blocks, enabling its application across multiple layers with only marginal computational overhead while better capturing long-range dependencies and facilitating network training. The primary distinction between SE blocks and GC blocks lies in their fusion modules, reflecting their different objectives. SE blocks recalibrate channel importances through rescaling but inadequately model long-range dependencies. In contrast, our GC block follows the NL block approach, utilizing addition to aggregate global contexts into all positions, effectively capturing long-range dependencies. Another distinction is the layer normalization within the bottleneck transformation. Given that our GCB employs addition for fusion, layer normalization simplifies the optimization of the two-layer architecture in the bottleneck transformation, enhancing performance. Furthermore, the global average pooling in SE blocks serves as a special case of the global attention pooling employed in GCBs.

The detailed architectural diagram of the Global Context Block (GCB) is illustrated in Figure 4, which specifically encapsulates the following structural representations. In Equation (1), ${\displaystyle \frac{e^{W_k{x}_j}}{{\displaystyle {\sum }_m{e}^{W_k{x}_m}}}}$ denotes the weight of the global attention pooling, while $W_{v2}\mathrm{ReLU}(\mathrm{LN}(W_{v1}(•)))$ represents the bottleneck transformation.

(1)$z_i=x_i+W_{v2}\mathrm{ReLU}(\mathrm{LN}(W_{v1}{\displaystyle \sum _{j=1}^{N_p}\frac{e^{W_k{x}_j}}{{\displaystyle \sum _{m=1}^{N_p}{e}^{W_k{x}_m}}}}{x}_j))$

Specifically, our GC block comprises (a) global attention pooling for contextual modeling, (b) a bottleneck transformation to capture channel dependencies, and (c) broadcast element-wise addition for feature fusion. Given its lightweight nature, the GC block can be seamlessly integrated into multiple layers, enhancing the capture of long-range dependencies with only a marginal increase in computational cost. In this paper, the GC block is strategically deployed within four modules of the lightweight backbone network, GC-Backbone, and applied to both branches within each module. This results in a modest increase in the total model’s computational complexity, from 33.3 GFLOPs to 35.1 GFLOPs, representing a 5.4% enhancement.

3.1.2. DN-Neck

As depicted in the figure, we innovatively stack multiple cross-scale fusion structures to construct a Dense Nested Neck (DN-Neck), leveraging a dense architecture that intertwines at various depths. Given the substantial disparity in optimal receptive fields across targets of different sizes, these fusion subnetworks embedded at varying depths inherently cater to a diverse range of target dimensions. Inspired by this insight, we strategically place multiple nodes along the backpropagation path of the Feature Pyramid Network (FPN), fostering a tightly knit, nested network topology. Each node within this intricate framework serves as a conduit, receiving features not only from its immediate layer but also from adjacent levels, enabling iterative and pervasive fusion of multi-scale features. Consequently, the representational richness of small targets is preserved deeper within the network, ultimately yielding enhanced outcomes and optimized performance.

In this paper, we integrate the current layer with its immediate upper and lower layers to form our feature fusion architecture. Without loss of generality, we take the layer indexed by i = 2 as an illustrative example to elaborate in Figure 5.

Each feature fusion module exhibits a similar structure, consisting of the current layer g(2) and the processed layers d12 and u32 derived from its immediate upper g(1) and lower g(3) layers, respectively (in cases where there is no upper or lower layer, only two inputs are present). The upper layer d12, processed through depthwise separable convolution, and the lower layer u32, subjected to upsampling, are scaled to match the dimensions of the current layer g(2). Subsequently, these three layers are concatenated and passed through a Cross-Scale Fusion (CSF) layer for further integration. This layer, drawing inspiration from DenseNet, incorporates several innovative design choices that significantly enhance its performance. Specifically, it introduces an increased number of skip connections, eliminating the need for convolutional operations within branches. Furthermore, it incorporates additional split operations, which not only enrich the feature representation but also reduce computational costs, thereby achieving a delicate balance between feature richness and efficiency. The output of the CSF module can be mathematically represented by Formula (2).

(2)$output=\mathrm{CSF}\left(\mathrm{concat}\left(g\left(2\right), \mathrm{DWConv}\left(g\left(1\right)\right), \mathrm{upsample}\left(g\left(3\right)\right)\right)\right)$

The design of this structure aims to further streamline the network while maintaining or even enhancing its feature extraction capabilities. It comprises a combination of convolutional modules and multiple Bottleneck units, where each Bottleneck incorporates two consecutive 3 × 3 convolutional blocks. By leveraging an increased number of skip connections and minimizing convolutional operations, this architecture seamlessly fuses feature maps from distinct stages of the Backbone, thereby enriching and amplifying feature representations. This multi-scale feature fusion technique is pivotal for augmenting the model’s ability to detect targets of varying sizes, enhancing its overall performance and versatility in object detection tasks. The output of the CSF layer is subsequently fed as input into the subsequent set of structures, where it undergoes either depthwise separable convolution or upsampling for progression into the next group of structures.

3.2. MMPW Metric

Given that most exceedingly minute real-world objects are not strictly rectangular in nature, the anchor-based bounding box representation often encompasses background pixels. Within these bounding boxes, foreground pixels and background pixels tend to cluster around the center and the edges, respectively. In other words, the pixels closer to the center of the rectangular box are more crucial, with their significance diminishing toward the periphery, thus becoming less relevant to the object’s attributes. To better characterize the varying weights of different pixels within a bounding box, we model the box as a two-dimensional Gaussian distribution, where the central pixels exhibit the highest weight, and the importance of pixels gradually decreases from the center toward the boundaries. In this section, we propose the MMPW metric, which aims to define a new similarity measure for tiny object bounding boxes. This metric draws inspiration from the Minimum Points Distance (MPD) and the Wasserstein Distance in the field of statistics. We leverage the advantage of the Wasserstein Distance being insensitive to object scale while retaining the simplicity and efficiency of the MPD in calculation. The two related methods are introduced in detail in Section 3.2.1 and Section 3.2.2, respectively.

Specifically, for a horizontal bounding box R = (cx, cy, w, h), where (cx, cy), w, and h represent the center coordinates, width, and height, respectively, the equation of its inscribed ellipse can be formulated by Formula (3).

(3)$\frac{{(x-cx)}^2}{{\left(w/2\right)}^2}+\frac{{(y-cy)}^2}{{\left(h/2\right)}^2}=\mathbf{1}$

To more precisely characterize the similarity between two bounding boxes, we model them as two Gaussian distributions and employ their distribution distance as the criterion for measurement. A general two-dimensional Gaussian distribution is typically defined by Equation (4).

(4)$f(x|\mu ,\mathrm{\Sigma })={\displaystyle \frac{\exp (-\frac{1}{2}{(x-\mu )}^{\mathrm{T}}{\mathrm{\Sigma }}^{-1}(x-\mu ))}{2\pi {\left|\mathrm{\Sigma }\right|}^{\frac{1}{2}}}}$

In this context, x, μ, and Σ represent the coordinates (x, y) of the Gaussian distribution, the mean vector, and the covariance matrix of the Gaussian distribution, respectively. When the equation ${(x-\mu )}^{\mathrm{T}}{\mathrm{\Sigma }}^{-1}(x-\mu )=1$ holds, the ellipse in (1) can be transformed into the density contour of a two-dimensional Gaussian distribution. Furthermore, consequently, the horizontal bounding box R = (cx, cy, w, h) can be modeled as a two-dimensional Gaussian distribution N(μ, Σ). Accordingly, it is represented by Equation (5).

(5)$\mu =\left[\begin{array}{c}cx\\ cy\end{array}\right], \mathrm{\Sigma }=\left[\begin{array}{cc}{(w/2)}^2& 0\\ 0& {(h/2)}^2\end{array}\right]$

3.2.1. Minimum Points Distance

Generally, an undirected bounding box is uniquely determined in an image by the point coordinates of its top-left and bottom-right corners. Inspired by the geometric properties of bounding boxes, the concept of Minimum Points Distance (MPD) was introduced to directly minimize the distance between the predicted bounding box and the ground truth bounding box at their top-left and bottom-right corners. As shown in Figure 6a, when computing the loss, in addition to calculating the Intersection over Union (IoU) between the predicted box and the ground truth box, it is necessary to incorporate a normalized MPD. The IoU-like expression and loss function constructed from the MPD can be formulated into Equation (6):

(6)$\mathrm{MPD}={\displaystyle \frac{d_1^2}{h^2+w^2}}+{\displaystyle \frac{d_2^2}{h^2+w^2}}, \mathrm{MPDIoU}=\mathrm{IoU}-{\displaystyle \frac{d_1^2}{h^2+w^2}}-{\displaystyle \frac{d_2^2}{h^2+w^2}}$

For medium-sized objects, IoU measured by the MPD exhibits advantages such as comprehensive evaluation and rapid convergence, which has been proven effective among numerous enhancement approaches. Nevertheless, when it comes to tiny-scale objects, as elucidated in Section 2, solely relying on IoU and the MPD to characterize rectangular bounding boxes proves to be limited in scope.

3.2.2. Wasserstein Distance

Inspired by the Optimal Transport Theory, we adopt the Wasserstein Distance from this domain to quantify the similarity between two rectangular boxes. Specifically, for two two-dimensional Gaussian distributions, denoted as ${\mu }_1={\mathrm{N}(\mathrm{m}}_1, {\mathrm{\Sigma }}_1)$ and ${\mu }_2={\mathrm{N}(\mathrm{m}}_2, {\mathrm{\Sigma }}_2)$, the second-order Wasserstein distance between μ₁ and μ₂ is formally defined by Formula (7):

(7)${\mathrm{W}}_2^2({\mu }_1,{\mu }_2)={‖{\mathrm{m}}_1{-\mathrm{m}}_2‖}_2^2+\mathbf{Tr}\left({\mathrm{\Sigma }}_1+{\mathrm{\Sigma }}_2-2{\left({\mathrm{\Sigma }}_2^{1/2}{\mathrm{\Sigma }}_1{\mathrm{\Sigma }}_2^{1/2}\right)}^{1/2}\right)$

After applying the Gaussian mapping detailed in Formula (5), for two generalized rectangular bounding boxes ${\mathrm{N}}_a$ and ${\mathrm{N}}_b$ in Figure 6b, with their respective boundaries defined as ${\mathrm{N}}_a$ = (cx_a, cy_a, w_a, h_a) and ${\mathrm{N}}_b$ = (cx_b, cy_b, w_b, h_b), the Wasserstein Distance between them can be succinctly characterized as outlined in Formula (8).

(8)${\mathrm{W}}_2^2({\mathrm{N}}_a{,\mathrm{N}}_b)={‖\left({\left[c{x}_a,c{y}_a,{\displaystyle \frac{w_a}{2}},{\displaystyle \frac{h_a}{2}}\right]}^{\mathrm{T}}, {\left[c{x}_b,c{y}_b,{\displaystyle \frac{w_b}{2}},{\displaystyle \frac{h_b}{2}}\right]}^{\mathrm{T}}\right)‖}_2^2$

Notably, the Wasserstein Distance is invariant to the scale and position of the bounding boxes, making it suitable for measuring targets of arbitrary dimensions. Nevertheless, despite its merit as a distance metric, Wasserstein Distance cannot be directly utilized as a similarity measure for rectangular bounding boxes.

3.2.3. IoU of Mixed Minimum Point-Wasserstein Distance

To capitalize on the advantageous attributes of both the Minimum Point Distance and the Wasserstein Distance, while addressing the discrepancies in measurement at conventional and minute scales, we introduce a novel concept: the Mixed Minimum Point-Wasserstein (MMPW) Distance. This innovative metric is designed to be integrated into bounding box evaluation methodologies, thereby enhancing its application in label assignment, Non-Maximum Suppression (NMS), and loss function formulations. The MMPW Distance offers a comprehensive solution that transcends the limitations imposed by bounding box size, making it particularly adept at capturing the nuanced distribution characteristics of tiny-scale objects within their respective bounding boxes. This results in faster convergence rates and significantly improved performance metrics compared to existing methods.

Specifically, our MMPW Distance can be divided into two components. One component is dedicated to adaptive optimization for small targets, while the other component accommodates medium-to-large-scale targets. The composition ratio of these two components is generally dynamically controllable and can be adjusted according to the distribution of targets in the dataset for optimal adaptation. As shown in Figure 6c, we first select the center point coordinates $(c{x}_a,c{y}_a)$, width $w_a$, and height $h_a$ of the ground truth bounding box ${\mathrm{N}}_a$, as well as the center point coordinates $(c{x}_b,c{y}_b)$, width $w_b$, and height $h_b$ of the predicted bounding box ${\mathrm{N}}_b$, with the distances between the top-left corners being d₁ and the bottom-right corners being d₂.

Then, we calculate their Wasserstein Distance according to Equation (8). Given that the Wasserstein Distance is not normalized, we leverage the properties of exponentiation to compute its normalized form for our calculations. Specifically, we divide the obtained Wasserstein Distance by the average scale C of the targets and then take the square root of the result. Furthermore, we utilize the property of the negative exponential of the natural logarithm e within the positive number range to process the distance, enabling it to vary within a local range. Regarding the other component, we follow the MPD approach and define it through the original Intersection over Union (IoU), the width $w_a$, and the height $h_a$ of the ground truth bounding box ${\mathrm{N}}_a$, as well as the two distances, d₁ and d₂. Finally, we combine these two components to serve collectively as our Mixed Minimum Point-Wasserstein (MMPW) Distance. The intersection over union (IoU) defined by a MMPW Distance can be expressed in Equation (9).

(9)$MMPWIoU=\lambda \cdot \exp (-\sqrt{{\displaystyle \frac{{\mathrm{W}}_2^2({\mathrm{N}}_a,{\mathrm{N}}_b)}{C}}})+\mu \cdot (\mathrm{IoU}-{\displaystyle \frac{d_1^2+d_2^2}{{h_a}^2+{w_a}^2}})$

Here, λ and μ represent the weights attributed to the Minimum Points Distance and the Wasserstein Distance, respectively. Given that the range of values for the loss function lies within [0, 1], $\lambda +\mu =1$. In addressing the challenge of detecting tiny objects under 16 pixels in size, we calibrate the parameters λ and μ, which are crucial for balancing the model’s sensitivity and specificity. Specifically, we set λ = 0.55 and μ = 0.45 to optimize the detection performance. The constant C, which appears in the formulation of the Wasserstein distance, is adjusted adaptively based on the average scale of targets present in the dataset. This ensures that the distances are scaled appropriately, maintaining consistency across different scales within the data. The constant C, which is used to adjust the threshold for object detection, was empirically determined to be 12.8 for the AI-TOD dataset, reflecting the dataset’s characteristics and requirements. For the Visdrone-19 dataset, with its distinct set of conditions, C was found to be more effective at 35.8.

In our model, the objective is to minimize the discrepancy between predicted and actual outcomes. Accordingly, the loss function, which quantifies this discrepancy, can be formulated as shown in Equation (10). This equation serves as a critical component in the optimization process, guiding the adjustment of model parameters to improve predictive accuracy.

(10)$los{s}_{MMPW}=1-MMPWIoU$

3.3. MMPW Detection Head

The object detection head serves as the component for performing object detection on the feature pyramid, incorporating convolutional layers, pooling layers, and fully connected layers, among others. In the MMPW-Net model, the detection head module primarily focuses on conducting multi-scale object detection on the feature maps extracted by the DN-Neck network. It employs a multi-level feature fusion approach, which initially reduces the dimensionality of the feature maps and scales them through a Conv module derived from the backbone network outputs. Subsequently, feature maps from different levels are fused to obtain richer feature information, thereby enhancing detection performance. For feature maps across four scales, the detection head predefines anchor boxes of various sizes and aspect ratios, obtained by applying K-means clustering to the ground truth boxes in the training set. These anchors are computed and stored in the model prior to training and utilized for generating detection boxes during prediction. Following this, a series of convolutional layers with variable channel numbers and kernel sizes process the feature maps to accommodate different detection tasks. During the prediction phase, the prediction layer is responsible for estimating bounding boxes, confidence scores, and categories, as illustrated in Figure 2. Specifically, the bounding box localization (bbox) loss employs the MMPW loss function proposed in our previous section’s Equation (10) to compute the positive sample localization loss. The classification (cls) loss adopts BCELoss, which is only computed for positive samples, while the confidence loss utilizes BCELoss to calculate the objectness (obj) loss for all samples. Notably, as the feature layer scale decreases, the weight of the obj loss increases progressively. Finally, in the Non-Maximum Suppression (NMS) process, the MMPW Distance proposed in Section 3.2.3 is also utilized to suppress redundant bounding boxes and retain valid ones.

4. Results

We evaluated our proposed method on the AI-TOD and VisDrone-19 datasets. Notably, the ablation study was conducted on AI-TOD, a challenging dataset specifically designed for tiny object detection.

4.1. Datasets

The AI-TOD [3] dataset is constructed based on a combination of the DOTA-v1.5 training set [14], the xView training set [13], the VisDrone2018-Det training set, the Airbus Ship training set, and the DIOR training and testing sets. The images captured originate from a multitude of diverse geographical contexts, thereby enhancing the dataset’s diversity and generalization capabilities. It encompasses eight categories, all of which are utilized in the experiments conducted in this paper. As shown in Table 1, the dataset comprises 28,036 aerial images, each with a resolution of 800 × 800 pixels, containing a total of 700,621 object instances. The average absolute size (actual pixel values) of these objects is merely 12.8 ± 5.9 pixels, with an average relative size (proportion to the image size) of 0.016 ± 0.007, significantly smaller than those in other object detection datasets. To ensure a rigorous evaluation process, the dataset is partitioned into training, validation, and test sets, utilizing 2/5, 1/10, and 1/2 of the images, respectively.

Furthermore, VisDrone-19 [15] is a drone-based object detection dataset collected and curated by the AISKYEYE team at Tianjin University. As illustrated in Table 1, the dataset comprises 10,209 images spanning across 10 categories, all of which were utilized in the experiments conducted in this paper. Due to the fact that the images were captured in 14 different cities and rural areas across China, encompassing various altitudes and varying degrees of target sparsity, VisDrone-19 presents numerous complex scenes and a plethora of small targets. The average absolute size of the targets is only 35.8 ± 32.8 pixels, while the average relative size is merely 0.030 ± 0.026. For dataset partitioning, 6471, 548, and 3190 images are designated for the training, validation, and test sets, respectively.

For evaluation, we adopted the same metrics as those used in the AI-TOD dataset, including Average Precision (AP), AP at IoU threshold of 0.5 (AP50), and F1-score, among others. These metrics provide a comprehensive assessment of our method’s performance in detecting tiny objects within these challenging datasets.

4.2. Implementation Details

The experimental environment can be summarized as follows: the system was equipped with 32 GB of RAM, running on Ubuntu 18.04 operating system, powered by an Intel Xeon CPU, and equipped with an NVIDIA Geforce RTX 3090 GPU featuring 24 GB of dedicated memory. To ensure a fair comparison across experiments, PyTorch was consistently utilized for all implementations. For single-stage methods, the training was uniformly conducted for 150 epochs, while for two-stage approaches, the training was standardized at 12 epochs. Additionally, the input image dimensions were set to 800 pixels for the AI-TOD dataset and 1000 pixels for the VisDrone-19 dataset, respectively.

4.3. Quantitative Results

To validate the efficacy of our proposed method in detecting distant small-scale targets within airborne imagery, we conducted experiments on two prominent micro-object detection datasets: AI-TOD and VisDrone-19. Furthermore, our approach was rigorously compared against mainstream algorithms designed for the detection of tiny objects, thereby offering a comprehensive evaluation of its performance.

To substantiate the superiority of MMPW-Net in achieving state-of-the-art performance with a relatively faster speed on the AI-TOD and VisDrone-19 datasets, we curated a comprehensive selection of both anchor-free and anchor-based mainstream object detectors alongside their variants tailored for detecting tiny objects. Within the anchor-free paradigm, we included methods that are centered around points (FoveaBox, FCOS) and those leveraging keypoints (Grid R-CNN, CenterNet, M-CenterNet). For anchor-based detectors, our analysis encompasses two-stage detectors (Faster R-CNN, Cascade R-CNN, DetectoRS) and their small-object-focused variants, as well as one-stage detectors (SSD-512, RetinaNet, ATSS, YOLOv5s, YOLOv8s) and their respective small object detection modifications. In addition, we also include some of the latest methods (NWD-RKA, SimD) for tiny targets in the comparison. We categorize the algorithms based on their respective classes and summarize the specific backbone networks utilized, the Intersection over Union (IoU) strategies adopted, and the detection accuracies achieved by each method in the accompanying table.

In AI-TOD data experiments, we also select the Average Precision of three types of objects that are most abundant in the dataset, exhibit the smallest scales, and hold the greatest research significance: ships, planes, and cars, as auxiliary metrics for assessing precision. Among the three categories we selected, our method achieved the highest accuracy for planes and cars, with ships slightly trailing behind YOLOv8.

As evident in Table 2, both the YOLO series in single-stage detectors and the DetectoRS in two-stage detectors achieved remarkable performance improvements in detecting small objects. This can be attributed to the combination of their powerful feature extraction capabilities and efficient feature fusion techniques. However, recent results using distance metric strategies such as NWD and SimD demonstrated that more reasonable distance metrics often play a decisive role in detecting extremely small objects. Inspired by these advanced methods, our proposed approach improves upon both aspects, achieving the highest accuracy in both AP and AP50 metrics.

In the VisDrone-19 data experiments, we employed a scale-specific evaluation strategy to better validate the performance of our model. Unlike the AI-TOD dataset, the scale variation of targets in this dataset is significant. Therefore, we evaluate targets of different scales in our dataset according to the commonly used definitions of small (2–32 pixels), medium (32–96 pixels), and large (exceeding 96 pixels) in the COCO dataset. This is to demonstrate that our method not only improves the detection of small-scale targets but also outperforms or is comparable to other methods in detecting medium and large-scale targets. This approach allows us to assess the model’s capabilities more comprehensively by considering different object scales within the dataset.

As evident in Table 3, our method exhibits optimal overall accuracy. This is attributed to the metric strategy defined by MMPW, which is unaffected by object scale and takes into account the characteristics of objects of different sizes. Furthermore, our method achieves the highest accuracy in both the small and medium scales. In the large scale, it ranks second only to YOLOv5, benefiting from its effective data augmentation and sensitivity to large objects.

4.4. Visualization Results

We selected typical scenarios of cities, roads, and ports from AI-TOD datasets and conducted detection on small objects, such as vehicles and ships. The visualization results are shown in Figure 7. To better illustrate the scale of the objects in the image, we added a grid to the picture. The original image has a width and height of 800 pixels each, and we divided it equally into 25 sections, with each grid cell measuring 32 pixels, which also corresponds to the small object scale defined in the COCO dataset. As evident in Figure 7, the majority of the objects are significantly smaller than the side length of a grid cell, demonstrating the minuteness of the object scales and the difficulty in detection. This observation further supports the superiority of our method in detecting tiny objects.

Similarly, we also selected typical scenarios from the VisDrone-19 dataset, and the visualization results of the detection are presented in Figure 8. Among them, (a) and (b) represent the detection result images and the local detection result images, respectively. As can be seen in Figure 7 and Figure 8, many tiny objects are closely arranged. The boxes of different colors in the figure represent the visual detection results of objects of different categories. Due to the lack of optimization and attention given to them by conventional IoU strategies, many valid detection results are identified as false positives due to overlap and are subsequently filtered out during the Non-Maximum Suppression (NMS) process. The MMPW Distance proposed in this study aims to better characterize the effective distance between small objects, allowing for the retention of these captured valid individuals and reducing the miss detection rate. Additionally, it is also evident that many objects are small in scale and have insignificant features or are even difficult to capture visually due to their distance from the imaging sensor. Therefore, we choose to enlarge the local detection results. The envisioned GC-DN Network is designed to provide shallow features effectively to deeper layers of information, thereby reducing the omissions in object detection.

4.5. Ablation Study

In this ablation study, we systematically evaluated the impact of various components on the performance of micro-object detection. Specifically, using the AI-TOD dataset, we sequentially removed or modified specific parts of the model by, in particular, removing the GC-DN Network and the MMPW metric to observe and record the changes in detection accuracy. As shown in Table 4, ✓ indicates that the component is used, while “base” represents the baseline algorithm without any additional innovations.

The experimental results presented in Table 4 indicate that removing the GC-DN Network significantly degrades the detection capability for tiny objects, demonstrating the crucial role of shallow features in capturing fine-grained details of these small targets. Furthermore, we also found that the choice of metric exerts a notable influence on the performance of tiny object detection. Compared to the traditional Intersection over Union (IoU) and its corresponding loss functions, adopting the MMPW Distance can significantly improve the selection strategy for small targets, facilitating faster convergence to a superior solution. At the same time, it mitigates the overfitting problem to a certain extent, further enhancing detection accuracy.

4.6. Model Efficiency Evaluation

To substantiate the efficacy of our model in achieving high detection performance with minimal parameter utilization and computational overhead, we meticulously selected a spectrum of single-stage detection models as benchmarks for comparative analysis. In this endeavor, we zeroed in on several key metrics, namely Average Precision (AP), AP at 50% Intersection over Union (AP50), Giga Floating-Point Operations Per Second (GFLOPs) as a measure of computational complexity, the total number of parameters, and Frames Per Second (FPS) for inference speed, to comprehensively evaluate and present our findings in Table 5.

By scrutinizing these metrics, we aim to underscore the prowess of our model in achieving exceptional detection accuracy, as evidenced by the AP and AP50 scores, while simultaneously demonstrating its parsimony in terms of resource consumption. The GFLOPS metric highlights the remarkable efficiency of our approach in minimizing the computational requirements, making it a viable option for resource-constrained environments. Moreover, the drastic reduction in the number of parameters, as compared to the contending models, underscores the elegance of our model design in achieving high performance with minimal complexity.

Lastly, the inference FPS, a crucial indicator of real-time performance, underscores the ability of our model to process frames swiftly, enabling seamless integration into time-sensitive applications. Table 5 presents a compelling narrative, demonstrating that our model stands tall amid the competition, offering a harmonious blend of high detection accuracy, low computational cost, minimal parameter footprint, and swift inference speed.

5. Discussion

In the realm of long-range airborne imagery captured by unmanned aerial vehicles (UAVs), the detection of tiny objects poses a formidable challenge, primarily due to their minute scales, vulnerability to feature loss, and susceptibility to being obscured by the intricate background or overwhelming noise. Consequently, tiny object detection suffers from low detection rates and high false alarm rates, significantly impeding their effectiveness in monitoring and early warning systems. To address these pressing issues and enhance the capability of detecting diminutive targets within UAV imagery, this study presents a novel, single-stage, lightweight detector tailored specifically for tiny object detection. This approach aims to not only conquer the complexities associated with detecting tiny objects but also significantly elevate the timeliness and efficiency of the detection process.

5.1. Discussion on Optimization of Methodology and Strategies

The methodology presented in this manuscript is tailored specifically for the detection of tiny targets, such as humans, vehicles, boats, and aircraft, within aerial imagery. To expedite the recognition process, we eschew anchor-free and two-stage architectures, instead embracing an anchor-based, single-stage detector framework. Furthermore, to curtail model training time and augment cross-scenario generalization capabilities, the pretrained model of choice is YOLOv5s [6]. Nonetheless, eschewing pretrained weights enables the model to concentrate more intently on the minuscule targets within the training samples, potentially enhancing its interpretability. In the realm of tiny target detection, the judicious application of data augmentation proves pivotal in elevating model performance and generalization prowess. Consequently, we incorporated a diverse suite of augmentation techniques, encompassing flips, scaling, translations, cutouts, color adjustments, and noise injection. Researchers may delve deeper into exploring novel augmentation methodologies, such as leveraging diffusion models for noising and denoising of samples or generative models to augment datasets with supplementary features, thereby fortifying the model’s ability to generalize.

5.2. Discussion on Data Selection and Processing

When employing aerial datasets, we strategically integrated a blend of tiny targets alongside general-sized objects, as shown in Figure 9, ensuring that the model retained its detection proficiency for commonplace targets without compromise. The boxes of different colors in the figure represent the visual detection results of objects of different categories. Notably, tiny target detection necessitates stringent data standards, as imprecise annotations may introduce spurious noise, potentially misleading the model’s learning trajectory. Conversely, an excess of unlabeled images may induce a conservative bias during prediction, escalating the risk of missed detections.

5.3. Discussion on the Strengths and Limitations of the Study

Experimental results demonstrate the feasibility of deploying the proposed model on edge devices mounted on unmanned aerial vehicles (UAVs). By capturing real-time image data from cameras as input, the model can autonomously detect micro-targets along with their confidence levels [6]. Upon detecting an unknown identity target, the system promptly alerts the relevant personnel, who then verify and decide on the necessary actions toward the target. While the algorithm presented in this study exhibits robust performance in detecting small targets within airborne imagery, it is not without limitations. Primarily optimized for visible light environments, the algorithm may struggle to detect targets in adverse lighting conditions, particularly during low-light scenarios at night. Additionally, the vibrations or instability inherent in UAV flights can impair the camera’s ability to capture clear images, subsequently degrading the detection performance [12].

5.4. Discussion on Future Research Directions

To address these limitations and further enhance the reliability of our system, we propose two potential avenues for future research. Firstly, integrating features extracted from infrared imagery could assist in improving the detection accuracy of small targets [42] under night-time and low-light conditions. Secondly, developing and integrating super-resolution reconstruction algorithms alongside deblurring techniques can elevate the imaging quality of the camera when capturing targets at extremely long distances [43], thereby enhancing detection precision under such circumstances. Through these endeavors, we aim to bolster the applicability and robustness of our algorithms across diverse environmental conditions.

6. Conclusions

In this paper, a novel single-stage tiny object detector, leveraging an innovative metric known as Mixed Minimum Point-Wasserstein Distance (MMPW), is introduced, featuring a lightweight backbone coupled with a heavy neck architecture. The proposed GC-DN Network framework, seamlessly integrating the Mixed Minimum Point-Wasserstein Distance, exhibits remarkable efficacy and efficiency in enhancing the detection of tiny objects, thereby enabling its widespread adoption across diverse scenarios requiring the identification of tiny targets. Extensive experiments conducted on the latest aerial visible light image datasets conclusively demonstrate the validity and robustness of the proposed approach, underscoring its potential to revolutionize the field of tiny object detection.

Author Contributions

Conceptualization, N.S. and Y.Y.; methodology, N.S. and Z.Z.; software, Z.Z., W.L. and Y.Q.; validation, Y.Y. and H.C.; formal analysis, H.C. and J.W.; investigation, Z.Z. and H.C.; data curation, J.W. and S.F.; writing—original draft preparation, Z.Z. and N.S.; writing—review and editing, S.F., Y.Y. and C.Z. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The Tiny Object Detection in Aerial Images (AI-TOD) dataset is obtained from https://github.com/jwwangchn/AI-TOD, accessed on 25 July 2024. The VisDrone-19 dataset is obtained from https://opendatalab.com/OpenDataLab/Visdrone_DET, accessed on 28 July 2024.

Acknowledgments

The authors are thankful to the providers for all the datasets used in this study. We also thank the anonymous reviewers and editors for their comments aimed at improving this paper.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Limitations of tiny object detection.

Figure 2. The overall structure diagram of MMPW-Net.

Figure 3. Structure diagram of GC module.

Figure 4. Structure diagram of global context block.

Figure 5. Illustration of local structure of DN-Neck.

Figure 6. Illustration of bounding box distance measurement.

Figure 7. Visualization of detection results from AI-TOD dataset.

Figure 8. Visualization of detection results from Visdrone-19 dataset.

Figure 9. Aerial imagery covering tiny-scale and general-scale targets.

Table 1

Overview of datasets.

Dataset	Categories	Number of Instances	Number of Images	Image Size(Pixels)	InstanceAbsolute Size (Pixels)	InstanceRelative Size (Pixels)
AI-TOD	8	700,621	28,036	800	12.8 ± 5.9	0.016 ± 0.007
VisDrone-19	10	343,205	10,209	1000	35.8 ± 32.8	0.030 ± 0.026

Table 2

Comparison with state of the art on AI-TOD dataset.

Method	Backbone	AP50	AP	APship	APplane	APcar
FoveaBox	Res50-FPN	19.8	8.1	17.7	13.8	11.4
FCOS	Res50-FPN	24.1	9.8	22.2	14.3	12.5
Grid R-CNN	Res50-FPN	27.7	12.2	22.0	22.6	12.9
CenterNet	DLA-34	39.2	13.4	21.9	17.4	16.5
M-CenterNet	DLA-34	40.7	14.5	22.3	18.6	18.6
Faster R-CNN *	Res50-FPN	43.8	17.8	26.40	27.55	16.71
Cascade R-CNN *	Res50-FPN	44.2	18.7	26.86	27.84	18.77
DetectoRS *	Res50-FPN	49.3	20.8	28.95	30.14	22.67
SSD-512	VGG-16	21.7	7.0	13.1	14.5	7.8
RetinaNet *	Res50-FPN	24.9	9.2	20.1	15.3	11.3
ATSS *	Res50-FPN	33.2	13.5	25.2	26.9	16.7
YOLOv5s	CSPDarknet53-FPN+PAN	45.4	18.2	29.2	22.8	26.7
YOLOv5s *	CSPDarknet53-GFPN	46.9	18.7	28.7	25.5	26.1
YOLOv8s	CSPDarknet53-FPN+PAN	46.8	20.5	31.3	21.9	29.5
YOLOv8s *	CSP-GFPN	47.8	20.8	32.6	26.1	29.6
DotD	Res50-FPN	29.5	12.9	23.7	26.8	11.3
NWD-RKA	Res50-FPN	51.8	21.9	31.9	29.7	29.3
SimD	Res50-FPN	53.5	22.2	31.9	33.5	29.6
MMPW-Net(ours)	GC-DNNetwork	54.8	22.5	32.3	33.5	29.9

All models are trained on the trainval set and tested on the test set. AP_ship, AP_plane, and AP_car, respectively, represent the average precision of ships, planes, and cars. The methods appended with an asterisk (*) signify their advanced versions tailored specifically for the detection of tiny objects. The red markings in the table represent the optimal performance values.

Table 3

Comparison with state of the art on Vidrone-19 dataset.

Method	Backbone	AP50	AP	APsmall	APmedium	APlarge
FoveaBox	Res50-FPN	29.0	17.2	7.4	28.3	36.5
FCOS	Res50-FPN	28.0	16.1	7.2	26.2	36.1
Grid R-CNN	Res50-FPN	32.1	20	10.1	33.4	41.3
CenterNet	DLA-34	29.1	15.1	7.4	24.6	31.9
M-CenterNet	DLA-34	30.7	16.5	8.3	26.2	41.1
Faster R-CNN *	Res50-FPN	28	16	7.4	27.3	34.3
Cascade R-CNN *	Res50-FPN	29.1	17.1	7.7	28.6	41.9
DetectoRS *	Res50-FPN	31.9	19.3	9.1	31.2	39.1
SSD-512	VGG-16	34.6	18.3	9.2	28.5	37.4
RetinaNet	Res50-FPN	24.1	14.4	4.6	25.2	33.5
ATSS	Res50-FPN	31.7	19.0	9.1	31.2	39.1
YOLOv5s	CSPDarknet53-FPN+PAN	46.3	27.4	17.2	37.1	46.7
YOLOv5s *	CSPDarknet53-GFPN	48.9	29.1	19.1	38.8	46.4
YOLOv8s	CSPDarknet53-FPN+PAN	46.8	27.5	19.4	37.4	43.8
YOLOv8s *	CSPDarknet53-GFPN	49.7	28.9	20.0	39.2	45.8
PP-YOLOE	CSPRepResNet	46.4	30.5	19.7	38.7	43.7
NWD-RKA	Res50-FPN	46.2	27.4	19.3	38.9	43.9
SimD	Res50-FPN	50.3	28.7	20.2	39.1	44.2
S+D	Res50-FPN	50.9	28.6	20.8	39.7	45.1
CN-DhVaSa	DLA-34	50.7	28.8	20.5	38.9	44.6
MMPW-Net(ours)	GC-DNNetwork	51.3	31	21.7	40.1	46.0

All models are trained on the trainval set and tested on the val set. AP_small, AP_medium, and AP_large, respectively, represent the average precision for small (2–32 pixels), medium (32–96 pixels), and large (exceeding 96 pixels) objects. The methods appended with an asterisk (*) signify their advanced versions tailored specifically for the detection of tiny objects. The red markings in the table represent the optimal performance values.

Table 4

Ablation study on AI-TOD dataset.

Method	GC-DN Network	MMPW	AP50	AP	F1-Score
base			45.4	18.2	51
		✓	49.6	19.8	54
	✓		52	20.7	55
ours	✓	✓	54.8	22.5	58

Table 5

Efficiency evaluation compared to other methods on VisDrone-19 dataset.

Method	AP	AP50	GFLOPs	Params(M)	FPS
YOLOv5s	27.4	46.3	15.8	7	68.0
YOLOv5s *	29.1	48.9	27.0	7.6	39.1
YOLOv8s	27.5	46.8	28.5	10.8	32.2
YOLOv8s *	28.9	49.7	37.8	11.1	25.9
PP-YOLOE	30.5	46.4	39.7	13.4	22.7
MMPW-Net	31	51.3	35.1	9.5	41.5

The methods appended with an asterisk (*) signify their advanced versions tailored specifically for the detection of tiny objects. The red markings in the table represent the optimal performance values.

References

1. Zhao, Y.; Sun, H.; Wang, S. Small Object Detection in Medium–Low-Resolution Remote Sensing Images Based on Degradation Reconstruction. Remote Sens.; 2024; 16, 2645. [DOI: https://dx.doi.org/10.3390/rs16142645]

2. Zhang, J.; Zhang, Y.; Shi, Z.; Zhang, Y.; Gao, R. Unmanned Aerial Vehicle Object Detection Based on Information-Preserving and Fine-Grained Feature Aggregation. Remote Sens.; 2024; 16, 2590. [DOI: https://dx.doi.org/10.3390/rs16142590]

3. Wang, J.; Yang, W.; Guo, H.; Zhang, R.; Xia, G. Tiny object detection in aerial images. Proceedings of the International Conference on Pattern Recognition; Milan, Italy, 10–15 January 2021; pp. 3791-3798.

4. Zhao, B.; Wang, C.; Fu, Q.; Han, Z. A Novel Pattern for Infrared Small Target Detection with Generative Adversarial Network. IEEE Trans. Geosci. Remote Sens.; 2021; 59, pp. 4481-4492. [DOI: https://dx.doi.org/10.1109/TGRS.2020.3012981]

5. Zhou, S.; Zhou, H. Detection Based on Semantics and a Detail Infusion Feature Pyramid Network and a Coordinate Adaptive Spatial Feature Fusion Mechanism Remote Sensing Small Object Detector. Remote Sens.; 2024; 16, 2416. [DOI: https://dx.doi.org/10.3390/rs16132416]

6. Liu, S.; Cao, L.; Li, Y. Lightweight Pedestrian Detection Network for UAV Remote Sensing Images Based on Strideless Pooling. Remote Sens.; 2024; 16, 2331. [DOI: https://dx.doi.org/10.3390/rs16132331]

7. Zhu, H.; Jing, D. Optimizing Slender Target Detection in Remote Sensing with Adaptive Boundary Perception. Remote Sens.; 2024; 16, 2643. [DOI: https://dx.doi.org/10.3390/rs16142643]

8. Pirinen, A.; Sminchisescu, C. Deep Reinforcement Learning of Region Proposal Networks for Object Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Salt Lake City, UT, USA, 18–23 June 2018; 6.

9. Zhang, S.; Li, S. Automated Targeted Training Set Selection for Improved Object Detection. IEEE Trans. Pattern Anal. Mach. Intell.; 2020; 42, pp. 2430-2442.

10. Cheng, G.; Han, J.; Zhou, P.; Guo, L. Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS J. Photogramm. Remote Sens.; 2014; 98, pp. 119-132. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2014.10.002]

11. Liu, Z.; Wang, H.; Weng, L.; Yang, Y. Ship rotated bounding box space for ship extraction from high-resolution optical satellite images with complex backgrounds. IEEE Geosci. Remote Sens. Lett.; 2016; 13, pp. 1074-1078. [DOI: https://dx.doi.org/10.1109/LGRS.2016.2565705]

12. Razakarivony, S.; Jurie, F. Vehicle detection in aerial imagery: A small target detection benchmark. J. Vis. Commun. Image Represent.; 2016; 34, pp. 187-203. [DOI: https://dx.doi.org/10.1016/j.jvcir.2015.11.002]

13. Lam, D.; Kuzma, R.; McGee, K.; Dooley, S.; Laielli, M.; Klaric, M.; Bulatov, Y.; McCord, B. xview: Objects in context in overhead imagery. arXiv; 2018; arXiv: 1802.07856

14. Xia, G.-S.; Bai, X.; Ding, J.; Zhu, Z.; Belongie, S.; Luo, J.; Datcu, M.; Pelillo, M.; Zhang, L. Dota: A large-scale dataset for object detection in aerial images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Salt Lake City, UT, USA, 18–23 June 2018; pp. 3974-3983.

15. Zhu, P.; Wen, L.; Bian, X.; Haibin, L.; Hu, Q. Vision meets drones: A challenge. arXiv; 2018; arXiv: 1804.07437

16. Yu, H.; Li, G.; Zhang, W.; Huang, Q.; Du, D.; Tian, Q.; Sebe, N. The unmanned aerial vehicle benchmark: Object detection, tracking and baseline. Int. J. Comput. Vis.; 2019; 128, pp. 1141-1159. [DOI: https://dx.doi.org/10.1007/s11263-019-01266-1]

17. Li, K.; Wan, G.; Cheng, G.; Meng, L.; Han, J. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS J. Photogramm. Remote Sens.; 2020; 159, pp. 296-307. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2019.11.023]

18. Pang, J.; Li, C.; Shi, J.; Xu, Z.; Feng, H. R²-cnn: Fast tiny object detection in large-scale remote sensing images. IEEE Trans. Geosci. Remote Sens.; 2019; 57, pp. 5512-5524. [DOI: https://dx.doi.org/10.1109/TGRS.2019.2899955]

19. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst.; 2015; 1, pp. 91-99. [DOI: https://dx.doi.org/10.1109/TPAMI.2016.2577031]

20. Cai, Z.; Vasconcelos, N. Cascade r-cnn: Delving into high quality object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Salt Lake City, UT, USA, 18–23 June 2018; pp. 6154-6162.

21. Li, Y.; Chen, Y.; Wang, N.; Zhang, Z. Scale-aware trident networks for object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV); Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6054-6063.

22. Huo, B.; Li, C.; Zhang, J.; Xue, Y.; Lin, Z. SAFF-SSD: Self-Attention Combined Feature Fusion-Based SSD for Small Object Detection in Remote Sensing. Remote Sens.; 2023; 15, 3027. [DOI: https://dx.doi.org/10.3390/rs15123027]

23. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV); Venice, Italy, 22–29 October 2017; pp. 2980-2988.

24. Han, Y.; Duan, B.; Guan, R.; Yang, G.; Zhen, Z. LUFFD-YOLO: A Lightweight Model for UAV Remote Sensing Forest Fire Detection Based on Attention Mechanism and Multi-Level Feature Fusion. Remote Sens.; 2024; 16, 2177. [DOI: https://dx.doi.org/10.3390/rs16122177]

25. Law, H.; Deng, J. Cornernet: Detecting objects as paired keypoints. Proceedings of the European Conference on Computer Vision (ECCV); Munich, Germany, 8–14 September 2018; pp. 734-750.

26. Lu, X.; Li, B.; Yue, Y.; Li, Q.; Yan, J. Grid r-cnn. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Long Beach, CA, USA, 15–20 June 2019; pp. 7363-7372.

27. Lee, J.-H.; Son, C.-H. Trap-Based Pest Counting: Multiscale and Deformable Attention CenterNet Integrating Internal LR and HR Joint Feature Learning. Remote Sens.; 2023; 15, 3810. [DOI: https://dx.doi.org/10.3390/rs15153810]

28. Mai, S.; You, Y.; Feng, Y. SGR: An Improved Point-Based Method for Remote Sensing Object Detection via Dual-Domain Alignment Saliency-Guided RepPoints. Remote Sens.; 2024; 16, 250. [DOI: https://dx.doi.org/10.3390/rs16020250]

29. Tian, Z.; Shen, C.; Chen, H.; He, T. Fcos: Fully convolutional onestage object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV); Seoul, Republic of Korea, 27 October–2 November 2019; pp. 9627-9636.

30. Kong, T.; Sun, F.; Liu, H.; Jiang, Y.; Shi, J. Foveabox: Beyond anchorbased object detector. arXiv; 2019; arXiv: 1904.03797

31. Han, X.; Zhong, Y.; Zhang, L. An efficient and robust integrated geospatial object detection framework for high spatial resolution remote sensing imagery. Remote Sens.; 2017; 9, 666. [DOI: https://dx.doi.org/10.3390/rs9070666]

32. Van Etten, A. You only look twice: Rapid multi-scale object detection in satellite imagery. arXiv; 2018; arXiv: 1805.09512

33. Huang, M.; Liu, T.; Chen, Y. CViTF-Net: A Convolutional and Visual Transformer Fusion Network for Small Ship Target Detection in Synthetic Aperture Radar Images. Remote Sens.; 2023; 15, 4373. [DOI: https://dx.doi.org/10.3390/rs15184373]

34. Qiao, S.; Chen, L.-C.; Yuille, A. DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Nashville, TN, USA, 10–25 June 2021.

35. Singh, B.; Najibi, M.; Davis, L.S. SNIPER: Efficient Multi-Scale Training. Proceedings of the Computer Vision and Pattern Recognition, Computer Vision and Pattern Recognition (CVPR); Salt Lake City, UT, USA, 18–23 June 2018.

36. Kim, Y.; Kang, B.-N.; Kim, D. SAN: Learning Relationship Between Convolutional Features for Multi-Scale Object Detection. Proceedings of the Computer Vision—ECCV 2018; Munich, Germany, 8–14 September 2018; Lecture Notes in Computer Science pp. 328-343.

37. Bai, Y.; Zhang, Y.; Ding, M.; Ghanem, B. SOD-MTGAN: Small Object Detection via Multi-Task Generative Adversarial Network. Proceedings of the Computer Vision—ECCV 2018; Munich, Germany, 8–14 September 2018; Lecture Notes in Computer Science pp. 210-226.

38. Zhao, W.; Fang, Z.; Cao, J.; Ju, Z. SPA: Annotating Small Object with a Single Point in Remote Sensing Images. Remote Sens.; 2024; 16, 2515. [DOI: https://dx.doi.org/10.3390/rs16142515]

39. Yang, Y.; Lang, P.; Yin, J.; He, Y.; Yang, J. Data Matters: Rethinking the Data Distribution in Semi-Supervised Oriented SAR Ship Detection. Remote Sens.; 2024; 16, 2551. [DOI: https://dx.doi.org/10.3390/rs16142551]

40. Wang, J.; Xu, C.; Yang, W.; Yu, L. A Normalized Gaussian Wasserstein Distance for Tiny Object Detection. arXiv; 2021; arXiv: 2110.13389

41. Ge, Z.; Liu, S.; Li, Z.; Yoshie, O.; Sun, J. OTA: Optimal Transport Assignment for Object Detection. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Nashville, TN, USA, 10–25 June 2021.

42. Zhang, Y.; Nian, B.; Zhang, Y.; Zhang, Y.; Ling, F. Lightweight Multimechanism Deep Feature Enhancement Network for Infrared Small-Target Detection. Remote Sens.; 2022; 14, 6278. [DOI: https://dx.doi.org/10.3390/rs14246278]

43. Zhang, J.; Lei, J.; Xie, W.; Fang, Z.; Li, Y.; Du, Q. SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery. IEEE Trans. Geosci. Remote Sens.; 2023; 61, 5605415.