1. Introduction
With the rapid development of artificial intelligence and robotics, Human–Machine Integration (HMI) is gradually becoming a key direction for future technology. Intelligent, personalized, and ubiquitous human–machine interaction technologies are not only key to promoting the widespread application of HMI but also show great potential in various fields. For example, in medical diagnosis, doctors can perform remote surgeries using intelligent robots [1]; in the education field, students can engage in interactive learning with virtual teachers [2]; in the drone domain, operators can use intelligent human–machine interaction systems for remote control, executing fine-grained monitoring and operational tasks [3,4].
In human–robot intelligent interaction applications, using vision to rapidly and accurately detect targets is a crucial part of achieving natural human–robot interaction [5,6]. Traditional target detection methods based on deep learning convolutional neural networks and other intelligent algorithms rely on predefined categories and labeled data, achieving high detection accuracy and speed in specific, well-defined tasks. These methods have been widely applied in scenarios such as industrial component inspection [7] and service industry face recognition [8,9]. For instance, algorithms like Faster R-CNN [10] and YOLO [11,12] have achieved remarkable results in target detection within images and videos and have been widely applied in fields such as industrial automation, autonomous driving, and security monitoring [13,14].
However, in complex HMI application scenarios, the target detection environment is highly variable, and the detection objects belong to a wide range of categories, making it difficult for traditional object detection methods to effectively meet the requirements. Specifically, the following issues are particularly prominent: Traditional object detection methods exhibit instability in environments with significant changes, making it difficult to achieve consistent high-precision detection. In human–machine interaction, the categories of targets are diverse, and traditional methods struggle to handle unseen categories. Furthermore, HMI applications typically require real-time detection, and traditional deep learning-based methods are relatively slow when processing low-resolution images.
In recent years, text-guided object detection methods, which use natural language to describe targets and dynamically adjust detection objectives, have demonstrated the capability to achieve object detection in complex and dynamic environments. These methods provide new technical solutions for various human–machine intelligent interaction applications. These methods provide new technical solutions for various human–machine intelligent interaction applications. For example, Q. Feng et al. [15] proposed a natural language description-based object detection framework that can dynamically locate targets in unannotated images. Similarly, X. Li et al. [16] demonstrated the robustness of text-guided object detection in complex environmental conditions in their research. This technology has been widely applied in areas such as smart homes, intelligent security, and autonomous driving, significantly improving the efficiency and accuracy of human–machine interaction. For example, in video object detection from drone footage, the language-guided approach can effectively improve the precision of small target detection and enhance the understanding of the internal structural information of the targets [17].
Despite the widespread progress in text-guided object detection methods, which can dynamically adjust the target objects to be detected through text and handle some unseen categories with textual guidance, there is still a certain gap in detection accuracy between the current text-guided object detection methods and traditional object detection methods. Therefore, the development of innovative detection methods that can improve the accuracy of text-guided object detection is of significant research importance.
Therefore, this paper introduces an enhanced text-guided object detection method based on an improved YOLO-World framework. The primary contributions of this paper are as follows:
Improved visual processing module: To enhance the processing capabilities for low-resolution images and strengthen the feature representation of small and medium-sized objects, we introduce the space-to-depth layer followed by a non-strided convolution layer (SPD-Conv) module [18]. This module effectively improves the model’s performance on small- and medium-sized target objects by combining spatial transformation and a non-strided convolution layer.
Efficient multi-scale attention mechanism: This paper is the first to apply the efficient multi-scale attention (EMA) mechanism [19] to text-guided visual feature representation, significantly enhancing the model’s ability to accurately detect objects based on text descriptions. Additionally, we introduce the spatial attention (SA) module [20], which focuses on specific regions mentioned in the text, thereby achieving more precise target localization and recognition.
Extensive experimental validation: We conducted extensive ablation studies to systematically verify the effectiveness of each introduced module. The experimental results show that the proposed method significantly outperforms four advanced methods on the custom tool dataset, especially in the knife detection task, achieving an accuracy of 93.3%, which is a 4.4% improvement over the baseline model. Additionally, we tested the proposed method on the public COCO dataset, demonstrating its good adaptability and cross-dataset generalization capability, with an accuracy of 34.7% for detecting new categories.
2. Related Work
Current text-guided object detection techniques primarily adopt the following three strategies: deep learning methods, attention mechanisms, and cross-modal learning.
The first category, deep learning methods, leverages the excellent feature extraction capabilities of multi-layer neural network architectures to automatically learn complex representations and patterns in the data, accurately matching text descriptions with visual targets to improve detection accuracy. Related innovative research in this area is also continuously deepening. For instance, Ren et al. [10] proposed a method using a deep learning-based region proposal network for object detection. This method generates a series of high-quality candidate regions at the initial detection stage by sharing feature computations with a convolutional neural network, thereby enhancing the accuracy and efficiency of detection. Anderson et al. [21] proposed a method for object detection that combines Bottom-Up and Top-Down strategies. This method enhances the capability of text-guided object detection by integrating rich low-level features and high-level semantic information, but its detection effectiveness in complex scenes is limited.
The second category, attention mechanisms, leverages their superior sequence modeling capabilities to integrate natural language text with visual information. This integration allows for dynamic focus on relevant sections within images and text, thereby enhancing the accuracy and precision of target recognition and localization. For instance, Carion et al. [22] proposed a DETR model method based on the Transformer architecture for end-to-end object detection. This method enhances object localization capabilities by directly generating detection results through the encoding-decoding structure of the Transformer. However, the model’s effectiveness is limited when dealing with long-tailed distribution data, and it has a higher computational complexity. To address this, Zhu et al. [23] proposed a Deformable DETR object detection method, which introduces a deformable attention mechanism to optimize object detection performance. This mechanism dynamically adjusts the allocation of computational resources to reduce computational costs and accelerate the model’s convergence and inference speed. However, the detection effectiveness in scenarios with complex text descriptions is not optimal.
The third category, cross-modal learning mechanisms, effectively fuse text and visual information by utilizing a shared feature space to capture the correlations between different modalities in order to detect visual objects related to textual descriptions. These methods contribute to enhancing the accuracy and robustness of object detection by effectively fusing cross-modal information, making them a current research focus. For example, Zareian et al. [24] proposed the OVR-CNN model, which uses a pre-trained visual encoder with image-text pairs to achieve alignment between global images and textual descriptions, thereby accomplishing object detection tasks. However, the OVR-CNN model primarily focuses on the alignment between the overall image and textual descriptions, neglecting the fine-grained alignment between image regions and text, which limits its performance in complex scenarios. To address this, Zhong et al. [25] proposed the RegionCLIP model, which establishes associations between image local regions and textual descriptions, achieving alignment between them to enhance the model’s object detection precision in complex tasks. Zhou et al. [26] introduced the Detic model, which treats detection tasks as multi-label classification problems. By directly predicting object categories through a pre-trained visual encoder, it achieves efficient object detection. Additionally, to balance the efficiency of object detection with the capability of processing text information, Cheng et al. [27] proposed the YOLO-World model. This model combines the efficient detection framework of the YOLO series [12] with the large-scale cross-modal pre-trained model CLIP [28], demonstrating strong text comprehension and cross-modal reasoning abilities and performing well in diverse object detection application scenarios.
In summary, among the three types of text-guided object detection methods, the first type utilizes the powerful feature extraction capability of deep learning to detect visual objects. However, this method requires a large amount of labeled data and has a longer training time. The second type employs the good sequence modeling ability of the attention mechanism to detect target objects. However, this method has high algorithm complexity and poor real-time performance. The third type uses a cross-modal learning mechanism to integrate text and visual information for object detection. Its efficient detection framework enables real-time object detection. However, the detection accuracy of this method for searching and identifying specific targets needs further improvement compared to traditional object detection methods.
In addition, the YOLO-World model, which is directly related to our research, is based on the traditional YOLO object detection framework proposed by Redmon et al [29]. It uses a simple convolutional architecture for real-time object detection and divides the input image into grids, predicting a fixed number of bounding boxes and their confidence scores. By combining the prediction results of all grids, the final detection results are generated. However, traditional YOLO object detection methods are limited by the categories defined during training. If new target categories need to be detected, it usually requires retraining the model or fine-tuning. The YOLO-World model combines the efficient detection capability of the traditional YOLO framework with the powerful text understanding and cross-modal reasoning ability of the CLIP model. It can be trained using unlabeled data because it guides the detection process through text descriptions. This has a significant advantage in situations where labeled data is insufficient, reducing the cost and time of data preparation.
Therefore, this paper utilizes the cross-modal learning mechanism of the YOLO-World model and makes improvements on this basis to enhance the model’s detection accuracy. The structure of the remaining Sections of this paper is as follows: Section 3 presents the specific research system of this paper; Section 4 elaborates on the improved YOLO-World text-guided object detection method; Section 5 details the object detection verification and comparison experiments; Section 6 describes the research results and conclusions.
3. System
The text-guided object detection system based on improved YOLO-World is shown in Figure 1. The system mainly consists of three parts: the human–robot voice interaction object detection device, the text-guided object detection method based on the improved YOLO-World, and the experimental results and evaluation. Initially, the operator sends specific object detection voice commands to the human–robot interaction object detection device through natural language interaction, and the audio data obtained by the microphone is converted into text information via a voice-to-text module. Meanwhile, the Kinect camera at the end of the robot is used to capture the images of the targets to be detected and perform preprocessing. Next, the visual information processing module and the text information processing module separately extract features from the acquired image and text information. Then, the text-visual fusion module integrates the extracted text and image features, thereby completing the detection of specific targets in human–robot interaction. Finally, experimental research is conducted using the text-guided object detection method based on the improved YOLO-World, and the object detection accuracy of the research method is evaluated using three metrics: AP, AP0.5, and AP0.75. The main innovation of this study focuses on the text-guided object detection method section. The feature representation capability of target objects in images is enhanced by incorporating SPD-Conv into the visual information processing module, and the accuracy of object detection is improved by incorporating EMA and SA mechanisms into the text-visual fusion module.
4. Methods
The text-guided object detection method based on improved YOLO-World is shown in Figure 2. The method consists of four parts: (1) text branch, (2) visual branch, (3) text-visual fusion module, and (4) prediction head. Initially, the input text is processed by the CLIP text model for feature extraction, generating the corresponding text embeddings. Meanwhile, the input image is processed by the improved YOLOV8 model for feature extraction, producing feature maps of three sizes: X3, X4, and X5. The text embeddings and feature maps are fused in the text-visual fusion module. The fused features are processed by a regional text-matching algorithm to identify the target area in the image with the highest similarity to the input text and return the result of object detection.
4.1. Text Branch
The text branch involves feature extraction from the input text to generate corresponding text embeddings. Compared to text encoders that rely solely on textual information [30], the CLIP [28] text encoder demonstrates stronger visual-semantic mapping capabilities in associating visual objects with text. Therefore, the text branch employs a CLIP pre-trained text encoder. Specifically, the input text is first processed through a lookup table to map the noun phrases “tape measure”, “scissors”, and “pliers” to their respective embedding representations. Subsequently, these embeddings are used as input for the text encoder to produce the corresponding high-level text embeddings, W ∈ R^(C × D), where C represents the number of noun phrases, and D is the embedding dimension.
4.2. Visual Branch with SPD-Conv
The visual branch extracts features from the input image data, producing the necessary feature maps. The visual branch employs an improved YOLOV8 [12] backbone to extract multi-scale features from images. In the original YOLOV8 backbone, the use of stride convolutions and pooling layers can reduce the spatial dimensions of the feature maps, thereby reducing computational load and increasing the receptive field. However, this design may result in the loss of fine-grained image information, particularly when dealing with low-resolution images and small to medium-sized target objects. To enhance object detection accuracy in such scenarios, the SPD-Conv module is introduced into the YOLOV8 backbone. The improved visual branch is shown in Figure 3, where Figure 3a represents the improved YOLOV8 backbone, and Figure 3b shows the SPD-Conv module with a scaling factor of scale = 2.
SPD-Conv rearranges the elements of the feature maps to achieve down-sampling without spatial information loss, transforming spatial information to the channel dimension. SPD-Conv processes the feature maps output by the convolutional module as follows: initially, the input feature map X(S, S, C1) is transformed through the SPD layer, sliced, and reassembled into multiple sub-maps, each with a size of (S/scale, S/scale, C1). These sub-maps are concatenated along the channel dimension to form an intermediate feature representation X′ (S/scale, S/scale, scale2C1). Subsequently, the feature map is processed with a non-strided convolutional layer with a stride of 1, converting the intermediate representation X′ into the final output X″ (S/scale, S/scale, C2), where C2 represents the number of filters in the convolutional layer (C2 < scale2C1). The use of the SPD-Conv module preserves the spatial information of the feature maps, avoiding information loss in traditional down-sampling, thereby finely tuning the image feature representation and improving the model’s detection accuracy in low-resolution images and complex scenes. The improved YOLOV8 backbone generates three feature maps of different scales, X3, X4, and X5, which serve as input for the text-visual fusion module.
4.3. Text-Visual Fusion Module with SA and EMA
The text-visual fusion module merges text embeddings with visual feature maps to obtain the fused feature information. The text-visual fusion module employs the RepVL-PAN structure, which constructs a feature pyramid from the three-scale feature maps output by the visual branch through bottom-up and top-down paths, aiding in the fusion of image features at different scales, Spatial Attention-Text-guided CSPLayer (SA-T-CSPLayer) is used to integrate text embeddings into multi-scale image features, which is a CSPLayer (Cross-Stage Partial Layers) [12] incorporating spatial attention to enhance the capability of the model to represent critical features, aiding the model in better-capturing image regions related to the text. Additionally, the I-Pooling Attention (Image-Pooling Attention) operation proposed by the YOLO-World model is used to update the text embeddings, integrating image information into the text embeddings, further enhancing the interaction between image features and text features, and improving the visual semantic representation capabilities of the text. During the down-sampling process of image features, the EMA attention mechanism module is introduced to capture the information of image features at different scales after fusion, enhancing the capability of the model to represent features and aiding in the dynamic weighting of features at different scales.
As shown in Figure 2, the SA-T-CSPLayer is used for bottom-up or top-down text-visual feature fusion, expanding the CSPLayer by integrating text information into multi-scale image features to achieve more effective information fusion. The structure of SA-T-CSPLayer is shown in Figure 4.
As shown in Figure 4, given the text embedding W and image features Xl ∈ R^(H × W × D) (l ∈ {3, 4, 5}), the input image features Xl are split, with one portion processed by the Spatial Attention mechanism and Dark Bottleneck modules, denoted as f(Xl), and the resulting feature maps are fed into Max-Sigmoid, while the text embedding is also processed by Max-Sigmoid, where the text embedding is aggregated into image features through Equation (1) by Max-Sigmoid attention.
(1)
where δ represents the Sigmoid function, and Xl’ denotes the updated image features.Through this operation, regions with higher correlation to text vocabulary in the feature maps are identified, enabling the updating of images by text. The updated image features are concatenated with the initially split feature maps, forming a residual connection that produces a richer output feature map.
In the process of image down-sampling, to enhance the text-guided visual feature representation capability, the EMA attention mechanism module is introduced, and the structure of EMA is shown in Figure 5. The module operates as follows: first, the features resulting from the fusion of text and image are used as input to the module, and then the features are divided into g groups of sub-features, which are inputted into parallel sub-networks. The network extracts feature map attention weights through three paths: two 1 × 1 branches and one 3 × 3 branch. The 1 × 1 branches employ one-dimensional global average pooling, encode channel information, and then generate channel attention maps through a shared 1 × 1 convolutional layer and Sigmoid function, which are multiplied with the original sub-features to emphasize key features. The 3 × 3 branch captures multi-scale feature representations through 3 × 3 convolutions. Finally, the outputs of the 1 × 1 and 3 × 3 branches are processed through two-dimensional global average pooling and a SoftMax function to generate spatial attention weights, which are then combined with the original input features through matrix dot product, element-wise addition and Sigmoid activation to obtain the spatial attention map, resulting in the processed feature map.
4.4. Prediction Head
The prediction head processes the features after text and vision fusion and outputs the final detection results. The feature map with semantic information output by the text and vision fusion module and the image-aware text embedding are used as inputs for the prediction head. According to previous work [12], a decoupled head with two 3 × 3 convolutions is used to regress the bounding boxes {bk} (k ∈ {1, 2,…, K}) and object embeddings {ek} (k ∈ {1, 2,…, K}), where K represents the number of objects. A text contrastive head is proposed to obtain the object-text similarity sk,j through Equation (2).
(2)
where wj ∈ W is the j-th text embedding, and |ek| and |wj| represent the L2 normalization of ek and wj, respectively.Introducing learnable scaling factor α and shift factor β, the model automatically adjusts the similarity measure during training, enhancing the stability and flexibility of the model and improving the connection ability between visual objects and text descriptions. The text contrastive head is used to calculate similarity scores to match categories in text with objects in images. Furthermore, through the Box Head, the boundary boxes are predicted, the target object categories and positions are determined through region-text matching, and the final detection results are output.
5. Experiments
5.1. Datasets
In order to verify the detection capability of the algorithm in the laboratory scene, a custom dataset was created. The dataset includes seven different tool categories: knife, Phillips screwdriver, slotted screwdriver, scissors, wire stripper, pliers, and tape measure, with nearly 5000 images taken from various angles of the test bench under different backgrounds and lighting conditions using the Azure Kinect DK camera, and after a filtering and detailed labeling process, data augmentation techniques such as color enhancement, random affine transformation, and random flipping were applied, resulting in a total of 4218 high-quality images selected to create a custom dataset specifically for this study. The ratio of training, validation, and test sets in the custom dataset is 8:1:1. For more details about the dataset, refer to Table 1.
In addition, to verify the robustness of the algorithm, this study used the public COCO dataset [31], which has 48 base classes and 17 novel classes. There are 107,761 training images containing 665,387 bounding box annotations for base classes and 4836 test images containing 28,538 bounding box annotations for both base and new classes.
5.2. Experimental Setup
In order to verify the feasibility of the proposed method for text-guided object detection tasks, an experimental apparatus for human–robot interaction object detection tasks was constructed, and relevant experimental research was carried out. The experimental setup for the human–robot interaction object detection task, as shown in Figure 6, comprises the following main components: DOBOT CR5 robot, Kinect camera, PGC-50-35 gripper, Maillard M2X wired microphone, and the tools to be detected. The DOBOT CR5 robot has a working radius of 900 mm and a repeat positioning accuracy of 0.03 mm. The Azure Kinect depth camera can output color images with a resolution of 3840 × 2160, and its depth measurement range is 250–2880 mm. The repeat positioning accuracy of the PGC-50-35 gripper is 0.03 mm. The frequency response of the Maillard M2X wired microphone is 30 Hz~20 kHz, and the sensitivity is between −36 dB~−30 dB.
Table 2 provides all the details of the experimental software configuration. During training, an open-source CLIP text encoder with pre-trained weights is used to encode the input text, and the YOLO-World-S model pre-trained on the Objects365 [32] and GoldG [33] datasets is used for parameter initialization. The AdamW optimizer is used to iterate the pre-trained model for 100 epochs, with an initial learning rate set to 0.0002.
5.3. Evaluation Indicators
When conducting comparative experiments of different algorithms on the custom dataset, AP, AP0.5, and AP0.75 are used to evaluate the detection performance of the model. Average Precision (AP) is the area under the precision-recall curve, as shown in Equation (3). AP0.5 is the average precision when the Intersection over Union (IoU) threshold between the predicted box and the ground truth box is 0.5, and AP0.75 is the average precision at an IoU threshold of 0.75.
(3)
where P represents precision, which is the ratio of the number of positive samples predicted by the model to the total number of samples detected. R represents recall, which is the ratio of the number of positive samples correctly predicted by the model to the total number of positive samples that actually appear.When conducting comparative experiments of different algorithms on the public COCO dataset, APbase, APnovel, and APall are used as evaluation metrics, representing the detection precision at an IoU threshold of 0.5 for the 48 base classes, 17 novel classes, and all classes on the COCO dataset, respectively.
5.4. Experimental Results
5.4.1. Ablation Experiment
To assess the effectiveness of the three optimization strategies introduced in the proposed method, namely the performance enhancement achieved by the SPD-Conv module, EMA module, and SA module, as well as which combination achieves better results, a series of ablation experiments were conducted on the custom dataset specifically for this study. The data in Table 3 represent the experimental results when individual modules, pairs of modules, and all three modules are active simultaneously.
The experimental results show that after introducing the SPD-Conv module, EMA module, and SA module into the proposed method, the AP increased by 1.9%, 1.6%, and 1.3%, respectively. In the case of pairwise combinations, the best performance was observed when YOLO-World-S was combined with the SPD-Conv module and EMA module, reaching an AP of 79.3%, which is a 2.3% increase over the original YOLO-World-S, with an inference time of 8.8 ms and a parameter size of 13.3 M. When all three optimization strategies were applied, the AP reached 79.5%, a 2.5% improvement over the original YOLO-World-S, and the AP0.75 reached 95.2%.
The experimental results indicate that when the SPD-Conv module is introduced alone, the inference time increases to 9.1 ms, and the parameter size is 13.3 M, indicating that while the SPD-Conv module enhances detection accuracy, it also increases the demand for computational resources, but the overall performance improvement is significant. When the EMA module is introduced alone, the inference time is 8.2 ms, and the parameter size remains at 13.0 M, indicating that the EMA mechanism not only improves detection accuracy but also reduces the model’s inference time, verifying its capability for attention computation across different feature scales and enabling more efficient fusion of text and visual information. When the SA module is introduced alone, the inference time is 8.6 ms, and the parameter size is 13.0 M; the SA module can focus more accurately on key regions by introducing an attention mechanism in the CSPLayer, thereby improving the detection accuracy of target objects. In the case of pairwise combinations, the SPD-Conv module and the EMA module perform best together, achieving an AP of 79.3%, which is a 2.3% improvement over the original YOLO-World-S. The inference time is 8.8 ms, and the number of parameters is 13.3 M. This indicates that the combination of the SPD-Conv module and the EMA module can further improve the model’s detection accuracy and efficiency. When all three modules are used simultaneously, the target detection performance is the best, achieving an AP of 79.5%. This indicates that under this optimization strategy, the model not only accurately detects the specified target objects but also precisely delineates them, ensuring the accuracy of target object localization. Although the introduction of these modules slightly increases the inference time and model complexity, leading to an increase in computational requirements, the improvement in detection accuracy is significant. Therefore, the improved YOLO-World-S method using all three optimization strategies will be adopted for subsequent experiments.
5.4.2. Comparison Experiments Based on Custom Dataset
To demonstrate the superior accuracy of the proposed method in object detection, comparative experiments were conducted between the proposed method and four advanced methods: RegionCLIP [25], Detic [26], OVR-CNN [24], and YOLO-World-S [27], to validate the superior accuracy of the proposed method in object detection. First, under the same conditions, comparative experiments on object detection were conducted on the custom dataset. Table 4 and Figure 7 present the results of the comparative experiments on object detection using the custom dataset, with the evaluation metrics being AP0.5, AP0.75, and AP.
The experimental results show that the proposed method achieved better performance on the custom dataset, with the highest AP0.5 and AP0.75 among the several methods compared. Among the seven categories of tools, for G03 (knife), the highest detection accuracy AP0.75 in the comparative methods was 89%, while the AP0.75 of the proposed method reached 93.3%, a significant increase of 4.4%.
The experimental results indicate that the proposed method can achieve higher detection accuracy for tool-class target objects, verifying the superiority of the proposed method over the four comparative methods in terms of target detection accuracy on the custom dataset. The enhancement of detection accuracy for small and medium-sized tools such as knives is attributed to the introduction of the SPD-Conv module, which reinforces the capturing capability of detailed information for small targets through spatial depth convolution. Through in-depth data analysis, we found that the proposed method can more accurately identify and locate tools with similar appearances or easily confused targets, such as scissors, pliers, and wire strippers, demonstrating its robustness in complex scenes. The EMA mechanism enhances the model’s focus on key regions through cross-branch learning and multi-scale feature fusion, thereby improving the discriminative ability for similar targets. Additionally, the superiority of the proposed method in detection accuracy is not only reflected in individual categories but also consistently across multiple categories in the entire custom dataset. This indicates that the proposed method possesses good generalization capabilities and can effectively address different types of object detection tasks.
5.4.3. Comparison Experiments Based on COCO Dataset
To validate the object detection performance of the proposed method on a public dataset, it was compared with four advanced methods in object detection experiments under the same conditions on the COCO dataset. The experimental results of the object detection accuracy comparison of advanced algorithms on the COCO dataset are presented in Table 5.
The experimental results show that the proposed method achieved better performance. Compared with previous methods, the proposed method achieved the best performance with an APnovel of 34.7% among the four comparative algorithms. Additionally, compared to the original YOLO-World-S method, the proposed method improved by 2.2% in APnovel and by 2.4% in APall. Meanwhile, the proposed method also showed an improvement in APbase for the base classes compared with other advanced methods.
The experimental results indicate that on the public COCO dataset, the proposed method achieved significant progress compared with the four advanced methods, especially showing the best performance in APnovel. Compared with RegionCLIP, Detic, and OVR-CNN, YOLO-World-S, and the proposed method introduce top-down and bottom-up pathways to establish multi-scale image features and the corresponding feature pyramids, thereby achieving multi-level cross-modal fusion between image features and text embeddings. Moreover, by using regional text contrastive loss for learning, the method unifies detection data, localization data, and image-text data as region-text pairs, hence demonstrating strong detection capabilities on unseen target objects. The proposed method introduces an EMA attention mechanism in the top-down and bottom-up text-vision fusion process, which effectively captures and utilizes key feature information in the image through a cross-scale spatial learning mechanism. By matching the predicted regions with the corresponding text descriptions, the model gains a better understanding of the relationship between objects and their descriptions, achieving efficient visual-semantic interaction. Furthermore, the SA mechanism enhances the model’s understanding of the overall image structure by integrating spatial and channel-wise attention. By continually focusing on text-related visual feature information, the model’s performance on complex and unseen visual tasks is augmented, leading to an improvement in detection accuracy for new class target objects. This confirms the superiority of the proposed method over the four compared advanced methods in terms of object detection accuracy.
5.4.4. Comparison Experiment of Object Detection Application
To verify the effectiveness of the proposed method in text-based object detection, four advanced methods are compared with the proposed method. The experiment is conducted under four different scene conditions, using the experimental setup for the human–robot interaction object detection task shown in Figure 6. Four control commands are issued through the microphone, each containing a known object (scissors), a color-described object (red screwdriver), an unseen object (wrench), and an object in a cluttered scene (tape measure), to perform the comparative experiment of object detection. The comparative results of text-based object detection are shown in Table 6.
The first column of data in the experimental results shows that all five methods can detect the area of the target object, scissors, corresponding to the text and box it out, but both OVR-CNN and RegionCLIP have deviations in the position of the detection box from the actual scissors. The proposed method, as well as YOLO-World-S and Detic, all accurately locate the detection box on the scissors, with a small deviation from the actual position of the scissors. The second column of data shows that when the input text contains information related to the description of the object’s color, OVR-CNN cannot distinguish between the yellow screwdriver and the red screwdriver, and boxes both of them out. In contrast, the proposed method and YOLO-World-S, Detic, and RegionCLIP are all able to accurately and uniquely box out the required red screwdriver. The third column of data shows that the target area corresponding to the input text is the area where the target object, the wrench, has never appeared in the dataset. OVR-CNN incorrectly boxes the scissors as a wrench, and both Detic and RegionCLIP also find the area where the wrench is located, but there is some deviation from the actual position of the wrench. However, the proposed method and YOLO-World-S are both able to accurately detect the wrench and box it out. The fourth column of data shows that for the target objects corresponding to the text in cluttered scenes, the proposed method can accurately locate and delineate the target object (tape measure), while the other four methods do not accurately find the target object.
The experimental results indicate that the proposed method can accurately determine the position of target objects from input text information, even when dealing with color descriptions and objects not seen by the model, and it remains capable of precisely bounding the target objects in various backgrounds. Specifically, the first column of the experimental data shows that the proposed method can accurately localize the detection box for scissors. This is primarily due to the SPD-Conv module, which dynamically adjusts the parameters of the convolutional kernel, enabling the model to capture more detailed information in the image and thereby enhancing the accuracy of object localization. The second column of the data shows that the proposed method can accurately distinguish between a yellow screwdriver and a red screwdriver. This is attributed to the introduction of the EMA attention mechanism in the text-visual fusion section, which enhances the model’s sensitivity to color features and its ability to recognize objects of different scales through multi-scale feature interaction learning. The third column of the data shows that the proposed method can accurately detect and box a wrench that is not present in the dataset. This indicates that the proposed method has strong generalization capabilities and can handle unseen objects, which is attributed to the EMA attention mechanism’s ability to effectively capture key features and improve the model’s generalization through cross-scale information interaction. The fourth column of the data shows that the proposed method can accurately locate and box a tape measure even when the target object is partially occluded and the scene is complex. This further demonstrates the effectiveness of the SA attention mechanism in the CSPLayer, which calculates the weight of each position in the image feature map, allowing the model to focus on the important regions of the target object and ignore background interference, thus improving the accuracy of the detection box in occluded scenes. These results demonstrate the effectiveness of the proposed method in object recognition and detection across different scenarios.
6. Conclusions
This paper proposes a method based on the improved YOLO-World for enhancing the accuracy of text-guided object detection, which can effectively improve the detection accuracy of target objects. The improved method of YOLO-World integrating convolution and attention mechanism is elaborated in detail, focusing on the algorithmic theories related to the convolution module and attention mechanism, and a series of validation experiments have been conducted. Ablation experiments have verified the effectiveness of the introduced modules in the proposed method. In the comparison experiments based on a custom dataset, the detection accuracy of the proposed method under seven tool categories is superior to the four advanced methods compared, reaching 93.3% for tools such as knives, which is a 4.4% improvement over the original model. In the comparison experiments based on the COCO dataset, the model’s good applicability has been verified, demonstrating a detection accuracy of 34.7% on novel classes. In the comparison experiment of object detection applications, the effectiveness of the proposed text-based method has been verified in real-world scenarios. This research provides effective technical support for actual human–robot interaction collaboration tasks and will continue to be optimized and deployed in the laboratory’s robot control platform in the future.
In practical applications involving human–computer interaction, detection speed is a factor that needs to be considered. The current model, while striving for high detection accuracy, sacrifices some detection speed. Future plans include optimizing the algorithm and introducing efficient computational methods to improve detection speed, thereby meeting the requirements for real-time detection. Currently, the experiments are primarily focused on simple scenarios involving desktop tools, and the detection performance of the model in high-clutter and large-object scenarios needs further validation.
Q.D.: conceptualization, methodology, software, experiment, and writing—original draft preparation; E.Z.: validation, formal analysis, resources, writing—review and editing, and supervision; Z.L.: data curation, formal analysis, and writing—review and editing; X.Y.: investigation, and writing—review and editing; G.P.: validation, formal analysis, and writing—review and editing. All authors have read and agreed to the published version of the manuscript.
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.
The authors declare no conflicts of interest.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
The composition of custom dataset.
| Sample Category | Number of Training Set | Number of Validation Set | Number of Test Set | Total |
|---|---|---|---|---|
| Phillips screwdriver | 2843 | 355 | 360 | 3558 |
| Slotted screwdriver | 2671 | 334 | 341 | 3346 |
| Knife | 2782 | 348 | 352 | 3482 |
| Tape measure | 2703 | 338 | 349 | 3390 |
| Scissors | 2775 | 347 | 362 | 3484 |
| Pliers | 2820 | 353 | 370 | 3543 |
| Wire strippers | 2654 | 332 | 328 | 3314 |
Software configuration.
| Name | Specific Information |
|---|---|
| Operating system | Linux (Ubuntu 20.04) |
| GPU | RTX3080Ti |
| Cuda | 11.3 |
| pytorch | 1.10.1 |
| python | 3.8.19 |
Ablation experiment results.
| YOLO-World-S | SPD-Conv | EMA | SA | AP | AP0.5 | AP0.75 | Inference Time/ms | Params/M |
|---|---|---|---|---|---|---|---|---|
| √ | 0.770 | 0.992 | 0.929 | 8.6 | 13.0 | |||
| √ | √ | 0.789 | 0.992 | 0.945 | 9.1 | 13.3 | ||
| √ | √ | 0.786 | 0.993 | 0.948 | 8.2 | 13.0 | ||
| √ | √ | 0.783 | 0.993 | 0.936 | 8.6 | 13.0 | ||
| √ | √ | √ | 0.793 | 0.994 | 0.951 | 8.8 | 13.3 | |
| √ | √ | √ | 0.791 | 0.993 | 0.946 | 9.1 | 13.3 | |
| √ | √ | √ | 0.786 | 0.993 | 0.949 | 7.9 | 13.0 | |
| √ | √ | √ | √ | 0.795 | 0.996 | 0.952 | 8.8 | 13.3 |
Comparison experimental results (G01: Phillips screwdriver, G02: slotted screwdriver, G03: knife, G04: tape measure, G05: scissors, G06: pliers, G07: wire strippers).
| Methods | Metrics | G01 | G02 | G03 | G04 | G05 | G06 | G07 |
|---|---|---|---|---|---|---|---|---|
| RegionCLIP [ | AP0.5 | 0.923 | 0.933 | 0.932 | 0.955 | 0.957 | 0.931 | 0.932 |
| AP0.75 | 0.901 | 0.929 | 0.885 | 0.766 | 0.952 | 0.905 | 0.915 | |
| Detic [ | AP0.5 | 0.942 | 0.927 | 0.912 | 0.962 | 0.933 | 0.925 | 0.933 |
| AP0.75 | 0.890 | 0.925 | 0.856 | 0.775 | 0.925 | 0.912 | 0.920 | |
| OVR-CNN [ | AP0.5 | 0.894 | 0.880 | 0.873 | 0.901 | 0.910 | 0.895 | 0.901 |
| AP0.75 | 0.860 | 0.874 | 0.805 | 0.695 | 0.890 | 0.873 | 0.897 | |
| YOLO-World-S [ | AP0.5 | 0.969 | 0.963 | 0.950 | 0.979 | 0.967 | 0.978 | 0.975 |
| AP0.75 | 0.944 | 0.963 | 0.890 | 0.769 | 0.967 | 0.948 | 0.954 | |
| Proposed | AP0.5 | 0.971 | 0.969 | 0.971 | 0.980 | 0.970 | 0.989 | 0.989 |
| AP0.75 | 0.950 | 0.965 | 0.933 | 0.803 | 0.970 | 0.960 | 0.965 |
Comparison experimental results based on COCO dataset.
| Methods | Vision | Text Model | AP novel | AP base | AP all |
|---|---|---|---|---|---|
| RegionCLIP [ | RN50-C4 | CLIP RN50 | 26.8 | 54.8 | 47.5 |
| Detic [ | RN50-FPN | CLIP | 27.8 | 47.1 | 45.0 |
| OVR-CNN [ | RN50-C4 | BERT | 22.8 | 46.0 | 39.9 |
| YOLO-World-S [ | YOLOV8-S | CLIP | 32.5 | 53.4 | 48.0 |
| Proposed | YOLOV8SC-S | CLIP | 34.7 | 57.1 | 50.4 |
Object detection comparison experimental results.
| Methods | Command 1: Please Give Me a Pair of Scissors | Command 2: Find the Red Screwdriver | Command 3: I Need a Wrench | Command 4: Fetch the Tape Measure and Give Me |
|---|---|---|---|---|
| RegionCLIP [ | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] |
| Detic [ | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] |
| OVR-CNN [ | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] |
| YOLO-World-S [ | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] |
| Proposed | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] |
References
1. Sharkawy, A.-N. Human-robot interaction: Applications. arXiv; 2021; arXiv: 2102.00928
2. Hamilton, A. Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education. Cureus; 2024; 16, e59747. [DOI: https://dx.doi.org/10.7759/cureus.59747] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38840993]
3. Li, Y.; Li, Q.; Pan, J.; Zhou, Y.; Zhu, H.; Wei, H.; Liu, C. Sod-yolo: Small-object-detection algorithm based on improved yolov8 for uav images. Remote Sens.; 2024; 16, 3057. [DOI: https://dx.doi.org/10.3390/rs16163057]
4. Wei, G.; Yuan, X.; Liu, Y.; Shang, Z.; Yao, K.; Li, C.; Yan, Q.; Zhao, C.; Zhang, H.; Xiao, R. OVA-DETR: Open vocabulary aerial object detection using image-text alignment and fusion. arXiv; 2024; arXiv: 2408.12246
5. Solanes, J.E.; Gracia, L.; Miro, J.V. Advances in Human–Machine Interaction, Artificial Intelligence, and Robotics. Electronics; 2024; 13, 3856. [DOI: https://dx.doi.org/10.3390/electronics13193856]
6. Zhang, X.; Shen, J.; Wang, Y.; Xiao, J.; Li, J. Zero-Shot Image Caption Inference System Based on Pretrained Models. Electronics; 2024; 13, 3854. [DOI: https://dx.doi.org/10.3390/electronics13193854]
7. Jung, D.-Y.; Oh, Y.-J.; Kim, N.-H. A Study on GAN-Based Car Body Part Defect Detection Process and Comparative Analysis of YOLO v7 and YOLO v8 Object Detection Performance. Electronics; 2024; 13, 2598. [DOI: https://dx.doi.org/10.3390/electronics13132598]
8. Chen, W.; Huang, H.; Peng, S.; Zhou, C.; Zhang, C. YOLO-face: A real-time face detector. Vis. Comput.; 2021; 37, pp. 805-813. [DOI: https://dx.doi.org/10.1007/s00371-020-01831-7]
9. Liu, L.; Wang, G.; Miao, Q. ADYOLOv5-Face: An Enhanced YOLO-Based Face Detector for Small Target Faces. Electronics; 2024; 13, 4184. [DOI: https://dx.doi.org/10.3390/electronics13214184]
10. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. Pattern Anal. Mach. Intell.; 2016; 39, pp. 1137-1149. [DOI: https://dx.doi.org/10.1109/TPAMI.2016.2577031] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27295650]
11. Hussain, M. YOLO-v1 to YOLO-v8, the rise of YOLO and its complementary nature toward digital manufacturing and industrial defect detection. Machines; 2023; 11, 677. [DOI: https://dx.doi.org/10.3390/machines11070677]
12. Jocher, G.; Chaurasia, A.; Qiu, J. YOLO by Ultralytics. 2023; Available online: https://github.com/ultralytics/ultralytics (accessed on 1 November 2024).
13. Ding, X.; Chen, X.; Wang, J.; Zhang, J. Lightweight Underwater Target Detection Using YOLOv8 with Multi-Scale Cross-Channel Attention. Comput. Mater. Contin.; 2024; pp. 1-10. [DOI: https://dx.doi.org/10.32604/cmc.2024.057655]
14. Li, J.; Wang, Y.; Liang, W.; Xiong, C.; Cai, W.; Li, L.; Liu, Y. Visual Anomaly Detection via CNN-BiLSTM Network with Knit Feature Sequence for Floating-Yarn Stacking during the High-Speed Sweater Knitting Process. Electronics; 2024; 13, 3968. [DOI: https://dx.doi.org/10.3390/electronics13193968]
15. Feng, Q.; Ablavsky, V.; Bai, Q.; Li, G.; Sclaroff, S. Real-time visual object tracking with natural language description. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; Snowmass Village, CO, USA, 1–5 March 2020; pp. 700-709.
16. Li, X.; Chen, X.; Qu, C.; Yu, B.; Liu, F.; Hu, X.; Zhu, D.; Lu, F.; Chen, R.; Yu, G. Open Set Object Detection in Complex Petrochemical Scenes. Proceedings of the 2024 7th International Symposium on Autonomous Systems (ISAS); Chongqing, China, 7–9 May 2024; pp. 1-7.
17. Zhang, P.; Li, Y.; Wang, S. RoICLIP: Text-Enhanced UAV-Based Video Object Detection. Proceedings of the International Conference on Image and Graphics; Nanjing, China, 22–24 September 2023; pp. 151-163.
18. Sunkara, R.; Luo, T. No more strided convolutions or pooling: A new CNN building block for low-resolution images and small objects. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases; Grenoble, France, 19–23 September 2022; pp. 443-459.
19. Ouyang, D.; He, S.; Zhang, G.; Luo, M.; Guo, H.; Zhan, J.; Huang, Z. Efficient multi-scale attention module with cross-spatial learning. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); Rhodes Island, Greece, 4–10 June 2023; pp. 1-5.
20. Xu, L.; Huang, J.; Nitanda, A.; Asaoka, R.; Yamanishi, K. A novel global spatial attention mechanism in convolutional neural network for medical image classification. arXiv; 2020; arXiv: 2007.15897
21. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; Zhang, L. Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Salt Lake City, UT, USA, 18–23 June 2018; pp. 6077-6086.
22. Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; Zagoruyko, S. End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision; Glasgow, UK, 23–28 August 2020; pp. 213-229.
23. Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; Dai, J. Deformable detr: Deformable transformers for end-to-end object detection. arXiv; 2020; arXiv: 2010.04159
24. Zareian, A.; Rosa, K.D.; Hu, D.H.; Chang, S.-F. Open-vocabulary object detection using captions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Nashville, TN, USA, 20–25 June 2021; pp. 14393-14402.
25. Zhong, Y.; Yang, J.; Zhang, P.; Li, C.; Codella, N.; Li, L.H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y. Regionclip: Region-based language-image pretraining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; New Orleans, LA, USA, 18–24 June 2022; pp. 16793-16803.
26. Zhou, X.; Girdhar, R.; Joulin, A.; Krähenbühl, P.; Misra, I. Detecting twenty-thousand classes using image-level supervision. Proceedings of the European Conference on Computer Vision; Tel Aviv, Israel, 23–27 October 2022; pp. 350-368.
27. Cheng, T.; Song, L.; Ge, Y.; Liu, W.; Wang, X.; Shan, Y. Yolo-world: Real-time open-vocabulary object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Seattle, WA, USA, 16–22 June 2024; pp. 16901-16911.
28. Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J. Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning; Vienna, Austria, 18–24 July 2021; pp. 8748-8763.
29. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Las Vegas, NV, USA, 27–30 June 2016; pp. 779-788. [DOI: https://dx.doi.org/10.1109/CVPR.2016.91]
30. Kenton, J.D.M.W.C.; Toutanova, L.K. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the NAACL-HLT; Minneapolis, MN, USA, 2–7 June 2019; 2.
31. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft coco: Common objects in context. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference; Zurich, Switzerland, 6–12 September 2014; Proceedings, Part V Volume 13, pp. 740-755.
32. Shao, S.; Li, Z.; Zhang, T.; Peng, C.; Yu, G.; Zhang, X.; Li, J.; Sun, J. Objects365: A large-scale, high-quality dataset for object detection. Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision; Seoul, Republic of Korea, 27 October–2 November 2019; pp. 8430-8439.
33. Kamath, A.; Singh, M.; LeCun, Y.; Synnaeve, G.; Misra, I.; Carion, N. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision; Montreal, BC, Canada, 11–17 October 2021; pp. 1780-1790.
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.