This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
1. Introduction
For unmanned ships and vessel traffic service (VTS), intelligent perception of the navigational environment is an important topic [1]. The navigational environment mainly includes two parts: the dynamic vessels and the navigational features marked by the navigational aids. According to the IALA (International Navigation and Lighthouse Administration Navigation Association), the definition of the term Aid to Navigation (AtoN) means any specific equipment, system, or service outside the ship, specifically used to assist navigators in determining their location or safe route or to warn them of dangers or obstacles to navigation [2]. AtoN mainly consists of buoy and beacon; the former is a floating object fixed at the bottom; the latter is a structure permanently set on the seabed or land. Both of them can be categorized as “marks.” They have distinctive shapes, colors, top marks, and other auxiliary markings which can be observed to indicate their purposes during the daytime. The relevant information about navigation marks is usually obtained through the Electronic Chart Display and Information System (ECDIS) or Automatic Identification System (AIS) [3]. However, it is a new challenge about how to visually and automatically detect navigational aids through the camera.
With the development of artificial intelligence technology, many intelligent detection technologies are applied to VTS [4, 5] and smart ships [6, 7]. Among them, the applications of deep learning technology in the detection and classification of ships are currently widely used [8, 9]. The purpose of this type of application is to supplement information about ships not in AIS [10]. For navigation marks, although their basic information can be obtained through ECDIS, sailors are still required to keep visual observing on their realtime states by the eye or with a telescope [11]. At the present stage, the detection and classification of navigation mark images is not as widely studied as ships [12]. Compared to ship image classification and recognition, there are fewer references available. In previous research [12], we exploited deep learning technology to study the navigation marks’ image recognition during the daytime. It proposed a fine-grained ResNet-based classification model to classify navigation marks named ResNet-Multiscale-Attention (RMA). The accuracy of this model reaches at 95.98% on a dataset including 10260 navigation mark images. However, the experimental results showed that the model also has some certain misclassifications of navigation marks, especially in aspect of the images with inconsistent shapes.
To solve these problems, this paper studied further to improve the classification model for navigation mark images, and the contributions are highlighted as follows.
(i) An improved navigation mark classification method with contour accentuation is proposed, and its classification accuracy arrives in 96.53%
(ii) An intelligent service system is developed and has been applied by the Changjiang Nanjing Waterway Bureau; it provides image recognition service of navigation marks on the Yangtze River
The contents of this article are organized as follows. Section 2 describes the related works. Section 3 describes the improved classification model for navigation mark images by contour accentuation method. Section 4 provides practical experimental results and discussion. Section 5 illustrates the intelligent application system. Finally, conclusions and future work are given in Section 6.
2. Related Work
In deep learning technology, convolution neural networks (CNN) are suitable for visual recognition and image classification tasks. AlexNet [13], VGG [14], GoogleNet [15], ResNet [16], and DenseNet [17] are some of the networks that attract attention from researchers. Various image classification methods based on CNN were applied to many fields, such as medical image analysis [18] and face recognition [19]. Some researches about vessel recognition also had been reported. Shi et al. [20] put forward a new deep learning framework, which combined the underlying functions and could effectively use useful information to classify the ship optical image. Oliveau et al. [21] proposed a new vessel classification theory based on semisupervised learning. Shin et al. [22] proposed a model using interest region combined a convolutional neural network for improving the ship images’ classification accuracy. Solmaz et al. [23] proposed a framework and a new loss function to recognize the marine and land vehicles in a fine-grained way using multitasking learning.
Comparing with the vessel images, the different types of navigation marks may only have subtle differences in certain specific positions. To some extent, their image classification is a fine-grained classification. An important method of fine-grained classification is the attention mechanism. The attention mechanism is essentially to imitate the way humans observe objects. Google [24] proposed a novel recurrent neural network model, which extracted information from images or videos by adaptively selecting regions or position sequences and only processing the selected areas with high resolution. Google [25] also presented an attention-based model for identifying multiple objects in an image. In addition to the research on the attention mechanism algorithm, many scholars apply the attention mechanism to image classification. Haut et al. [26] proposed a new visual attention-based classification algorithm. Yang [27] proposed a RetinaNet model based on attention mechanism to match and classify the target ship accurately. In our previous model for navigation mark image classification [12], an attention mechanism based on three scale fusion of feature map was proposed to locate the area of attention and obtain characteristic.
However, the attention mechanism weakens the contour features. The results of the previous study [12] show that the RMA model has misclassification due to inconsistent appearance. The contour accentuation method can correct these problems [28]. In some fields, this method was widely used. Shotton [29] proposed a new type of automatic visual recognition system based on local contour features, which can locate objects in space and scale. It also confirmed that contour was a powerful hint for the multiscale and the multitype visual object recognition. Lin [30] also developed a new technology for detecting fruits in natural environments based on contour information. Their experiments showed that the proposed method was competitive for most types of fruits in natural environments, such as green, orange, circular, and nonround. To obtain higher accuracy of ship recognition, a contour accentuation method combined a ship recognition method based on transfer learning was proposed to analyse the ship images to detect the ship types. The actual results showed that the contour accentuation method with the transfer learning could obtain higher accuracy in ship image recognition [31]. Obviously, contour features are helpful to visual recognition. Therefore, in this paper, contour accentuation was expected to complement the affect of attention mechanism, and it was combined into the RMA model for navigation mark classification to further improve accuracy.
Recently, there are some intelligent information systems were reported about navigation mark management and service [32–34]. However, these systems were mainly developed based on telemetry and remote control; their identification mechanism of navigation marks is different from image recognition. Qi et al. [35] proposed a maritime navigation mark system based on electromagnetic waves, and Zhang [36] proposed a navigation mark communication system based on WLAN. These systems mainly provide information service of navigation marks by position instead of visual recognition. In this paper, a novel intelligent service system for image recognition of navigation marks was developed, and it orients to the application scenarios from camera.
3. Classification Models for Navigation Marks
This section firstly introduces the classification model of navigation marks called ResNet-Multiscale-Attention (RMA) model, then describes how to combine the RMA model with contour accentuation.
3.1. The ResNet-Multiscale-Attention (RMA) Model
In the daytime, navigation marks can be recognized by their shape, color, and other auxiliary features. However, some kinds of navigation marks have a similar contour with subtle differences. Accordingly, for the visual navigation mark image recognition, the fine-grained image classification method is better than the general-level ones. Generally, in the deep neural networks of classification, low-level features have less semantic information but more information about the target’s position. Instead, high-level features have more semantic information but less detailed information about the target’s position. General-level models usually do not perform well in fine-level tasks with the high-level features [12].
To tackle the fine-grained classification of navigation marks, a model called RMA was proposed in which the ResNet-50 was enhanced by adding a multiple scale attention mechanism [12]. As shown in Figure 1, in the network structure of RMA, the images of navigation mark were enhanced firstly by an improved ResNet-50, then classified by the second ResNet-50. The first ResNet-50 layer is designed as an attention matrix to capture the attention regions. Three-channel feature maps
[figure omitted; refer to PDF]
Experiment results on a navigation mark image dataset showed that the RMA had classification accuracy about 95.98%, which was better than 94.14% of the ResNet-50.
3.2. RMA Model with Contour Accentuation
Contour features are helpful to visual recognition by enhancing target in the image, which was verified in many types of research and our other experiment about ship recognition. The multiple scale attention mechanism of RMA is aimed at locating the target’s region, and the objective of contour is the enhancement of the target’s features. In this paper, the contour accentuation method is considered to be combined into the RMA model for further improving the classification accuracy of navigation marks.
The original image of navigation mark as shown in Figure 2(a) is a
[figures omitted; refer to PDF]
The function
Furthermore, by Equation (3), which keeps the original color of pixels that is not on contour instead of white, an image with contour accentuation
To combine the contour accentuation method with the RMA model, the contour accentuation algorithm can be used as an image preprocessing method, and the RMA model adopts the navigation images with contour accentuation as inputs directly.
4. Experiments and Results
To validate the effectiveness of the RMA model with contour accentuation, a navigation mark image dataset is firstly preprocessed with contour accentuation and then trained and tested with the RMA model.
4.1. Dataset
A total of 10260 images of 42 kinds of navigation marks in the Yangtze River are collected. All images are clipped into a uniform size of
In Equation (2), critical value
[figures omitted; refer to PDF]
From 1 to 5, the affection of different
Therefore, finally chose
[figure omitted; refer to PDF]
In the back-end, there are three platforms which are deployed independently but interacted with each other through an interface. The web service platform is developed and deployed based on the framework of Spring Boot. It interacts with front-end directly, accepts and transforms the image of request into required size and format, then sends it to the recognition module and gets recognition result. In the recognition module, TensorFlow Serving is used to deploy the RMA models, and a REST API for navigation mark recognition based on the latest model is exposed to the web service platform. In the training module, the RMA will be trained periodically in TensorFlow. Simultaneously, the dataset was enlarged by the image collection process of the digital waterway system (the production system for channel maintenance in Nanjing Waterway Bureau), and the model will be saved with a version number, updated, and loaded into TensorFlow Serving.
The front-end can be a variety of clients, Web, APP, or WeChat Mini Program. The clients accept the uploaded image, send it to the web service platform, and get the responses of recognition and rendering them on the page as Figure 9 showed.
[figure omitted; refer to PDF]6. Conclusions and Future Work
This paper applies deep learning technology to study the navigation mark image recognition. It proposes a navigation mark classification model based on the combination of multiscale attention mechanism and contour accentuation. The effect of multiple scale attention mechanisms for improving classification accuracy has been validated in our previous works about the RMA model. This paper mainly focused on the impact of contour accentuation. Experimental results on 10260 navigation mark images showed that by enhancing the contour of the object, contour accentuation could improve the image classification accuracy of most general classification models. It also improves the RMA model well and increases the classification accuracy from 95.98% to 96.53%.
Based on the improved classification model, this paper further developed an intelligent service system for the recognition of navigation marks. The system has a flexible architecture based on front-end and back-end separation. It is connected with the digital waterway system to obtain a continuously updated dataset and then realized an automatic navigation mark recognition service including dataset preparation, model training, model deployment, and model update.
In the future, the value of
Acknowledgments
This work was partially supported by the Fundamental Research Funds for the Central Universities under Grant 3132019400. Thanks are due to the Changjiang Nanjing Waterway Bureau of the People’s Republic of China for providing the image dataset of navigation marks and application scenario of the research results. This work was also partially supported by the National Natural Science Foundation of China (Nos. 61906043, 61902313, 61902072, 62002063, 61877010, 11501114, and 11901100), the Fujian Natural Science Funds (Nos. 2020J05112, 2020J05111, 2020J01498, and 2019J01243), the Funds of Education Department of Fujian Province (No. JAT190026), and the Fuzhou University (Nos. 0330/50016703, 0330/50009113, 510930/GXRC-20060, 510872/GXRC-20016, 510930/XRC-20060, 510730/XRC-18075, 510809/GXRC -19037, 510649/XRC-18049, and 510650/XRC-18050).
[1] I. Im, D. Shin, J. Jeong, "Components for smart autonomous ship architecture based on intelligent information technology," Procedia Computer Science, vol. 134, pp. 91-98, DOI: 10.1016/j.procs.2018.07.148, 2018.
[2] "," International Dictionary of Marine Aids to Navigation, . October 2020, https://www.ialaaism.org/wiki/dictionary/index.php/Aid_to_Navigation
[3] X. Guo, "Application and management of AIS aids to navigation," Ship Electronic Engineering, vol. 36 no. 6, pp. 54-58, 2016.
[4] J. Pandy, K. Hasegawa, "Autonomous navigation of catamaran surface vessel," 2017 IEEE Underwater Technology (UT),DOI: 10.1109/ut.2017.7890342, .
[5] J. Zhuang, L. Zhang, S. Zhao, J. Cao, B. Wang, H. Sun, "Radar-based collision avoidance for unmanned surface vehicles," China Ocean Engineering, vol. 30 no. 6, pp. 867-883, DOI: 10.1007/s13344-016-0056-0, 2016.
[6] A. Garcia-Dominguez, "Mobile applications, cloud and bigdata on ships and shore stations for increased safety on marine traffic; a smart ship project," 2015 IEEE International Conference on Industrial Technology (ICIT), pp. 1532-1537, DOI: 10.1109/icit.2015.7125314, .
[7] Y. Tang, N. Shao, "Design and research of integrated information platform for smart ship," 2017 4th International Conference on Transportation Information and Safety (ICTIS), pp. 37-41, DOI: 10.1109/ictis.2017.8047739, .
[8] B. Liu, S. Wang, J. Zhao, M. Li, "Ship tracking and recognition based on Darknet network and YOLOv3 algorithm," Journal of Computer Applications, vol. 39 no. 6, pp. 1663-1668, 2019.
[9] H. Fu, Y. Li, Y. Wang, P. Li, "Maritime ship targets recognition with deep learning," 2018 37th Chinese Control Conference (CCC), pp. 9297-9302, DOI: 10.23919/chicc.2018.8484085, .
[10] Z. Li, L. Zhao, X. Han, M. Pan, F. J. Hwang, "Lightweight ship detection methods based on YOLOv3 and DenseNet," Mathematical Problems in Engineering, vol. 2020,DOI: 10.1155/2020/4813183, 2020.
[11] International Maritime Organization, "International Convention on Standards of Training, Certification and Watchkeeping for Seafarers, 1978, as amended in 1995," 1997.
[12] M. Pan, Y. Liu, J. Cao, Y. Li, C. Li, C. Chen, "Visual recognition based on deep learning for navigation mark classification," IEEE Access, vol. 8, pp. 32767-32775, DOI: 10.1109/ACCESS.2020.2973856, 2020.
[13] A. Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol. 60 no. 6, pp. 84-90, DOI: 10.1145/3065386, 2017.
[14] K. Simonyan, A. Zisserman, "Very deep convolutional networks for large-scale image recognition," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), .
[15] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, "Going deeper with convolutions," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), .
[16] K. He, X. Zhang, S. Ren, J. Sun, "Deep residual learning for image recognition," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, .
[17] G. Huang, Z. Liu, L. V. D. Maaten, K. Q. Weinberger, "Densely connected convolutional networks," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, .
[18] X. Yao, X. Wang, S. Wang, Y. Zhang, "A comprehensive survey on convolutional neural network in medical image analysis," Multimedia Tools and Applications,DOI: 10.1007/s11042-020-09634-7, 2020.
[19] M. Wang, W. Deng, "Deep face recognition: a survey," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 471-478, .
[20] Q. Shi, W. Li, F. Zhang, W. Hu, X. Sun, L. Gao, "Deep CNN with multi-scale rotation invariance features for ship classification," IEEE Access, vol. 6, pp. 38656-38668, DOI: 10.1109/ACCESS.2018.2853620, 2018.
[21] Q. Oliveau, H. Sahbi, "From transductive to inductive semi-supervised attributes for ship category recognition," IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 4827-4830, DOI: 10.1109/IGARSS.2018.8518265, .
[22] H. C. Shin, K.-I. Lee, "Classification maritime vessel image utilizing a region of interest extracted and convolution neural network," Journal of Korean Institute of Intelligent Systems, vol. 29 no. 4, pp. 321-326, DOI: 10.5391/JKIIS.2019.29.4.321, 2019.
[23] B. Solmaz, E. Gundogdu, V. Yucesoy, A. Koç, A. A. Alatan, "Fine-grained recognition of maritime vessels and land vehicles by deep feature embedding," IET Computer Vision, vol. 12 no. 8, pp. 1121-1132, DOI: 10.1049/iet-cvi.2018.5187, 2018.
[24] J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, K. Saenko, T. Darrell, "Long-term recurrent convolutional networks for visual recognition and description," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2204-2212, .
[25] J. Ba, V. Mnih, K. Kavukcuoglu, "Multiple object recognition with visual attention," International Conference on Learning Representations, .
[26] J. M. Haut, M. E. Paoletti, J. Plaza, A. Plaza, J. Li, "Visual attention-driven hyperspectral image classification," IEEE Transactions on Geoscience and Remote Sensing, vol. 57 no. 10, pp. 8065-8080, DOI: 10.1109/TGRS.2019.2918080, 2019.
[27] T. Yang, Z. Chen, Y. Lv, Y. Wu, B. Hua, "Multi-resolution ocean target detection method based on deep learning," Electronics Optics & Control, 2020. http://kns.cnki.net/kcms/detail/41.1227.TN.20200817.1258.020.html
[28] J. Victorino, F. Gómez, "Contour analysis for interpretable leaf shape category discovery," Plant Methods, vol. 15 no. 1,DOI: 10.1186/s13007-019-0497-6, 2019.
[29] J. Shotton, A. Blake, R. Cipolla, "Multiscale categorical object recognition using contour fragments," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30 no. 7, pp. 1270-1281, DOI: 10.1109/TPAMI.2007.70772, 2008.
[30] G. Lin, Y. Tang, X. Zou, J. Cheng, J. Xiong, "Fruit detection in natural environment using partial shape matching and probabilistic Hough transform," Precision Agriculture, vol. 21 no. 1, pp. 160-177, DOI: 10.1007/s11119-019-09662-w, 2020.
[31] C. Chen, Y. Zhang, W. Guo, M. Pan, L. Lyu, C. Lin, "Contour accentuation for transfer learning-based ship recognition method," Proceedings of the Web Conference 2020 (WWW’20),DOI: 10.1145/3366424.3382697, .
[32] S. Beatriz, C. Nicoleta, F. Francisco, "Artifical intelligence to determine if liquified natural gas in short sea shipping is a social bet," Ingeniería y Desarrollo, vol. 36, pp. 418-436, DOI: 10.14482/inde.36.2.10458, 2018.
[33] D. S. Cristea, L. M. Moga, M. Neculita, O. Prentkovskis, K. M. D. Nor, A. Mardani, "Operational shipping intelligence through distributed cloud computing," Journal of Business Economics and Management, vol. 18 no. 4, pp. 695-725, DOI: 10.3846/16111699.2017.1329162, 2017.
[34] M. Sun, "Research on management informationization of inland waterway," People’s Transportation, vol. 5, 2019.
[35] S. Qi, H. Zhang, J. Tian, "Research on wireless location method of short baseline marine beacons based on phase measurement," 2018 2nd IEEE Advanced Information Management,Communicates,Electronic and Automation Control Conference (IMCEC), pp. 1123-1129, DOI: 10.1109/imcec.2018.8469665, .
[36] J. Zhang, "Research of application of communication technology of WLAN based on ship," 2011 2nd International Conference on Artificial Intelligence, Management Science and Electronic Commerce (AIMSEC),DOI: 10.1109/AIMSEC.2011.6010055, .
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
Copyright © 2021 Yanke Du et al. This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Abstract
Sensing navigational environment represented by navigation marks is an important task for unmanned ships and intelligent navigation systems, and the sensing can be performed by recognizing the images from a camera. In order to improve the image recognition accuracy, this paper combined a contour accentuation algorithm into a multiple scale attention mechanism-based classification model for navigation marks. Experimental results show that the method increases the accuracy of navigation mark classification from 95.98% to 96.53%. Based on the classification model, an intelligent navigation mark recognition system was developed for the Changjiang Nanjing Waterway Bureau, in which the model is deployed and updated by the TensorFlow Serving.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
Details






1 Navigation College, Dalian Maritime University, Dalian 116026, China
2 College of Mathematics and Computer Sciences, Fuzhou University, Fuzhou 350108, China; Key Laboratory of Intelligent Metro of Universities in Fujian, Fuzhou University, Fuzhou 350108, China