Relative Localization and Circumnavigation of a UGV0 Based on Mixed Measurements of Multi-UAVs by Employing Intelligent Sensors

Abstract

Relative localization (RL) and circumnavigation is a highly challenging problem that is crucial for the safe flight of multi-UAVs (multiple unmanned aerial vehicles). Most methods depend on some external infrastructure for positioning. However, in some complex environments such as forests, it is difficult to set up such infrastructures. In this paper, an approach to infrastructure-free RL estimations of multi-UAVs is investigated for circumnavigating a slowly drifting UGV0 (unmanned ground vehicle 0), where UGV0 serves as the RL and circumnavigation target. Firstly, a discrete-time direct RL estimator is proposed to ascertain the coordinates of each UAV relative to the UGV0 based on intelligent sensing. Secondly, an RL fusion estimation method is proposed to obtain the final estimate of UGV0. Thirdly, an integrated estimation control scheme is also proposed for the application of the RL fusion estimation method to circumnavigation. The convergence and the performance are analyzed. The simulation results validate the effectiveness of the proposed algorithm for RL fusion estimations and of the integrated scheme.

1. Introduction

Multi-UAV RL and navigation not only play an increasingly crucial role in civil and military fields [1,2], but have also achieved notable successes in commerce, agriculture, and medical rescue [3,4,5]. Among them, UAV positioning technology [6] is a core element ensuring its safe and efficient operation. By controlling the circumnavigation of UAVs around a central point, the system [7] can achieve precise positioning and navigation in complex environments. In the application of circumnavigation, a multi-UAV RL system [8] must be capable of responding to changing environments and mission requirements in real time. Achieving a good real-time performance may necessitate more complex algorithms and hardware. Therefore, in the design and implementation of a multi-UAV RL system, it is essential to comprehensively address the aforementioned shortcomings and identify corresponding solutions to enhance the robustness and adaptability of the system.

The most common methods include external positioning systems, such as the Global Positioning System (GPS) [9,10] and anchor-based ultra-wideband (UWB) positioning [11,12], which have notably enhanced positioning accuracies. However, in specific environments, such as urban canyons, indoors, or during severe weather conditions, satellite signals may be obstructed or interfered with, thereby limiting the reliability and accuracy of traditional Global Navigation Satellite Systems (GNSSs) [13,14] in these situations. Furthermore, the deployment of an external positioning system can lead to complexities in maintenance and updating. In particular, in systems necessitating long-term operation, the maintenance of both hardware and software can pose challenges.

To address these challenges, researchers are integrating other positioning technologies into UAV systems, including vision sensors [15,16,17]. The integrated use of these technologies enables UAVs to operate in more intricate and demanding environments, accomplishing tasks such as navigating through urban buildings or conducting search and rescue operations in forests. While this approach eliminates the need for infrastructure, visual localization often entails extensive image processing and computational tasks, demanding high-performance hardware and sophisticated algorithms. This can pose challenges in meeting real-time requirements, particularly on resource-constrained UAV platforms.

On the other hand, there are also examples of infrastructure-independent solutions, but they may be inadequate in certain aspects. In [18], a method for RL using radar is proposed. Radar is capable of providing high-precision distance measurements and is very useful for precise RL. However, its equipment cost is high, and it is sensitive to ambient light and transparent objects. Radio Frequency Identification (RFID) systems are suitable for tag recognition at short distances and are applicable for indoor positioning [19]. However, for applications with large-range, high-precision RL requirements, the accuracy of the RFID system may be low. In [20], an RL algorithm based on the fusion of relative navigation sensors was proposed. It mainly involves combining information from different sensors, such as inertial navigation systems and magnetometers, to improve the robustness of RL. Nevertheless, the design and calibration of sensor fusion algorithms are relatively complex and can be susceptible to sensor errors. In [21], an RL algorithm based on visual–inertial navigation fusion was proposed. Combining visual and inertial navigation can overcome the sensor switching problem when transitioning between indoors and outdoors, providing more comprehensive RL information. However, covering large areas may require additional devices to capture enough feature points. The main idea of [22] is to present a weight matrix, simplifying the average consensus algorithm over mobile wireless sensor networks and thereby prolonging the network lifetime as well as ensuring the proper operation of the algorithm. However, the majority of wireless sensor networks discussed in this article are depicted as undirected graphs, which may not adequately address the complexities and fluctuations present in real-world environments.

More recently, a consensus-based leader–follower algorithm was developed in mobile sensor networks, where the goal for the entire network is to converge to the state of the leader [23]. However, this article does not address the positioning issue. A control strategy for a quadrotor elliptical target orbit based on uncertain and non-periodic updates of angle measurements was proposed in [24]. At the translational level, only orientation data are utilized, without incorporating the target’s prior position and velocity information. A position estimator is developed for locating unknown targets. However, the influence of measurement noise is not addressed. In [25], a UAV group circumnavigation control strategy is proposed, in which the UAV circumnavigation orbit is an ellipse whose size can be adjusted arbitrarily. However, the article does not include any research on RL-related aspects. Unlike the work in [26], earlier research focused on utilizing multiple agents to localize targets under fixed topology assumptions. This paper addresses the challenge of cooperative localization in a time-varying topology without an infrastructure, such as anchor points.

In this paper, a directed graph model is employed to represent exchanged information. As measurement failures may occur or a UAV could move beyond the sensing range of its neighbors, the directed graph describing the information flow relationship is time-varying. When a UAV strays beyond the operation area, it triggers a return-to-base protocol, where the UAV autonomously navigates back to the designated area using predefined waypoints or by following a path generated by the control system. To address the RL problem in this scenario, a discrete-time RL direct estimation is proposed for each UAV. The estimation of the relative position concerning its neighbors leverages distance and rate-of-change measurements, the angle of arrival, and the velocities of both itself and its neighbor. The displacements of neighbors during the intervals when distance and rate of change measurements are lost are also taken into account. Moreover, an RL fusion estimation method is devised for each UAV. This fusion estimation involves fusing the relative position estimates of the UAV concerning UGV0 and its neighbors. This approach enables UAVs without direct distance or angle measurements to locate UGV0 with the assistance of their neighbors. Subsequently, we apply the proposed RL fusion estimation algorithm to circumnavigation. In contrast to [25], our system integrates RL fusion estimations with a circumnavigation controller.

The primary contributions are outlined as follows:

(1) When the information flow graph between adjacent UAVs is unidirectional and time-varying, this paper proposes a distributed state observer with state switching to dynamically estimate the positions of UGV0. Only local measurements and limited information exchanges between nearby UAVs are used to estimate the relative coordinates of a group of UAVs concerning a single UGV0. The RL direct estimation error is bounded even in the presence of measurement noise.

(2) To enhance the robustness of RL, consensus-based RL fusion estimation is proposed. The boundedness of the RL fused estimation error is analyzed, and the experimental results demonstrate the effectiveness of the proposed method. The proposed RL method enables each UAV to continuously estimate its relative coordinates to UGV0, even in the absence of any relative measurements concerning UGV0 or its neighbors.

(3) The effectiveness of the entire system was demonstrated through numerical simulations of UAVs using RL fusion estimation for circumnavigation. The system integrates RL into circumnavigation control through UWB ranging and communication networks. The RL scheme proposed in this article applies not only to two-dimensional space but also to three-dimensional space.

The remainder of this article is structured as follows: Section 2 presents the problem formulation. Section 3 proposes an indirect RL method and consistency-based RL fusion estimation. Section 4 discusses the use of RL fusion estimation for circumnavigation. Section 5 conducts simulation experiments, and the article is summarized in Section 6.

2. Problem Formulation

This paper considers a network consisting of a single dynamic UGV and N UAVs, labeled 0 and $1,2,\dots ,N$, respectively. Define the position of each UAV as ${\psi }_i$. If $j\in {\zeta }_i$, then in the local coordinate system of UAVi, the relative position of j is ${\psi }_{ij}={\psi }_j-{\psi }_i$. Simultaneously, each UAV uses these relative estimates for circumnavigation. Let ${\zeta }_i$ represent the neighbors of UAV or UGV0. When $j\in {\zeta }_i$, UAVi can obtain the distance $d_{ij}$ of UAVj. Utilizing the obtained angle measurement information ${\alpha }_{ij}$, UAVi can deduce the relative position ${\psi }_{ij}$ under UAVj, as illustrated in Figure 1. Subsequently, leveraging the relative position estimates of its neighbors, each UAV generates corresponding circumnavigation control commands. During circumnavigation, the neighboring UAVj is effectively within the sensing radius; i.e., $d_{ij}$ is less than the sensing radius. Assuming a sampling period of T, and denoting the sampling time as k, for simplicity, k is used to represent $kT$.

Assuming UAVs follow a standard particle model with UAV speeds denoted as $v_i$, the relationship between relative speed and position is given by ${\psi }_{ij}(k+1)={\psi }_{ij}\left(k\right)+T{v}_{ij}\left(k\right)$, where $v_{ij}\left(k\right)=v_j\left(k\right)-v_i\left(k\right)$ represents the relative speed of UAVj in the i coordinate system. The angle measurement of neighbor j is denoted as ${\alpha }_{ij}\left(k\right)$, and this neighbor can be the target UGV0. It is assumed that each UAVi has access to its own speed; distance, represented by $v_{ij}\left(k\right), d_{ij}\left(k\right)$; and angle measurement ${\alpha }_{ij}\left(k\right)$ in its own inertial frame. The reference frame is denoted as ${\sum }_i,i=\{1,2,\dots ,N\}$ and is the same as ${\sum }_0$. This can be achieved if the UAV is equipped with a compass. Furthermore, assume that each UAVi is equipped with airborne sensors, allowing it to obtain the distance measurement ${\dot{\alpha }}_{ij}\left(k\right)$ and change rate ${\dot{d}}_{ij}\left(k\right)$ of its neighbor UAVj, or the distance measurement ${\dot{\alpha }}_{i0}\left(k\right)$ and change rate ${\dot{d}}_{i0}\left(k\right)$ of UGV0. As shown in Figure 1, $d_{ij}\left(k\right)={\psi }_{ij}\left(k\right)$ and ${\dot{\psi }}_{ij}\left(k\right)=v_{ij}\left(k\right)$ can be obtained. The goal is to develop an estimator so that each UAV can estimate its relative coordinate ${\psi }_{i0}\left(k\right)$ in UGV0’s frame ${\sum }_0$. With these RL estimates and measurements of distance and orientation between UAVs, the next goal is to integrate RL into circumnavigation control. Next, let us introduce graph theory.

If each UAV is viewed as a node, their interrelationships can be represented by a directed graph denoted as $\Gamma =(u,\mathcal{E})$, where $u=\{1,2,\dots ,N\}$ is the set of all nodes and u corresponds to the set of N UAVs. If $j\in {\zeta }_i$, then there is a corresponding arc $(i,j)\in \mathcal{E}$ in the directed graph, and UAVi can measure the distance, angle, and its corresponding rate of change. To study RL problems (e.g., estimating the position relative to UAVj), another weighted directed graph $\mathcal{\ell }=(u_j,{\mathcal{E}}_j,\mathrm{A})$ is also considered. Here, $u_j=\{1,2,\dots ,N\}$ is the set of all nodes, ${\mathcal{E}}_j\subseteq u_j\times u_j$ is the set of all arcs in the graph, $\mathrm{A}=\left[m_{ij}\right]\in {\mathbb{R}}^{N\times N}$ is the weighted adjacency matrix, and each element of the matrix is positive. Given $i\in u_j,m_{ii}=0$, if there is an arc $(i,j)$ in graph ℓ, then $m_{ij}>0$; otherwise, $m_{ij}=0$. It is worth noting that ℓ may be time-varying due to possible interruptions in the measurement. Assume that the set $\Gamma$ comprises all ordinary UAVs, referred to as the original set. UGV0 is introduced as the source point to form a new set $\overline{\Gamma }=(\overline{u},\overline{\mathcal{E}})$, known as the expanded set. Here, $\overline{u}=\left\{0\right\}\cup u, \overline{\mathcal{E}}={\mathcal{E}}_0\cup \mathcal{E}$, where ${\mathcal{E}}_0$ denotes the edge set comprising UGV0 and its surrounding neighbor UAVs. $(i,j)\in \overline{\mathcal{E}}$ signifies that UAVi and UAVj can exchange speed and data packets.

Define the Laplacian matrix of the weighted directed graph ℓ as $L_{\mathcal{\ell }}$, and the diagonal matrix $\mathrm{P}=\mathrm{diag}\{p_1,p_2,\dots ,p_N\}\in {\mathbb{R}}^{N\times N}$ as the degree matrix of ℓ, where the diagonal element is $p_i={\sum }_{j\in {\zeta }_i}{m}_{ij},i=\{1,2,\dots ,N\}$. To investigate the RL problem, a system comprising N UAVs and UGV0 is associated with another graph. The graph ℓ composed of N UAVs is a subgraph of $\overline{\Gamma }$. Let ${\overline{\zeta }}_i$ represent the set of neighbor nodes of node i in $\overline{\Gamma }$, which may include UGV0. If UAVi can obtain direct observation of the UGV0 distance $d_{i0}\left(k\right)$ or angle measurement ${\alpha }_{i0}\left(k\right)$, then $0\in \overline{{\zeta }_i}$; otherwise, $0\notin \overline{{\zeta }_i}$. Define a diagonal matrix $\beta \in {\mathbb{R}}^{N\times N}$ as the target adjacency matrix associated with $\overline{\Gamma }$, with diagonal element ${\mathrm{s}}_i, i=\{1,2,\dots ,N\}$. If UAVj is the neighbor of UAVi, and UAVi can directly obtain the distance and angle measurement of UAVj, then $s_i>0$; otherwise, $s_i=0$. For $\overline{\Gamma }$, if there exists a pathway from UGV0 to UAVi, we consider UGV0 to be jointly reachable.

3. Cooperative RL Algorithm

In this section, we propose a distributed RL algorithm based on mixed measurements. It addresses the challenge of estimating the relative coordinates of a UAV in the local frame of its neighbors. Through this approach, if UGV0 is a neighbor of the UAV, the UAV can acquire relative measurements of UGV0. Subsequently, the UAV can directly estimate its coordinates in the local frame of UGV0, termed a direct estimate. Conversely, if UGV0 is not a neighbor of the UAV, the UAV cannot utilize the relative measurements between UGV0 and itself to estimate its coordinates concerning UGV0. In this scenario, if the UAV can access the relative coordinates of its neighbors and the neighbors know the relative coordinates of UGV0, the UAV can deduce the relative coordinates of UGV0 through its neighbors. Nevertheless, multiple neighbors could potentially aid the UAV in establishing the relative coordinates of UGV0 through this method. To avoid dependence on unique neighbors, it is essential to combine multiple estimates. Furthermore, even with the availability of direct estimation, combining indirect estimation can enhance accuracy and expedite convergence. Hence, a consensus-based fusion estimation method is devised for each UAV in the second part of this section to fuse both direct estimates and all accessible indirect estimates.

3.1. RL Direct Estimation Relies on Persistent Excitation

As we all know, UAVs can experience measurement and communication losses due to harsh environments or sensor failures. In this subsection, we assume that UAVi ($i=1,2,\dots ,N$) can communicate with UGV0 and has access to distance measurement $d_{i0}\left(k\right)$ and the rate of change ${\dot{d}}_{i0}\left(k\right)$ in certain time intervals, in addition to angle measurement ${\alpha }_{i0}\left(k\right)$ and ${\dot{\alpha }}_{i0}\left(k\right)$. A direct estimator is designed to estimate the relative coordinates ${\psi }_{ij}\left(k\right)$ of UAVi in the local frame of UAVj.

Due to unreliability in local relative measurements, assume that UAVi obtains measurements relative to UAVj, $d_{ij}\left(k\right)$ and ${\dot{d}}_{ij}\left(k\right)$, or angle measurements ${\alpha }_{ij}\left(k\right)$ and ${\dot{\alpha }}_{ij}\left(k\right)$ at time $k\in [k_0,k_1)\cup [k_2,k_3)\cup \dots$, with a measurement break at $k\in [k_0,k_1)\cup [k_2,k_3)\cup \dots$. An indicator function, denoted as ${\sigma }_{ij}\left(k\right)$, is defined to represent the status. When ${\sigma }_{ij}\left(k\right)=1$, UAVi can acquire local relative measurements $d_{ij}\left(k\right)$ and ${\dot{d}}_{ij}\left(k\right)$ or angle measurement ${\alpha }_{ij}\left(k\right)$ and ${\dot{\alpha }}_{ij}\left(k\right)$; otherwise, ${\sigma }_{ij}\left(k\right)=0$. As illustrated in Figure 2, ${\sigma }_{ij}\left(k\right)=\left\{\begin{array}{c}1,k\in [k_{2t},k_{2t+1}),\\ 0,k\in [k_{2t+1},k_{2t+2}),& t=0,1,\dots \end{array}\right.$. Taking the derivatives on both sides of $d_{ij}^2\left(k\right)={∥{\psi }_{ij}\left(k\right)∥}^2,$ we can obtain $d_{ij}\left(k\right){\dot{d}}_{ij}\left(k\right)=v_{ij}^{\mathrm{T}}\left(k\right){\psi }_{ij}\left(k\right)$. Two unit vectors are constructed from the angle measurement information ${\alpha }_{ij}\left(k\right)$: the unit vector ${\Phi }_{ij}\left(k\right)={\left[\begin{array}{cc}cos{\alpha }_{ij}\left(k\right)& sin{\alpha }_{ij}\left(k\right)\end{array}\right]}^{\mathrm{T}}$ pointing from UAVi to UAVj, and the vector ${\Psi }_{ij}\left(k\right)={\left[\begin{array}{cc}-sin{\alpha }_{ij}\left(k\right)& cos{\alpha }_{ij}\left(k\right)\end{array}\right]}^{\mathrm{T}}$ obtained by rotating it 90° counterclockwise. Because vectors ${\Phi }_{ij}\left(k\right)$ and ${\psi }_{ij}\left(k\right)$ have the same direction, and ${\Phi }_{ij}\left(k\right)$ and ${\Psi }_{ij}\left(k\right)$ are perpendicular to each other, the constraint equation ${\Psi }_{ij}{\left(k\right)}^{\mathrm{T}}{\psi }_{ij}\left(k\right)=0$ can be obtained. When ${\sigma }_{ij}\left(k\right)=1$, UAVi can obtain the estimation algorithm of ${\widehat{\psi }}_{ij}\left(k\right)$ through information from measurements and communication. Estimates can become inaccurate due to UAV motion, as the sensors are subject to interference. Assume that the estimated value ${\widehat{\psi }}_{ij}\left(k\right)$ before the interruption is initially employed. Once communication and measurement are restored, that is, when ${\sigma }_{ij}\left(k\right)=1$, UAVi will correct the deviation caused by the estimated value in the case of ${\sigma }_{ij}\left(k\right)=0$. Considering sensor noise, the RL direct estimation of UAVi in the local coordinate system of UAVj at time k is estimated as follows:

${\widehat{\psi }}_{ij}(k+1)={\widehat{\psi }}_{ij}\left(k\right)+T(v_{ij}\left(k\right)+\tau \left(k\right))\left[K{\widehat{{\rm Y}}}_{ij}\left(k\right)-{(v_{ij}\left(k\right)+\tau \left(k\right))}^{\mathrm{T}}{\widehat{\psi }}_{ij}\left(k\right)\right]$

(1)$\left\{\begin{array}{cc}{\widehat{{\rm Y}}}_{ij}\left(k\right)=\left(d_{ij}\left(k\right)+\sigma \left(k\right)\right)({\dot{d}}_{ij}\left(k\right)+\dot{\sigma }\left(k\right))& \delta =1\\ {\widehat{{\rm Y}}}_{ij}\left(k\right)=\frac{{\left(v_{ij}\left(k\right)+\tau \left(k\right)\right)}^{2}cos\left({\alpha }_{ij}\left(k\right)+\Delta \left(k\right)\right)sin\left({\alpha }_{ij}\left(k\right)+\Delta \left(k\right)\right)}{({\dot{\alpha }}_{ij}\left(k\right)+\dot{\Delta }\left(k\right))+{\dot{\theta }}_{ij}}·& \delta =0\end{array}\right.k\in [k_{2t},k_{2t+1})$

${\widehat{\psi }}_{ij}(k+1)={\widehat{\psi }}_{ij}\left(k\right)k\in [k_{2t+1},k_{2t+2}).$

where $\tau \left(k\right), \sigma \left(k\right), \dot{\sigma }\left(k\right), \Delta \left(k\right), \dot{\Delta }\left(k\right)$ represent the measurement noise of $v_{ij}\left(k\right), d_{ij}\left(k\right), {\dot{d}}_{ij}\left(k\right), {\alpha }_{ij}\left(k\right), {\dot{\alpha }}_{ij}\left(k\right)$ at time k, respectively. K is a tunable constant gain. ${\widehat{\psi }}_{ij}\left(k\right)$ represents the coordinate of UAVi in the local coordinate system ${\sum }_j^M$ of UAVj. When $\delta =1$, it indicates the distance measurement is normal, and when $\delta =0$, it indicates the angle measurement is normal. It will be demonstrated later that the RL direct estimation error is bounded in the presence of noise.

Let the estimation error of the above observer (1) be denoted as ${\tilde{\psi }}_{ij}\left(k\right)={\widehat{\psi }}_{ij}\left(k\right)-{\psi }_{ij}\left(k\right)$, and the dynamic equation of the estimation error is

(2)${\tilde{\psi }}_{ij}(k+1)=\left\{\begin{array}{cc}[I-T(v_{ij}\left(k\right)+\tau \left(k\right)){(v_{ij}\left(k\right)+\tau \left(k\right))}^{\mathrm{T}}]{\tilde{\psi }}_{ij}\left(k\right)\hfill & k\in [k_{2t},k_{2t+1})\hfill \\ {\tilde{\psi }}_{ij}\left(k\right)+T(v_{ij}\left(k\right)+\tau \left(k\right))\hfill & k\in [k_{2t+1},k_{2t+2})\hfill \end{array}\right..$

Let us begin by introducing the concept of persistent excitation [27] before moving on to the discussion of the convergence of error system (2). There exists a positive integer m, ${\alpha }_1,{\alpha }_2$ such that for any $k\in {\mathbb{Z}}^{+}$, there is

(3)${\alpha }_{1}I\le \sum _{f=k}^{k+m}\left(v_{ij}\left(k\right)+\tau \left(k\right)\right){\left(v_{ij}\left(k\right)+\tau \left(k\right)\right)}^{\mathrm{T}}T\le {\alpha }_{2}I,$

where T is the sampling period. Next, let us analyze the physical meaning of persistent excitation. Assuming that the speed of each UAV is continuously differentiable and bounded, the upper bound obviously holds. Therefore, the main focus is on (3), the lower bound. Expanding Equation (3), we can obtain

$\begin{array}{cc}& \sum _{f=k}^{k+m}\left(v_{ij}\left(f\right)+\tau \left(f\right)\right){\left(v_{ij}\left(f\right)+\tau \left(f\right)\right)}^{\mathrm{T}}=\hfill \\ & \left[\begin{array}{cc}\sum _{f=k}^{k+m}{\left(v_{ijx}\left(f\right)+\tau \left(f\right)\right)}^2& \sum _{f=k}^{k+m}\left(v_{ijx}\left(f\right)+\tau \left(f\right)\right)\left(v_{ijy}\left(f\right)+\tau \left(f\right)\right)\\ \sum _{f=k}^{k+m}\left(v_{ijx}\left(f\right)+\tau \left(f\right)\right)\left(v_{ijy}\left(f\right)+\tau \left(f\right)\right)& \sum _{f=k}^{k+m}{\left(v_{ijy}\left(k\right)+\tau \left(f\right)\right)}^2\end{array}\right]\hfill \end{array}.$

The two components of $v_{ij}\left(k\right)$ are represented as $v_{ijx}\left(k\right)$ and $v_{ijy}\left(k\right)$, which are linearly independent of $f\in \{k,\dots ,k+m\}$. According to the inequality of Cauchy–Bunyakovsky, for any $k\ge 0$, the following formula

$\begin{array}{c}{\left({\displaystyle \sum _{f=k}^{k+m}}\left(v_{ijx}\left(f\right)+\tau \left(f\right)\right)\left(v_{ijy}\left(f\right)+\tau \left(f\right)\right)\right)}^2\le {\displaystyle \sum _{f=k}^{k+m}}{(v_{ijx}\left(f\right)+\tau \left(f\right))}^2{\displaystyle \sum _{f=k}^{k+m}}{(v_{ijy}\left(f\right)+\tau \left(f\right))}^2\end{array}$

holds. Both sides of the above inequality are equal if and only if $v_{ijx}\left(f\right)$ and $v_{ijy}\left(f\right)$ are linearly related.

Theorem 1.

According to Equation (1), when persistent excitation (3) holds for UAVi and if the sampling period T satisfies the given criterion

(4) $0<T<\frac{2(\overline{v}+\overline{\tau })}{K^2}$

then the estimation error of system (2) is bounded. The upper bound $v_i$ of the error is determined by a specific value $\overline{v}$, ensuring that the error satisfies $0<\parallel v_{ij}\parallel \le 2\overline{v}$, the specified condition. To proceed, suppose there exists a constant $\overline{\tau }>0$ such that $\parallel \tau \left(k\right)\parallel \le \overline{\tau }.$

Proof. 

Firstly, consider the system related to (2):

(5)$\epsilon (k+1)=[I-TK{\sigma }_{ij}\left(k\right)(v_{ij}\left(k\right)+\tau \left(k\right)){(v_{ij}\left(k\right)+\tau \left(k\right))}^{\mathrm{T}}]{\tilde{\psi }}_{ij}\left(k\right).$

Construct the Lyapunov function $\mathrm{V}\left({\tilde{\psi }}_{ij}\left(k\right)\right)={\tilde{\psi }}_{ij}\left(k\right){\tilde{\psi }}_{ij}^{\mathrm{T}}\left(k\right)$, and the difference in the Lyapunov function within m time steps can be expressed as

(6)$\begin{array}{cc}\hfill \Delta {\mathrm{V}}_m\left(k\right)& =\mathrm{V}\left({\tilde{\psi }}_{ij}(k+m)\right)-\mathrm{V}\left({\tilde{\psi }}_{ij}\left(k\right)\right)\hfill \\ \\ & =\sum _{f=k}^{k+m}\mathrm{C}\left(k\right){\left({(v_{ij}\left(k\right)+\tau \left(k\right))}^{\mathrm{T}}{\tilde{\psi }}_{ij}\left(k\right)\right)}^2.\hfill \end{array}$

In addition, applying the induction method to system (5), the f-step RL direct estimation error range $\parallel {\tilde{\psi }}_{ij}\left(f\right)\parallel \le \parallel {\tilde{\psi }}_{ij}\left(k\right)\parallel +\tau \left(f\right)T(1+K(2\overline{v}+\overline{\tau }))(f-k)$ is obtained. Now, (6) can be expressed as

(7)$\begin{array}{cc}\hfill \Delta {\mathrm{V}}_m\left(k\right)& \le \sum _{f=k}^{k+m}C(k)\left(v_{ij}\left(f\right)+\tau \left(f\right)\right){\left(v_{ij}\left(f\right)+\tau \left(f\right)\right)}^{\mathrm{T}}\hfill \\ & +m\tau \left(f\right)T\left(1+K(2\overline{v}+\tau \left(f\right))\right)\left(2\parallel {\tilde{\psi }}_{ij}\left(k\right)\parallel +m\tau \left(f\right)T\left(1+K(2\overline{v}+\overline{\tau })\right)\right).\hfill \end{array}$

Observing (2) and (5), we can see that if ${\tilde{\psi }}_{ij}\left(k\right)=\epsilon \left(k\right)$, then for any $k\in [k_{2t},k_{2t+1}),$ ${\tilde{\psi }}_{ij}\left(k_{2t}\right)=\epsilon \left(k_{2t}\right)$ is satisfied. In the $k\in [k_{2t},k_{2t+1})$ interval, $\epsilon \left(k\right)$ remains unchanged, while ${\tilde{\psi }}_{ij}\left(k\right)$ is time-varying. However, it can be seen from (2) that when $k=k_{2t+2}$, ${\tilde{\psi }}_{ij}\left(k_{2t+2}\right)={\tilde{\psi }}_{ij}\left(k_{2t+1}\right)$. To sum up, it can be deduced that as long as ${\tilde{\psi }}_{ij}\left(0\right)=\epsilon \left(0\right)$, then for any $k\in [k_{2t}$, $k_{2t+1}),{\tilde{\psi }}_{ij}\left(k\right)=\epsilon \left(k\right)$.

In addition, according to the definition of ${\sigma }_{ij}\left(k\right)$, when $k\in [k_{2t}$, $k_{2t+1}), {\sigma }_{ij}\left(k\right){\tilde{\psi }}_{ij}\left(k\right)={\tilde{\psi }}_{ij}\left(k\right)$. When $k\in [k_{2t+1}$, $k_{2t+2}), {\sigma }_{ij}\left(k\right){\tilde{\psi }}_{ij}\left(k\right)=0$. It can be seen from (3) that the RL direct estimation error of (5) is bounded in the case of noise interference. That is, there exists $\alpha >0,$ $C\left(k\right)>0$ that satisfies for any $k\ge 0, \parallel \epsilon \left(k\right)\parallel \le C\left(k\right)e^{-\alpha k}$, if and only if there exists ${\alpha }_1>0,$ ${\alpha }_2>0, T>0$, and (3) holds for any $k\ge 0$. To now, the RL direct estimation error bound can be expressed as follows. Combining (4) and (7) leads to $\Delta {\mathrm{V}}_m\left(k\right)\le {\sigma }_{ij}\left(k\right){\tilde{\psi }}_{ij}\left(k\right)+{\alpha }_{1}C\left(k\right)e^{-\alpha k}+{\alpha }_2$, where ${\sigma }_{ij}\left(k\right)>0, \alpha >0, C\left(k\right)>0$. Therefore, when $k\in [k_{2t+1},k_{2t+2}), {\tilde{\psi }}_{ij}\left(k\right)\le {\alpha }_{1}C\left(k\right)e^{-\alpha k}+{\alpha }_2$. It can be proven that when $k\in [k_{2t},k_{2t+1}),{\tilde{\psi }}_{ij}\left(k\right)\le (1+{\alpha }_1)C\left(k\right)e^{-\alpha k}+{\alpha }_2$; the proof of Theorem 1 is complete. □

3.2. Fusion-Based RL Estimation

In the previous Section 3.1, we assumed that local relative measurements ($d_{ij}\left(k\right)$, rate of change ${\dot{d}}_{ij}\left(k\right)$, and ${\alpha }_{ij}\left(k\right),{\dot{\alpha }}_{ij}\left(k\right)$) are unreliable, with measurement values available only at certain moments. An estimator was then designed for the UAVi to position UGV0. If UGV0 is a neighbor of UAVi, then UAVi can obtain its direct estimate ${\widehat{\psi }}_{i0}\left(k\right)$ in the local coordinate system of UGV0. However, due to harsh environments or temporary sensor failures, a UAV may lose its local relative measurements. Worse still, some UAVs may not always have relative measurements with respect to UGV0 because UGV0 is outside their sensing range. In this case, cooperation among neighbor UAVs is needed to help UGV0 localize. In this subsection, an RL indirect estimation method is developed for each UAVi to estimate its relative coordinates with respect to UAVj, even when it may lack direct measurements for UGV0 or experience measurement failures over time.

If UAVi can estimate its coordinate ${\widehat{x}}_{ir}$ relative to neighboring UAVr, UAVr can also share its estimate $Z_r$ with UAVi. As a result, UAVi can indirectly obtain its estimate relative to UAVj through UAVr, as illustrated in Figure 3. The formula is as follows:

(8)${\widehat{x}}_{ij}\left(k\right)={\widehat{x}}_{ir}\left(k\right)+Z_r\left(k\right).$

Expanding on both direct and indirect RL algorithms, we also explore RL fusion estimation between UAVs to achieve target positioning. RL fusion estimation serves a dual purpose. It aids UAVs lacking direct target measurements, enhancing their ability to locate targets. Simultaneously, it bolsters the robustness of relying on RL direct estimation. Utilizing information gathered by UAVs and integrating both direct (1) and indirect (8) RL estimators, an RL fusion estimation method is proposed. Each UAVi employs the following estimation method to update its final estimate, as expressed in the formula:

(9)$\begin{array}{cc}\hfill Z_i(k+1)& =Z_i\left(k\right)+T\left(v_{ij}\left(k\right)+\tau \left(k\right)\right)\hfill \\ & +\sum _{j\in {\zeta }_i\left(k\right)}{\beta }_{ij}\left(k\right)[{\widehat{x}}_{i0}\left(k\right)-Z_i\left(k\right)].\hfill \end{array}$

Consequently, each UAVi updates its fused estimate using (9), irrespective of its ability to directly obtain relative measurements about UGV0.

Theorem 2.

If the conditions of Theorem 1 are met; if it is assumed that, for every node pair $(i,j)$ in $\Gamma \left(k\right)$ that appears infinitely many times, the persistent excitation (3) is satisfied; if each node in $\Gamma \left(k\right)$ is uniformly jointly reachable from UGV0; and if the fusion weight satisfies $0<{\sum }_{j\in {\zeta }_i\left(t\right)}{\beta }_{ij}\left(k\right)<1$, then the fusion estimate $Z_i\left(k\right)$ of each UAVi asymptotically converges to its true coordinates. In the presence of measurement noise, the RL fused estimation error is bounded.

Definition 1

(Stability of a discrete system input state [28]). Consider a nonlinear system given by equation $x(k+1)=f\left(x\right(k),u(k\left)\right)$. If there exists a function class $\mathcal{K}\mathcal{L}$: $\beta :{\mathbb{R}}_{\ge 0}\times {\mathbb{R}}_{\ge 0}\to {\mathbb{R}}_{\ge 0}$ and a class$\mathcal{K}$ function γ such that for any control input $u\in l_{\infty }^m$ and any $\xi \in {\mathbb{R}}^n$, the following formula $\left|x\right(k,\xi ,u\left)\right|\le \beta \left(\right|\xi |,k)+\gamma (\parallel u\parallel )$ holds for any positive integer k, then the system in Definition 1 is said to be input-to-state stable (ISS).

Lemma 1

([28]). If A is a Schur matrix, then the discrete system $x(k+1)=Ax\left(k\right)+Bu\left(k\right)$ is ISS.

Lemma 2

([29]). Matrix ${\mathcal{L}}_j+{\mathcal{B}}_j$ is positively stable, indicated by eigenvalues with positive real parts, if and only if UAVj is jointly reachable in Γ.

Proof. 

For any given $i=\{1,2,\dots ,N\}$, let $y_i\left(k\right)=Z_i\left(k\right)-{\psi }_{ij}\left(k\right)$. Then, (9) can be rewritten as

(10)$y_i(k+1)=y_i\left(k\right)+\sum _{j\in {\zeta }_i\left(k\right)}{\beta }_{ij}\left(k\right)\left[y_j\left(k\right)-y_i\left(k\right)\right]+u_{ij}\left(k\right),$

where the $u_{ij}\left(k\right)={\sum }_{j\in {\zeta }_i\left(t\right)}{\beta }_{ij}\left(k\right){\tilde{\psi }}_{ij}\left(k\right)+T\tau \left(k\right)$ part serves as the input signal. Aggregating all the equations in $i=\{0,1,2,\dots ,N\}$, (10) can be expressed in matrix form as

(11)$y_{ij}(k+1)=\left(\left(I-{\mathcal{L}}_j-{\mathcal{B}}_j\right)\otimes I_p\right)y_{ij}\left(k\right)+u_{ij}\left(k\right),$

where $y_{ij}\left(k\right)={[y_0^{\mathrm{T}}\left(k\right)y_1^{\mathrm{T}}\left(k\right)\dots y_N^{\mathrm{T}}\left(k\right)]}^{\mathrm{T}}, u_{ij}\left(k\right)={[u_0^{\mathrm{T}}\left(k\right)u_1^{\mathrm{T}}\left(k\right)\dots u_N^{\mathrm{T}}\left(k\right)]}^{\mathrm{T}}.$p is two or three, determined by the dimensions of the environmental space. $\mathcal{L}j$ represents the weighted Laplacian matrix of the graph $\Gamma$, and $\mathcal{B}j$ is the adjacency matrix of j associated with $\Gamma$.

Firstly, considering the unforced system in

(12)$y_{ij}(k+1)=\left(\left(I-{\mathcal{L}}_j-{\mathcal{B}}_j\right)\otimes I_p\right)y_{ij}\left(k\right),$

it can be verified that all eigenvalues of ${\mathcal{L}}_j+{\mathcal{B}}_j$ strictly lie within the unit circle centered at the origin. Therefore, ${\mathcal{L}}_j+{\mathcal{B}}_j$ is a Schur matrix. As UAVi and UGV0 are uniformly and jointly reachable, according to Lemmas 1 and 2, when $k\to \infty$, all components of the solution of (12) uniformly converge exponentially to a certain common value. It can be concluded that when $k\to \infty ,y_i\left(k\right)\to c$ (c is a constant), indicating the exponential stability of (12).

Consider (11) next. According to Definition 1, for any $k\ge 0,\parallel y_{ij}\left(k\right)(y_{ij}\left(0\right),u_{ij}\left(k\right))\parallel \le \beta (\parallel y_{ij}\left(0\right)\parallel ,k)+\gamma (\parallel u_{ij}\left(k\right)\parallel )$. According to Theorem 1, for any positive constant ${\beta }_{ij}\left(k\right),$ $\parallel u_{ij}\left(k\right)\parallel \le {\sum }_{j\in {\zeta }_i\left(k\right)}{\beta }_{ij}\left(k\right)+T\tau \left(k\right)$. Since $\gamma (·)$ is a function of class$\mathcal{K}$, it follows that when $k\to \infty ,\gamma (\parallel u_{ij}\left(k\right)\parallel )\to c$. Because $y_{ij}\left(0\right)$ is bounded and $\beta (·,·)$ is a function of class$\mathcal{K}\mathcal{L}$, when $k\to \infty ,\beta (\parallel y_{ij}\left(0\right)\parallel ,k)\to 0$. Therefore, $\parallel y_{ij}\left(k\right)(y_{ij}\left(0\right),u_{ij}\left(k\right))\parallel \to c$ when $k\to \infty$. The proof of Theorem 2 is complete. □

4. Integrated Solutions for RL and Circumnavigation

In this section, we propose an integrated solution combining RL and circumnavigation to facilitate the rotation of multi-UAVs around UGV0 while maintaining a circular formation. This capability is particularly valuable in practical scenarios like surrounding and entrapping a hostile target. As depicted in Figure 4, it relies on distance $d_i\left(k\right)=\parallel Z_i\left(k\right)-p^{\ast }\parallel$ and angle measurements ${\alpha }_{ij}\left(k\right)$. An adaptive estimator will be formulated to attain relative positioning with the assistance of a specifically designed bounded input $u_i\left(k\right)$. Let $q_i\left(k\right)=Z_i\left(k\right)-p^{\ast }$ denote the relative position to the UGV0. ${\widehat{q}}_i\left(k\right)$ is expressed as an estimate of $q_i\left(k\right)$. Note that $u_i\left(k\right)$ should also satisfy $\parallel u_i\left(k\right)\parallel \le U$ to address the circumnavigation problem (13), where U is the maximum velocity.

Given UGV0 at any location $p^{\ast }\left(k\right)$, the objective is to enable the UAV to orbit around UGV0. The formula is as follows:

(13)$\underset{k\to \infty }{lim}{Z}_i\left(k\right)=p^{\ast }\left(k\right),$

where $Z_i\left(k\right)$ represents the position of UAVi at time step k. For trajectory planning purposes, consider a discrete-time integrator model with bounded velocity: $Z_i(k+1)=Z_i\left(k\right)+T{u}_i\left(k\right),$ where T is the sampling period.

Based on this, a circumnavigation control law involving RL fusion estimation is proposed:

(14)$u_{i,k}=\left\{\begin{array}{cc}{u}_{0,k}-\left[\left({\widehat{d}}_i^2\left(k\right)-d_i^2\left(k\right)\right)I-\beta E\right]{\widehat{q}}_i\left(k\right)\hfill & \delta =1\hfill \\ \left({\widehat{d}}_i\left(k\right)-d_i\left(k\right)\right){\Phi }_{ij}\left(k\right)+\alpha {\Psi }_{ij}\left(k\right)\hfill & \delta =0\hfill \end{array}\right.,$

where $\beta$ is any non-zero real scalar and E is the rotation matrix. ${\widehat{d}}_i\left(k\right)$ represents the estimated value of $d_i\left(k\right)$. Additionally, a constant positive scalar $\alpha$ is introduced. By integrating the previously developed RL algorithm (9) with the circumnavigation control algorithm (14), we propose an integrated RL fusion estimation and circumnavigation control algorithm.

5. Simulation

In this part, we consider a cooperative RL fusion estimation involving five mobile UAVs and a slow UGV0 and subsequently apply the estimated values to circumnavigation control. The simulation workflow diagram is illustrated in Figure 5.

5.1. Results of RL Fusion Estimation Simulations

In this section, a simulation of the RL of five mobile UAVs and a slowly moving UGV0 in two-dimensional space is conducted. The information flow graph $\Gamma \left(k\right)$ between the five UAVs and UGV0 is allowed to switch between two graphs periodically, $\Gamma \left(1\right)$ and $\Gamma \left(2\right)$, as illustrated in Figure 6, or switch between three graphs randomly, $\Gamma \left(1\right)$, $\Gamma \left(2\right)$, and $\Gamma \left(3\right)$, as illustrated in Figure 7. It can be confirmed that in the information flow graph, UGV0 is jointly reachable. UAV 1 and 3 periodically acquire direct measurements of UGV0, while UAV 2, 4, and 5 do not; they rely solely on indirect estimates through their neighbors. To validate the proposed estimation scheme, the target UGV0 is positioned at the origin, and the five UAVs are subjected to the following dynamic control:

$\begin{array}{cc}\hfill {\dot{x}}_1\left(k\right)& =\left[\begin{array}{c}-sink\\ cosk\end{array}\right], {\dot{x}}_2\left(k\right)=\left[\begin{array}{c}0\\ {\displaystyle \frac{1}{4}}\end{array}\right]\hfill \\ \hfill {\dot{x}}_3\left(k\right)& =\left[\begin{array}{c}{\displaystyle \frac{1}{6}}\\ 0\end{array}\right], {\dot{x}}_4\left(k\right)=\left[\begin{array}{c}0\\ {\displaystyle \frac{1}{8}}\end{array}\right], {\dot{x}}_5\left(k\right)=\left[\begin{array}{c}-2sink\\ 2cosk\end{array}\right].\hfill \end{array}$

Furthermore, the initial positions of the five UAVs are labeled as (−1, −2), (6, 8), (8, 9), (11, −2), and (1, 9), respectively. The initial position of UGV0 is (0.3, 0).

To validate the feasibility of the proposed RL fusion estimation scheme, we designated UGV0 as the relative target for estimation. In the first simulation, the trajectory diagram is depicted in Figure 8. In the second simulation, we let all of the UAVs move randomly within the range [−50, 50]. Communication protocols were used to ensure that UAVs were aware of each other’s positions and velocities, allowing them to maintain safe distances. Upon receiving position and velocity updates from neighboring UAVs, each UAV processes this information to determine the relative positions and velocities of nearby UAVs. By comparing this information with its trajectory, a UAV can assess the risk of potential collisions and take appropriate action to avoid them. With unlimited energy, UAVs can perform their tasks without the need to consider energy efficiency or battery life. They can fly for extended periods, cover long distances, and execute complex maneuvers without the risk of power depletion.

For the first simulation, it can be verified that, in the case of a time-varying information flow graph, each UAV is uniformly jointly reachable to UGV0. Additionally, it can be confirmed that the persistent excitation (3) is satisfied. For the second simulation, it is challenging to strictly confirm the condition that each UAV is consistently jointly reachable to UGV0 and that the persistent excitation (3) is satisfied. The estimator described in (9) is employed to estimate the coordinates of each UGV0. The direct and indirect estimates are amalgamated to derive the ultimate fusion estimate $Z_i\left(k\right)$. According to Theorem 2, each UAV has its estimate $Z_i\left(k\right)$ that converges to the true coordinates ${\psi }_{i0}\left(k\right)$ of UGV0. Here, let $Z_i\left(k\right)$ represent the final fusion estimate. For simplicity, the corresponding real RL is $x_{i0}\left(k\right)$. Therefore, the evolution of estimation errors is represented by $\parallel er{r}_{i0}\left(k\right)\parallel =\parallel Z_i\left(k\right)-x_{i0}\left(k\right)\parallel$. In the first and second simulations, the evolution of estimation errors for periodically and randomly switching simulations is depicted in Figure 9b,e and Figure 10a,b, respectively. It can be observed that even in the presence of noise, the fusion estimation error remains bounded uniformly. The RL fusion estimation errors are bounded with 0.6m in both cases. Our proposed method demonstrates robustness in simulation results, performing well in both prescribed trajectories and random scenarios.

In addition, in order to validate the effectiveness of the proposed RL fusion estimation scheme, we conducted comparative experiments. Figure 9a–c depict the simulation results of periodic switching, while Figure 9d–f show the results of random switching. Among them, (a), (d), and (c), (f) represent the corresponding error estimation curves under the methods in paper [30] and paper [31], respectively. (b) and (e) depict the error curves corresponding to the method proposed in this article. In the simulation results of the first simulation, a comparison of the graphs in (a) and (b) reveals that the error of the algorithm proposed in this paper is significantly smaller than the error of the method in paper [30]. Simultaneously, the algorithm proposed in this paper demonstrates faster convergence compared to that in paper [30]. Further, in the comparison between graphs (b) and (c), although the error in paper [31] is lower, its convergence speed is significantly slower.

A comparison of the results in Figure 9d,e, illustrating the first simulation’s outcomes, reveals that the algorithm proposed in this study exhibits a substantially lower error than the method presented in paper [30]. Additionally, our approach achieves faster convergence compared to paper [30]. While the error in [31] is lower, the rate of convergence is relatively slower, as evident in graphs (e) and (f). This demonstrates the robustness of our proposed cooperative RL method in various scenarios. Despite the errors in Figure 9a being one or two orders of magnitude larger than those in Figure 9b, they remain unacceptable as they are on the meter scale. The errors in (d) and (e) are also similar. In the presence of noise and data loss, the fusion estimator mitigates the impact of these factors to some extent. The additional information in indirect estimation contributes to expediting the convergence of RL fusion estimation. The RL fusion estimation scheme errors between Figure 9b,e in the first simulation and Figure 10a,b in the second simulation were compared. While the convergence speed of the second simulation is slower, it still satisfies the convergence criteria. It can be seen that based on the proposed method, RL fusion estimation scheme errors are bounded by 0.6 m. However, the errors using the method in [30,31] can be up to 1.5 m, which are much larger than those of the proposed method.

5.2. Integration of RL Fusion Estimation and Circumnavigation Control [25]

In this section, we will validate the integrated RL and circumnavigation scheme proposed in Section 4 through numerical simulations to confirm its robustness. Consider a five-UAV team, where each UAV is able to measure the distance or angle to UGV0. The control goal is to enable five UAVs to hover around UGV0. In the third simulation, we verified the scheme proposed when UGV0 slowly drifts with an angular velocity equal to 0.005 rad/s. The UAV maintains its distance in a neighborhood of the desired range from UGV0. It is worth noting that the speed of UGV0 is consistently much lower than that of the other UAVs. In this scenario, we assume that $p^{\ast }\left(0\right)=$ [−20, −20] is the initial position of UGV0. The initial positions of the UAVs are set as [15, −8], [−20, 10], [−12, −6], [16, −13] and [18, 10], respectively. We can observe from Figure 11 that the UAV is capable of orbiting around UGV0 very well. Under these conditions, positive constants $\beta =5$ and $\alpha =4$ are chosen to satisfy the given criteria. As depicted in Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16, all trajectories converge to zero, validating our theoretical analysis.

To demonstrate the effectiveness of the integrated RL and circumnavigation solution, tests were conducted with five UAVs. The graph in Figure 17 illustrates the reduction in distance between UAVs and UGV0. It is evident that the plane distance consistently decreases rapidly until the circumnavigation radius of 0.7 m is reached, greater than the minimum spacing of 0.2 m. In summary, the experimental results demonstrate the superior performance of the integrated RL and circumnavigation solution. It is anticipated that similar integration concepts can be further applied to more practical scenarios.

6. Conclusions

This paper proposes an RL fusion estimation and distance- or angle-based UAV circumnavigation control scheme that does not rely on infrastructure or the GPS. The proposed algorithm enables a UAV to locate UGV0 using only the distance or the azimuth angle, without any explicit distinction between the measured data. Integrated algorithms of RL estimators and circumnavigation controllers are explored. The concepts in this algorithm can also be extended to scenarios where the UAV’s motion is subject to non-holonomic constraints. A possible generalization is discussed in [32]. Finally, an integrated cooperative RL and circumnavigation control algorithm is proposed by combining the RL and circumnavigation control algorithms. Numerous simulation experiments have verified the effectiveness of the proposed algorithm.

Given the recent innovation and research outcomes in artificial intelligence (AI), we will explore opportunities to enable and improve functionalities in UAVs using AI techniques in the future. Both AI and UAV technology have become popular in recent times, along with research to bring the two fields together. Many problems inherent in UAVs today can be solved with the use of AI. Future research directions include addressing challenges in three-dimensional space and enhancing the algorithm’s convergence speed to reduce tracking and estimation errors within a limited timeframe. Additionally, considering a more general UAV model and accounting for the impact of noise are crucial aspects of future investigations. Another avenue for research involves exploring scenarios with UAV swarms and UGVs. Estimating the location of UGVs may be easier in the case of UAV swarms, as different UAVs can share their estimates of the UGV’s location. These shared estimates can expedite the estimation of UGV positions. Additionally, collision avoidance technology should be taken into account when deploying UAV swarms. Furthermore, it is essential to ensure that the distance between UAVs does not exceed their communication range, preventing the disruption of the connection link between them.

Author Contributions

Conceptualization, J.G. and K.H.; methodology, J.G. and K.H.; software, J.G; validation, J.G. and K.H.; formal analysis, M.G.; investigation, J.G.; resources, K.H.; writing—original draft preparation, J.G.; writing—review and editing, M.G. and K.H.; visualization, J.G.; supervision, K.H.; project administration, J.G. and K.H.; funding acquisition, M.G. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Acknowledgments

The authors would like to thank the editor and the anonymous reviewers for their careful reading and valuable suggestions that helped to improve the quality of this article.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures

Figure 1. Local measurements and relative positions.

Figure 2. Diagram illustrating the indicator function.

Figure 3. Obtain the indirect RL estimate [Forumla omitted. See PDF.] from UAVi to UAVj through UAVr.

Figure 4. An integrated RL and circumnavigation solution.

Figure 5. Workflow diagram for circumnavigation control.

Figure 6. The periodic switching graph [Forumla omitted. See PDF.] that switches between two different topologies [Forumla omitted. See PDF.] and [Forumla omitted. See PDF.].

Figure 7. The three graphs [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], and [Forumla omitted. See PDF.], among which [Forumla omitted. See PDF.] randomly switches.

Figure 8. Movement trajectories of the five UAVs and UGV0.

Figure 9. The first simulation. (a) Results of RL fusion estimation errors under periodic switching conditions using the method in [30]. (b) Results of RL fusion estimation errors under periodic switching conditions using the proposed method. (c) Results of RL fusion estimation errors under periodic switching conditions using the method in [31]. (d) Results of RL fusion estimation errors under random switching conditions using the method in [30]. (e) Results of RL fusion estimation errors under random switching conditions using the proposed method. (f) Results of RL fusion estimation errors under random switching conditions using the method in [31].

Figure 10. The second simulation. (a) Results of RL fusion estimation errors under periodic switching conditions using the proposed method. (b) Results of RL fusion estimation errors under random switching conditions using the proposed method.

Figure 11. Simulation of a slowly drifting UGV0. The green circle denotes the initial position of UGV0, while the other circles indicate the initial positions of the UAVs.

Figure 12. Trajectory diagram depicting the relative positions between UAV1 and UGV0.

Figure 13. Trajectory diagram depicting the relative positions between UAV2 and UGV0.

Figure 14. Trajectory diagram depicting the relative positions between UAV3 and UGV0.

Figure 15. Trajectory diagram depicting the relative positions between UAV4 and UGV0.

Figure 16. Trajectory diagram depicting the relative positions between UAV5 and UGV0.

Figure 17. The rapid and stable approach of UGV0.

References

1. Xu, L.; Cao, X.; Du, W.; Li, Y. Cooperative path planning optimization for multiple UAVs with communication constraints. Knowl.-Based Syst.; 2023; 260, 110164. [DOI: https://dx.doi.org/10.1016/j.knosys.2022.110164]

2. Dang, Y.; Benzaïd, C.; Yang, B.; Taleb, T.; Shen, Y. Deep-ensemble-learning-based GPS spoofing detection for cellular-connected UAVs. IEEE Internet Things J.; 2022; 9, pp. 25068-25085. [DOI: https://dx.doi.org/10.1109/JIOT.2022.3195320]

3. Zhang, G.; Hsu, L.T. Intelligent GNSS/INS integrated navigation system for a commercial UAV flight control system. Aerosp. Sci. Technol.; 2018; 80, pp. 368-380. [DOI: https://dx.doi.org/10.1016/j.ast.2018.07.026]

4. Basiri, A.; Mariani, V.; Silano, G.; Aatif, M.; Iannelli, L.; Glielmo, L. A survey on the application of path-planning algorithms for multi-rotor UAVs in precision agriculture. J. Navig.; 2022; 75, pp. 364-383. [DOI: https://dx.doi.org/10.1017/S0373463321000825]

5. Khan, S.I.; Qadir, Z.; Munawar, H.S.; Nayak, S.R.; Budati, A.K.; Verma, K.D.; Prakash, D. UAVs path planning architecture for effective medical emergency response in future networks. Phys. Commun.; 2021; 47, 101337. [DOI: https://dx.doi.org/10.1016/j.phycom.2021.101337]

6. Yang, K. Research on key technologies of UAV navigation and positioning system. Proceedings of the 2021 International Conference on Wireless Communications and Smart Grid (ICWCSG); Hangzhou, China, 13–15 August 2021; pp. 29-33.

7. Yao, W.; Lu, H.; Zeng, Z.; Xiao, J.; Zheng, Z. Distributed static and dynamic circumnavigation control with arbitrary spacings for a heterogeneous multi-robot system. J. Intell. Robot. Syst.; 2019; 94, pp. 883-905. [DOI: https://dx.doi.org/10.1007/s10846-018-0906-5]

8. Liu, Y.; Wang, Y.; Shen, X.; Wang, J.; Shen, Y. UAV-aided relative localization of terminals. IEEE Internet Things J.; 2021; 8, pp. 12999-13013. [DOI: https://dx.doi.org/10.1109/JIOT.2021.3064216]

9. Nemra, A.; Aouf, N. Robust INS/GPS sensor fusion for UAV localization using SDRE nonlinear filtering. IEEE Sens. J.; 2010; 10, pp. 789-798. [DOI: https://dx.doi.org/10.1109/JSEN.2009.2034730]

10. Bianchi, M.; Barfoot, T.D. UAV localization using autoencoded satellite images. IEEE Robot. Autom. Lett.; 2021; 6, pp. 1761-1768. [DOI: https://dx.doi.org/10.1109/LRA.2021.3060397]

11. Niculescu, V.; Palossi, D.; Magno, M.; Benini, L. Energy-efficient, precise uwb-based 3-d localization of sensor nodes with a nano-uav. IEEE Internet Things J.; 2022; 10, pp. 5760-5777. [DOI: https://dx.doi.org/10.1109/JIOT.2022.3166651]

12. Xu, H.; Zhang, Y.; Zhou, B.; Wang, L.; Yao, X.; Meng, G.; Shen, S. Omni-swarm: A decentralized omnidirectional visual–inertial–uwb state estimation system for aerial swarms. IEEE Trans. Robot.; 2022; 38, pp. 3374-3394. [DOI: https://dx.doi.org/10.1109/TRO.2022.3182503]

13. Zhang, J.; Yu, B.; Ge, Y.; Gao, J.; Sheng, C. An Effective GNSS/PDR Fusion Positioning Algorithm on Smartphones for Challenging Scenarios. Sensors; 2024; 24, 1452. [DOI: https://dx.doi.org/10.3390/s24051452]

14. Yasyukevich, Y.V.; Zhang, B.; Devanaboyina, V.R. Advances in GNSS Positioning and GNSS Remote Sensing. Sensors; 2024; 24, 1200. [DOI: https://dx.doi.org/10.3390/s24041200] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38400358]

15. Couturier, A.; Akhloufi, M.A. A review on absolute visual localization for UAV. Robot. Auton. Syst.; 2021; 135, 103666. [DOI: https://dx.doi.org/10.1016/j.robot.2020.103666]

16. Qian, J.; Chen, K.; Chen, Q.; Yang, Y.; Zhang, J.; Chen, S. Robust visual-lidar simultaneous localization and mapping system for UAV. IEEE Geosci. Remote Sens. Lett.; 2021; 19, pp. 1-5. [DOI: https://dx.doi.org/10.1109/LGRS.2021.3099166]

17. Zheng, F.; Zhou, L.; Lin, W.; Liu, J.; Sun, L. LRPL-VIO: A Lightweight and Robust Visual–Inertial Odometry with Point and Line Features. Sensors; 2024; 24, 1322. [DOI: https://dx.doi.org/10.3390/s24041322] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38400480]

18. Caballero, F.; Merino, L. DLL: Direct LIDAR Localization. A map-based localization approach for aerial robots. Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Prague, Czech Republic, 27 September–1 October 2021; pp. 5491-5498.

19. Li, C.; Tanghe, E.; Suanet, P.; Plets, D.; Hoebeke, J.; De Poorter, E.; Joseph, W. ReLoc 2.0: UHF-RFID relative localization for drone-based inventory management. IEEE Trans. Instrum. Meas.; 2021; 70, pp. 1-13. [DOI: https://dx.doi.org/10.1109/TIM.2021.3069377]

20. Zeng, Q.; Jin, Y.; Yu, H.; You, X. A UAV Localization System Based on Double UWB Tags and IMU for Landing Platform. IEEE Sens. J.; 2023; 23, pp. 10100-10108. [DOI: https://dx.doi.org/10.1109/JSEN.2023.3260311]

21. Dong, X.; Gao, Y.; Guo, J.; Zuo, S.; Xiang, J.; Li, D.; Tu, Z. An Integrated UWB-IMU-Vision Framework for Autonomous Approaching and Landing of UAVs. Aerospace; 2022; 9, 797. [DOI: https://dx.doi.org/10.3390/aerospace9120797]

22. Kenyeres, M.; Kenyeres, J. Average consensus over mobile wireless sensor networks: Weight matrix guaranteeing convergence without reconfiguration of edge weights. Sensors; 2020; 20, 3677. [DOI: https://dx.doi.org/10.3390/s20133677]

23. Safavi, S.; Khan, U.A. Leader-follower consensus in mobile sensor networks. IEEE Signal Process. Lett.; 2015; 22, pp. 2249-2253. [DOI: https://dx.doi.org/10.1109/LSP.2015.2474134]

24. Yue, X.; Shao, X.; Zhang, W. Elliptical encircling of quadrotors for a dynamic target subject to aperiodic signals updating. IEEE Trans. Intell. Transp. Syst.; 2021; 23, pp. 14375-14388. [DOI: https://dx.doi.org/10.1109/TITS.2021.3127722]

25. Wang, Z.; Luo, Y. Elliptical Multi-Orbit Circumnavigation Control of UAVS in Three-Dimensional Space Depending on Angle Information Only. Drones; 2022; 6, 296. [DOI: https://dx.doi.org/10.3390/drones6100296]

26. Chen, L. Triangular angle rigidity for distributed localization in 2D. Automatica; 2022; 143, 110414. [DOI: https://dx.doi.org/10.1016/j.automatica.2022.110414]

27. Anderson, B. Exponential stability of linear equations arising in adaptive identification. IEEE Trans. Autom. Control; 1977; 22, pp. 83-88. [DOI: https://dx.doi.org/10.1109/TAC.1977.1101406]

28. Ha, M.; Wang, D.; Liu, D. Discounted iterative adaptive critic designs with novel stability analysis for tracking control. IEEE/CAA J. Autom. Sin.; 2022; 9, pp. 1262-1272. [DOI: https://dx.doi.org/10.1109/JAS.2022.105692]

29. Xiao, S.; Ge, X.; Han, Q.L.; Zhang, Y. Secure distributed adaptive platooning control of automated vehicles over vehicular ad hoc networks under denial-of-service attacks. IEEE Trans. Cybern.; 2021; 52, pp. 12003-12015. [DOI: https://dx.doi.org/10.1109/TCYB.2021.3074318] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34033572]

30. Fang, X.; Li, X.; Xie, L. 3-D distributed localization with mixed local relative measurements. IEEE Trans. Signal Process.; 2020; 68, pp. 5869-5881. [DOI: https://dx.doi.org/10.1109/TSP.2020.3029399]

31. Cao, K.; Han, Z.; Lin, Z.; Xie, L. Bearing-only distributed localization: A unified barycentric approach. Automatica; 2021; 133, 109834. [DOI: https://dx.doi.org/10.1016/j.automatica.2021.109834]

32. Cao, S.; Li, R.; Shi, Y.; Song, Y. Safe convex-circumnavigation of one agent around multiple targets using bearing-only measurements. Automatica; 2021; 134, 109934. [DOI: https://dx.doi.org/10.1016/j.automatica.2021.109934]