Orbital Pursuit–Evasion–Defense Linear-Quadratic Differential Game

Abstract

To find superior guidance strategies for preventing possible interception threats from space debris, out-of-control satellites, etc., this paper investigates an orbital pursuit–evasion–defense game problem with three players called the pursuer, the evader, and the defender, respectively. In this game, the pursuer aims to intercept the evader, while the evader tries to escape the pursuer. A defender accompanying the evader can protect the evader by actively intercepting the pursuer. For such a game, a linear-quadratic duration-adaptive (LQDA) strategy is first proposed as a basic strategy for the three players. Later, an advanced pursuit strategy is designed for the pursuer to evade the defender when they are chasing the evader. Meanwhile, a cooperative evasion–defense strategy is proposed for the evader and the defender to build their cooperation. Simulations determined that the proposed LQDA strategy has higher interception accuracy than the classic LQ strategy. Meanwhile, the proposed two-sided pursuit strategy can improve the interception performance of the pursuer against a non-cooperative defender. But if the evader and defender employ the proposed cooperation strategy, the pursuer’s interception will be much more difficult.

1. Introduction

Recently, increasing research efforts [1] have been dedicated to orbital differential games where spacecraft are regarded as conflicting players, and attempts are being made to maximize their interests. The most typical problem among them is the orbital pursuit–evasion game in which the spacecraft aims to optimize the survival performance indices such as miss distance, game duration, energy consumption, or a combination of them. While most previous research focus on the two-player orbital pursuit–evasion games [2], the game scenario that includes three spacecraft lacks much attention and remains quite open. In this paper, the three-player orbital pursuit–evasion–defense (PED) game has been investigated, in which the roles of the three spacecraft are the pursuer, the evader, and the defender, respectively. A motivating scenario for the PED problem is the active protection of in-orbit spacecraft from space debris (or out-of-control satellites) [3]. In this scenario, the in-orbit spacecraft is the evader, the space debris is considered to be the pursuer, and the spacecraft accompanying the evader is the defender. The defender can reduce the impact threat of the debris by actively intercepting the pursuer. This study formulated this active protection problem as a PED differential game, as mentioned in [4].

Pontani and Conway [5] conducted an early study on a type of two-player orbital pursuit–evasion problem for spacecraft interception, where the pursuer attempts to reduce the interception time while the evader aims to maximize it. A two-sided optimal solution, i.e., the saddle-point solution, is obtained by solving a challenging high-dimensional two-point boundary value problem (TPBVP). Thereafter, many researchers merge the state-of-the-art intelligent optimization algorithms, such as the evolutionary algorithms [6], and the traditional gradient optimization algorithms, such as the gradient descent algorithm [7], to form a variety of numerical methods for solving the TPBVP, such as the sensitive method [8], the shooting method [9], and the combined shooting and collocation methods [10]. Instead of focusing on the orbital pursuit–evasion game with interception time as the only objective, Jagat and Sinclair [11] examined the orbital linear-quadratic differential game (LQDG), where the pursuer and evader try to optimize the individual performance index, combining both the miss distance and the energy consumption. A pair of two-sided linear-quadratic (LQ) guidance laws are obtained. The orbital LQDG was further extended to nonlinear-quadratic cases by considering the nonlinear spacecraft dynamics [12]. Taking into account a more realistic information condition in LQDG, Li et al. [13] investigated the orbital pursuit–evasion game with incomplete information and proposed an optimal strategy for the evader. The above research primarily focuses on the orbital differential game between two players, whereas this paper investigates the game that involves three players.

Three-player differential games have been chiefly examined in the field of missile defense over the past couple of years [14], with a focus on the target–attacker–defender (TAD) issue. In this problem, a missile (attacker) is pursuing a non-maneuverable aircraft (target), while the target has the potential to launch another missile (defender) to protect itself by actively intercepting the attacker. Research efforts from various perspectives have been devoted to the problem. Shaferman and Shima [15] proposed a cooperative guidance law for the defender in which the possible guidance laws and parameters of the incoming homing missile are represented by employing a multiple-model adaptive estimator. Ratnoo and Shima [16] came forth with a guidance strategy for the pursuer in a situation when a set of guidance laws like the line-of-sight guidance are implemented for the defender-missile. Considering both sides, Perelman et al. [17] reported cooperative evasion and pursuit strategies based on linear kinematics. More meticulously, Prokopov and Shima [18] categorized the possible cooperation schemes between the target and the defender into three types: one-way cooperation realized by the target, one-way cooperation realized by the defender, and two-way cooperation. Comparison results highlighted that the two-way cooperation exhibited the best performance. Taking energy consumption as the optimization objective, Weiss et al. [19] determined the minimum interception and evasion effort required to achieve the desired performance in terms of miss distance. Some prominent studies have improved the above strategies from different aspects: the optimization of the switched system [20], the accessibility of the control information [21], suitability for the large distance case [22], and the utilization of the learning-based method [23]. Moreover, there is also a qualitative analysis of the three-player conflict problem presented by Rubinsky and Gutman [24] in which the algebraic conditions for the pursuer to capture the evader while escaping from the defender are examined.

Although extensive research has been conducted for the three-player TAD differential game in the fields of missiles, few research works have been presented for the orbital PED game that involves three spacecraft. Due to the dissimilarity in dynamics and game environment, the method developed for the missile TAD problem may not be directly applicable to the orbital PED game in which the gravitational difference among the players needs to be considered. Moreover, the game duration of the orbital game cannot be estimated using a linearized collision triangle like that in the TAD problem, which makes it more complex and difficult to solve. Moreover, the defender in space is not necessarily to be launched by the evader but can hover or lurk at a distance away from the evader at the beginning of the game. Existing research on the orbital PED problem is very limited. The interception trajectories of the orbital PED game under continuous thrust assumption have been optimized by Liu et al. [25]. The obtained solutions are open-loop and thus cannot reflect real-time conflicts among the three players, as they are being treated as a trajectory optimization problem rather than a closed-loop guidance problem. Later on, they also optimized an impulsive-transfer solution to this problem [26]. But, the solution is still open-loop. Liang et al. [27] considered space active defense problems in a two-on-two engagement, including an interceptor, a protector, a target spacecraft, and a defender. The closed-loop guidance laws are framed for them, neglecting the differences in the gravitational force among the players in the game formulation.

In this paper, the three-player orbital PED game is examined and treated as a closed-loop guidance problem. Orbital dynamics accounting for the gravitational force are employed for the spacecraft. Considering both the energy consumption and the miss distance, an LQDG formulation is adopted to model the game. Three categories of guidance strategies are progressively designed and compared. The key contributions of this study are as follows: (1) a two-sided optimal pursuit strategy is designed for the pursuer, which can enhance its self-defense ability while chasing a mission; (2) a cooperative evasion–defense strategy is devised for the evader and the defender to enhance their cooperation. Moreover, a variant of the LQ strategy, the linear-quadratic duration-adaptive (LQDA) strategy, is presented with time-to-go exclusively designed to achieve the interception condition.

The remainder of this paper is organized as follows: Section 2 briefly reviews the orbital dynamics and the two-player game model, and further introduces the formulation of the three-player orbital PED game. Section 3 represents the details of deriving and computing the LQDA guidance law, along with the two-sided optimal pursuit strategy, and the design of the cooperative evasion–defense strategy. Section 4 highlights the effectiveness of the proposed strategies via numerical simulations, followed by concluding remarks in Section 5.

2. Problem Formulation

Throughout this paper, vectors are represented in bold lowercase letters and matrices in bold uppercase. $‖·‖$ denotes the Euclidean norm of a vector. The superscript ${·}^T$ represents the transpose. The subscript ${·}_i$ indicates the player $i$, where $i=P$ for the pursuer, $i=E$ for the evader, and $i=D$ for the defender.

2.1. Orbital Relative Dynamics

The orbital PED game in this study is established in the Local Vertical Local Horizontal (LVLH) frame (refer to Figure 1), which is attached to a moving point in the orbit of the Earth. The $x$-axis is directed along the radius vector from the Earth’s center to the moving point. The $z$-axis is along the reference orbit normal and the $y$-axis is in the reference orbit plane which follows the right-hand coordinate system. If the moving point is regarded as the chief spacecraft, then the pursuer, the evader, and the defender are all deputies of it. Define ${\mathit{x}}_i={(x_i,y_i,{\mathit{z}}_i,{\dot{x}}_i,{\dot{y}}_i,{\dot{\mathit{z}}}_i)}^T\in {ℝ}^6$ as the vector of states in the LVLH reference frame and ${\mathit{u}}_i={(a_{i,x},a_{i,y},a_{i,\mathit{z}})}^T\in {ℝ}^3$ as the vector of control inputs, where the subscripts $i=P, E, D$ denote the pursuer, the evader, and the defender, respectively. Neglecting the orbital perturbations and assuming that the distances between the deputies and the chief are much shorter than those from the deputies to the Earth’s center, the dynamics of the deputy spacecraft can be described using the following Clohessy–Wiltshire Equation (1) [28]:

(1)${\dot{\mathit{x}}}_i=\mathit{A}{\mathit{x}}_i+\mathit{B}{\mathit{u}}_i$

where

$\mathit{A}=\left[\begin{array}{cccccc}0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 1\\ 3{\omega }^2& 0& 0& 0& 2\omega & 0\\ 0& 0& 0& -2\omega & 0& 0\\ 0& 0& -{\omega }^2& 0& 0& 0\end{array}\right],\mathit{B}=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\\ 1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$

$\omega$ is the angular velocity of the reference moving point. When the spacecraft has no controls, the state-transfer matrix $\mathbf{\Phi }\left(t_f,t\right)$ from the current game time $t$ to the terminal game time $t_f$ has the following form (2):

(2)$\mathbf{\Phi }\left(t_f,t\right)=\left[\begin{array}{cccccc}4-3c& 0& 0& s/\omega & -2(c-1)/\omega & 0\\ 6(s-\omega \tau )& 1& 0& 2(c-1)/\omega & (4s-3\omega \tau )/\omega & 0\\ 0& 0& c& 0& 0& s/\omega \\ 3\omega s& 0& 0& c& 2s& 0\\ 6\omega (c-1)& 0& 0& -2s& -3+4c& 0\\ 0& 0& -\omega s& 0& 0& c\end{array}\right]$

where $\tau =(t_f-t)$, $s=\sin \omega \tau$, and $c=\cos \omega \tau$.

2.2. Two-Player Pursuit–Evasion Game

The linear-quadratic pursuit–evasion game between two players is defined by a linear dynamical system as Equation (3):

(3)$\dot{\mathit{x}}(t)=\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{u}}_P(t)-\mathit{B}{\mathit{u}}_E(t)$

A quadratic payoff function [2] that the pursuer attempts to minimize while the evader aims to maximize is in Equation (5):

(4)$J=\frac{1}{2}{\mathit{x}}^T(t_f){\mathit{Q}}_f\mathit{x}(t_f)+\frac{1}{2}{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}[{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)}-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)]dt$

where $\mathit{x}(t)={\mathit{x}}_P(t)-{\mathit{x}}_E(t)$ signifies a six-dimensional vector describing the state of the pursuit–evasion game system; $t_{\mathbf{0}}$ and $t_f$ are the start and end time, respectively; the weighting matrix of the terminal state ${\mathit{Q}}_f\in {ℝ}^{6\times 6}$ is a semi-positive definite matrix; and the weighting matrices of the consumed energy ${\mathit{R}}_P,{\mathit{R}}_E\in {ℝ}^{3\times 3}$ represent positive definite matrices. Diagonal weighting matrices are used in this study. The weighting matrices are assumed to be in the following diagonal form (5):

(5)${\mathit{Q}}_{fi}=q_{fi}\left[\begin{array}{cc}{\mathbf{I}}_{\mathrm{3}}& {\mathbf{0}}_{\mathrm{3}}\\ {\mathbf{0}}_3& {\mathbf{0}}_3\end{array}\right],{\mathit{R}}_{1i}=r_{1i}{\mathbf{I}}_3,{\mathit{R}}_{2i}=r_{2i}{\mathbf{I}}_3$

where ${\mathbf{I}}_3$ denotes a three-dimensional identity matrix, and $q_{fi}$, $r_{1i}$, and $r_{2i}$ are the weighting coefficients; the form of ${\mathit{Q}}_{fi}$ signifies that the game is an interception game.

The optimal control laws of the pursuer and the evader can be determined by solving for the saddle-point solution of the game, which is defined as a pair of strategies $({\mathit{u}}_P^{*},{\mathit{u}}_E^{*})$ that satisfies the following inequalities [10]:

(6)$J({\mathit{u}}_P^{*},{\mathit{u}}_E)\le J({\mathit{u}}_P^{*},{\mathit{u}}_E^{*})\le J({\mathit{u}}_P,{\mathit{u}}_E^{*})$

According to Ref. [29], the saddle-point strategies of the pursuer and the evader have the following forms:

(7)${\mathit{u}}_P^{*}(t)=-{{\mathit{R}}_P}^{-1}{\mathit{B}}^T\mathit{P}(t)\mathit{x}(t)$

(8)${\mathit{u}}_E^{*}(t)=-{{\mathit{R}}_E}^{-1}{\mathit{B}}^T\mathit{P}(t)\mathit{x}(t)$

where $\mathit{P}(t)$ represents the solution of the following matrix Riccati differential equation [11]:

(9)$\dot{\mathit{P}}(t)=\mathit{P}(t)(\mathit{B}{{\mathit{R}}_1}^{-1}{\mathit{B}}^T-\mathit{B}{{\mathit{R}}_2}^{-1}{\mathit{B}}^T)\mathit{P}(t)-{\mathit{A}}^T(t)\mathit{P}(t)-\mathit{P}(t)\mathit{A}(t)-\mathit{Q},\mathit{P}(t_f)={\mathit{Q}}_f$

The saddle-point strategies $({\mathit{u}}_P^{*},{\mathit{u}}_E^{*})$ constitute a Nash equilibrium [2], at which any deviation from each player’s separation from $({\mathit{u}}_P^{*},{\mathit{u}}_E^{*})$ may lead to an increase in its payoff. In the subsequent sections, the strategies of this linear-quadratic game in Equations (7) and (8) are also identified as the LQ strategy.

2.3. Three-Player Pursuit–Evasion–Defense Game

If a defender is added to the above two-player game, the game gets converted into a three-player pursuit–evasion–defense game. In this case, the purpose of the defender is to protect the evader from being intercepted by the pursuer through a medium of actively intercepting the pursuer.

To begin with, consider a simple case where the three spacecraft are “independent” of each other. In this case, the three-player game can be formulated as two sets of two-player pursuit–evasion games. Firstly, a pursuit–evasion game is formulated between the pursuer and the evader:

(10)$\dot{\mathit{x}}(t)=\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{u}}_P(t)-\mathit{B}{\mathit{u}}_E(t)$

(11)$J_{PE}=\frac{1}{2}\mathit{x}{(t_f)}^T{\mathit{Q}}_f\mathit{x}(t_f)+{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}[{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)}-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)]dt$

where the superscript “PE” marks the game between the pursuer and the evader. Further, between the pursuer and the defender, another pursuit–evasion game appears as follows:

(12)$\dot{\mathit{y}}(t)=\mathit{A}\mathit{y}(t)+\mathit{B}{\mathit{u}}_D(t)-\mathit{B}{\mathit{u}}_P(t)$

(13)$J_{DP}=\frac{1}{2}\mathit{y}{(t_f)}^T{\mathit{Q}}_f\mathit{y}(t_f)+{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}[{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)}-{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)]dt$

where $\mathit{y}(t)={\mathit{x}}_D(t)-{\mathit{x}}_P(t)$ is six-dimensional relative state of the system (12); ${\mathit{u}}_D$ denotes the control input vector of the defender; the subscript “DP” marks the game between the pursuer and the defender; and ${\mathit{R}}_D\in {ℝ}^{3\times 3}$ indicates the weighting matrix of the defender’s energy consumption. In this game, the defender plays the role of a chaser against the pursuer.

Two more complex cases are further considered, in one of which the pursuer simultaneously chases the evader and avoids interception from the defender. In the other case, the evader and the defender cooperatively confront the pursuer. Different guidance laws are derived for each case, as described in Section 3.

The three-player game terminates if the pursuer intercepts the evader or the defender intercepts the pursuer. Let m be the permitted terminal miss distance between the spacecraft, $r_{PE}=‖\mathit{D}\mathit{x}(t)‖$, and $r_{DP}=‖\mathit{D}\mathit{y}(t)‖$; then, the terminal time of the game is defined as

(14)$t_f=\min \{t | r_{PE}(t)=m \left|\right| r_{DP}(t)=m\}$

where $\mathit{D}$ = [${\mathbf{I}}_3:\mathbf{0}$]. If the defender’s interception of the pursuer is earlier than the moment when the pursuer arrives at the evader, the evader and defender will win the game; otherwise, the pursuer will win the game. Based on the above game model, the guidance strategies for the three players will be designed subsequently.

3. Strategy Design

Different strategies are designed for the three introduced cases. The simplest scenario is where the three players are independent. The pursuer merely chases the evader and has no maneuvers to evade the defender, the evader simply dodges the interception of the pursuer and has no coordination with the defender, and the defender focuses solely on chasing the pursuer. For this scenario, a linear-quadratic duration-adaptive (LQDA) strategy has been designed for the three players. Further, considering that the pursuer can chase the evader and simultaneously evade the defender, a two-sided optimal pursuit strategy has been proposed for the pursuer. Eventually, considering the potential coordination between the evader and defender, a cooperative optimal evasion–defense strategy has been designed.

3.1. Linear-Quadratic Duration-Adaptive Strategy

As discussed in Section 2.3, the three-player game in the first scenario can be formulated as two sets of two-player pursuit–evasion games. But, the LQ strategy in Equations (7) and (8) cannot be applied directly to the game since it terminates at a fixed duration. For this three-player game terminating by the interception condition in Equation (15), an LQDA strategy is designed for the spacecraft players.

The primary idea of the LQDA strategy is to adjust the time-to-go according to the real-time miss distance and achieve a terminal interception, or to say, to complete a terminal control (TC) [30] in the pursuit–evasion game. More specifically, the LQDA strategy has the same form of control laws as the classic LQ strategy, but the time-to-go is designed in a different manner. The idea of adjusting the time-to-go can be seen in the literature concerning missiles [31]. Gutman and Rubinsky [32] derived analytical vector guidance laws through the analysis of the time-to-go. Later on, Ye et al. [33] applied this idea to a kind of orbital pursuit–evasion game that considers the terminal miss distance as the only payoff function. But the methods do not apply to this LQ pursuit–evasion game that considers both the miss distance and energy consumption. So, this section presents the details of the LQDA strategy.

3.1.1. Derivation

For the convenience of expression, a reduced state vector is utilized to describe such an interception game:

(15)$\mathit{z}(t)=\mathit{D}\mathbf{\Phi }(t_f,t)\mathit{x}(t)$

where $\mathit{z}(t)\in {ℝ}^3$ is a three-dimensional state vector known as the zero-effort miss (ZEM) [31], predicted at time t based on the fact that no control effort will be applied during the time interval $[t,t_f]$; $\mathbf{\Phi }(t_f,t)$ represents the state-transition matrix.

Further, the game is defined by Equations (10) and (11) can be simplified using the following equations:

(16)$\dot{\mathit{z}}(t)={\mathit{G}}_P(t_f,t){\mathit{u}}_P(t)-{\mathit{G}}_E(t_f,t){\mathit{u}}_E(t)$

(17)$J=\frac{1}{2}\mathit{z}{(t_f)}^T{\mathit{Q}}_T\mathit{z}(t_f)+{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}[{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)}-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)]dt$

where the control matrix ${\mathit{G}}_i(t_f,t)=\mathit{D}{\mathbf{\Phi }}_i(t_f,t)\mathit{B}$ and the weighting matrix ${\mathit{Q}}_T=\mathit{D}{\mathit{Q}}_f{\mathit{D}}^T$.

Accordingly, the form of the saddle-point solution becomes the following [34]:

(18)${\mathit{u}}_P(t)=-{\mathit{R}}_P^{-1}{\mathit{G}}_P^T(\tau ){\mathit{K}}^{-1}(\tau )\mathit{z}(t)$

(19)${\mathit{u}}_E(t)=-{\mathit{R}}_E^{-1}{\mathit{G}}_E^T(\tau ){\mathit{K}}^{-1}(\tau )\mathit{z}(t)$

where the time-to-go $\tau =t_f-t$ replaces the game duration. The gain

(20)$\mathit{K}(\tau )={\mathit{Q}}_T^{-1}+{\mathit{M}}_P(\tau )-{\mathit{M}}_E(\tau )$

where the reduced controllability matrices ${\mathit{M}}_i(\tau ), i=P,E$ are computed by:

(21)${\mathit{M}}_i(\tau )={\displaystyle \underset{t}{\overset{t+\tau }{\int }}{\mathit{G}}_i(\tau )}{\mathit{R}}_i^{-1}{\mathit{G}}_i^T(\tau )dt$

The above solutions are derived from a standard LQ finite-time game, lacking a terminal constraint according to Equations (14) and (15), i.e.,

(22)$‖\mathit{z}(t_f)‖=‖\mathit{D}\mathit{x}(t_f)‖=r_{PE}=m$

which transforms the original problem into a free-time pursuit–evasion TC problem [30]. To make the solutions in Equations (18) and (19) adaptive to the problem, the time-to-go is exclusively designed.

With the controls of the saddle-point strategy input, the final ZEM depends on the following state equation:

(23)$\dot{\mathit{z}}(t)=[{\mathit{G}}_E(\tau ){\mathit{R}}_E^{-1}{\mathit{G}}_E^T(\tau )-{\mathit{G}}_P(\tau ){\mathit{R}}_P^{-1}{\mathit{G}}_P^T(\tau )]{\mathit{K}}^{-1}(\tau )\mathit{z}(t)$

Denoting $\mathit{H}(\tau )=[{\mathit{G}}_E(\tau ){\mathit{R}}_E^{-1}{\mathit{G}}_E^T(\tau )-{\mathit{G}}_P(\tau ){\mathit{R}}_P^{-1}{\mathit{G}}_P^T(\tau )]{\mathit{K}}^{-1}(\tau )$ and combining the initial condition $\mathit{z}(t_0)={\mathit{z}}_0$, Equation (23) can be rewritten as follows:

(24)$\dot{\mathit{z}}(t)=\mathit{H}(\tau )\mathit{z}(t),\mathit{z}(t_0)={\mathit{z}}_0$

For this first-order linear time-variant ordinary differential equation, a closed-form solution of $\mathit{z}(t)$ does not exist unless ${\displaystyle \int \mathit{H}(\tau ){\mathit{H}}^T(\tau )}dt={\displaystyle \int {\mathit{H}}^T(\tau )\mathit{H}(\tau )}dt$ [35]. The $\mathit{z}(t_f)$ can be computed by numerically integrating Equation (24), but an approximate method is adopted to solve it efficiently.

First, the interval $[t_0,t_f]$ is evenly divided into N ($N\to \infty$) subintervals, $[t_0,t_1]$,$[t_1,t_2]$, …, $[t_{l-1},t_l]$, …, $[t_{N-1},t_N]$, where $t_N=t_f,t_{l-1}=t_0+(l-1)(t_f-t_0)/N,l=1,2,\dots ,N$. In each subinterval $[t_{l-1},t_l]$, $\mathit{H}(\tau )$ is taken as a constant matrix $\mathit{H}({\tau }_{l-1})$, where ${\tau }_{l-1}=t_f-t_{l-1}={\tau }_0(N+1-l)/N$. According to the analytical solution of the linear time-invariant system without control inputs [35],

(25)$\left\{\begin{array}{l}\mathit{z}(t_1)=e^{\mathit{H}({\tau }_0)(t_1-t_0)}\mathit{z}(t_0),\\ \mathit{z}(t_2)=e^{\mathit{H}({\tau }_1)(t_2-t_1)}\mathit{z}(t_1),\\ \dots ,\\ \mathit{z}(t_f)=e^{\mathit{H}({\tau }_{N-1})(t_f-t_{N-1})}\mathit{z}(t_{N-1})\end{array}\right.$

Let $\Delta t=t_l-t_{l-1},l=1,2,\dots ,N$, then

(26)$\begin{array}{ll}\mathit{z}(t_f)& =[e^{\mathit{H}({\tau }_0)\Delta t}{e}^{\mathit{H}({\tau }_1)\Delta t}\cdot e^{\mathit{H}({\tau }_{N-1})\Delta t}]\mathit{z}(t_0)=[e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}({\tau }_{l-1})}}]\mathit{z}(t_0)\\ & =[e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}(\frac{(N+1-l){\tau }_0}{N})}}]{\mathit{z}}_0=[e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}(\frac{(N+1-l)\tau }{N})}}]\mathit{z}(t)\\ & =[e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}(\frac{(N+1-l)\tau }{N})}}]\mathit{D}\mathbf{\Phi }(t)\mathit{x}(t)\end{array}$

Combining Equations (22) and (26), the following relationship can be derived:

(27)$‖\mathit{z}(t_f)‖=‖e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}(\frac{(N+1-l)\tau }{N})}}\mathit{D}\mathbf{\Phi }(\tau )\mathit{x}(t)‖=‖\mathit{D}\mathit{x}(t_f)‖=r_{PE}=m$

The designed time-to-go $\tau$ is the exact solution of Equation (27). Given the current miss distance, the current time-to-go can be computed by reversely solving the above equation.

The procedure for computing the time-to-go is presented in Section 3.1.2. Given the time-to-go $\tau$, the control laws for the pursuer and the evader will be derived as follows:

(28)${\mathit{u}}_P(t)={\mathit{F}}_P(\tau )\mathit{z}(t)=-{\mathit{R}}_P^{-1}{\mathit{G}}^T(\tau ){\mathit{K}}^{-1}(\tau )\mathit{D}\mathbf{\Phi }(\tau )\mathit{x}(t)$

(29)${\mathit{u}}_E(t)={\mathit{F}}_E(\tau )\mathit{z}(t)=-{\mathit{R}}_E^{-1}{\mathit{G}}^T(\tau ){\mathit{K}}^{-1}(\tau )\mathit{D}\mathbf{\Phi }(\tau )\mathit{x}(t)$

where the values of the control gains keep fluctuating with the real-time miss distance. Note that although the control laws have a linear form, they are nonlinear since the equation of the time-to-go is nonlinear.

Similar to the pursuer in Equation (28), the control law of the defender is given by

(30)${\mathit{u}}_D(t)={\mathit{F}}_D(\overline{\tau })\overline{\mathit{z}}(t)=-{\mathit{R}}_D^{-1}{\mathit{G}}^T(\overline{\tau }){\mathit{K}}^{-1}(\overline{\tau })\mathit{D}\mathbf{\Phi }(\overline{\tau })\mathit{y}(t)$

where $\overline{\tau }$ represents the time-to-go of the defender calculated by Equation (13) and $\overline{\mathit{z}}$ indicates the ZEM of the defender.

Eventually, Equations (28)–(30) constitute the LQDA strategy for the three independent players.

3.1.2. Calculation of Time-to-Go

Precisely, the time-to-go is the minimum positive zero-point of the following function $\mathit{g}(\tau )$:

(31)$\mathit{g}(\tau )=‖e^{\Delta t{\displaystyle \sum _{l=1}^N\mathit{H}(\frac{(N+1-l)\tau }{N})}}\mathit{D}\mathbf{\Phi }(\tau )\mathit{x}(t)‖-m$

Since that $\mathit{g}(\tau )$ is not a monotone function and may have multiple zero points, solving the time-to-go demands special techniques. Note that ${\mathit{g}}_k(\tau )$ is used here to denote the $\mathit{g}(\tau )$ when $t=t_k,k\in \mathbb{N}$, because $\mathit{g}(\tau )$ changes with $\mathit{x}(t)$ during the game.

Two rational assumptions are made that (1) at the beginning of the game, the distance between the two players is larger than the permitted miss distance [m in Equation (14)]; (2) given stronger maneuverability of the chasing player along with sufficient time, the miss distance will be smaller than the permitted value; $\mathit{g}(0)=‖\mathit{D}\mathbf{\Phi }(0)\mathit{x}(t_0)‖-m=‖r(t_0)‖-m>0$, and $\mathit{g}(\infty )<0$. These two assumptions guarantee the existence of the time-to-go.

Based on these assumptions, the first zero point must lie in the descending segment of the curve of $\mathit{g}(\tau )$, i.e., ${\mathit{g}}^{\mathbf{\prime }}(\tau )\le 0$, excluding all the other zero points with ${\mathit{g}}^{\mathbf{\prime }}(\tau )>0$. After extensive simulations with a variety of different parameters, m, ${\mathit{R}}_P$, ${\mathit{R}}_E$, and $\mathit{x}(t)$, it can be concluded that (3) the time interval between a pair of adjacent peak and trough of $\mathit{g}(\tau )$ is no less than the 50 s even for extremely fluctuating $\mathit{g}(\tau )$ (refer Figure 2); (4) the change in the time-to-go between two consecutive steps is significantly less than 50 s unless there is a critical point (refer Figure 3). Because of this, the time-to-go calculated in the last step is taken as the initial guess, and Newton’s method [36] is used to efficiently solve the current time-to-go irregularities.

But as for the time-to-go at the initial phase of the game or when a critical point appears, it is mandatory to look for an appropriate initial guess. According to the analysis (3) and (4), a minimum $\tau$ can be obtained, subject to ${\mathit{g}}^{\mathbf{\prime }}(\tau )<0$ from $\tau =0$ to $\tau =T$, with a step of $\Delta \tau =50 \mathrm{s}$ ($T$ is sufficiently large). This $\tau$ is later used as the initial guess in Newton’s method. The entire algorithm is summarized in Algorithm 1.

Algorithm 1. Calculation of the time-to-go in the game
1:	Initialize all the parameters in ${\mathit{g}}_0(\tau )$ when $t=t_0$.
2:	For $\tau =0,\Delta \tau ,2\Delta \tau ,\dots ,T$ do
3:	Calculate the derivative of ${\mathit{g}}_0(\tau )$ with respect to $\tau$, ${{\mathit{g}}_0}^{\mathbf{\prime }}(\tau )$.
4:	If ${{\mathit{g}}_0}^{\mathbf{\prime }}(\tau )\le 0$,
5:	Take $\tau$ as the initial guess and solve ${{\mathit{g}}_0}^{\mathbf{\prime }}({\tau }_0^{\ast })=0$ using the Newton method.
6:	If ${\tau }_0^{\ast }$ is founded within the accuracy of iteration,
7:	Break the loop.
8:	End if
9:	End if
10:	End for
11:	For $t=t_1,\dots ,t_i,\dots ,t_N$ do
12:	Let $\tau ={\tau }_{i-1}^{\ast }$ and calculate the value of ${{\mathit{g}}_i}^{\mathbf{\prime }}(\tau )$.
13:	If ${{\mathit{g}}_i}^{\mathbf{\prime }}(\tau )\le 0$,
14:	Take $\tau$ as the initial guess and solve ${\mathit{g}}_i({\tau }_i^{\ast })=0$ using the Newton method.
15:	If ${\tau }_i^{\ast }$ is founded within the accuracy of iteration,
16:	Break the loop.
17:	Else
18:	Repeat steps 2 to 8 to obtain the ${\tau }_i^{\ast }$ and break the loop.
19:	End if
20:	Else
21:	Let $\tau ={\tau }_{i-1}^{\ast }-\Delta \tau$, calculate the value of ${{\mathit{g}}_i}^{\mathbf{\prime }}(\tau )$.
22:	End if
23:	Repeat steps 11 to 18.
24:	End for

3.2. Two-Sided Optimal Pursuit Strategy

In the second scenario, the pursuer can chase the evader and simultaneously evade the defender. A two-sided optimal pursuit strategy for the pursuer is further proposed, where “two-sided” implies to take both the evader and the defender into consideration.

From the pursuer’s angle of view, the game system is a combination of Equations (10) and (12):

(32)$\left\{\begin{array}{l}\dot{\mathit{x}}(t)=\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{u}}_P(t)-\mathit{B}{\mathit{u}}_E(t)\\ \dot{\mathit{y}}(t)=\mathit{A}\mathit{y}(t)+\mathit{B}{\mathit{u}}_D(t)-\mathit{B}{\mathit{u}}_P(t)\end{array}\right.$

To behave both as a pursuer as well as an evader, a comprehensive payoff function is needed to be constructed for the pursuer to simultaneously handle the two opponents:

(33)$J_P=\frac{1}{2}\mathit{x}{(t_f)}^T{\mathit{Q}}_f\mathit{x}(t_f)-\frac{1}{2}\mathit{y}{(t_f)}^T{\mathit{Q}}_f\mathit{y}(t_f)+\frac{1}{2}{\displaystyle \underset{t_0}{\overset{t_f}{\int }}}{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)dt$

To solve the optimal pursuing controls for the pursuer a Hamiltonian $H$ and a terminal function $\mathbf{\Phi }$ are constructed as follows:

(34)$\begin{array}{ll}H(t)& =\frac{1}{2}[{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)]\\ & +{\mathit{\lambda }}_E^T[\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{u}}_P(t)-\mathit{B}{\mathit{u}}_E(t)]+{\mathit{\lambda }}_D^T[\mathit{A}\mathit{y}(t)+\mathit{B}{\mathit{u}}_D(t)-\mathit{B}{\mathit{u}}_P(t)]\end{array}$

(35)$\mathbf{\Phi }(t_f)=\frac{1}{2}\mathit{x}{(t_f)}^T{\mathit{Q}}_f\mathit{x}(t_f)-\frac{1}{2}\mathit{y}{(t_f)}^T{\mathit{Q}}_f\mathit{y}(t_f)$

where ${\mathit{\lambda }}_E$ and ${\mathit{\lambda }}_D$ denote the co-states associated with the state equations of $\mathit{y}(t)$ and $\mathit{z}(t)$, respectively. Substituting Equations (29) and (30) into Equation (34),

(36)$\begin{array}{ll}H& =\frac{1}{2}[{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)-{\mathit{x}}^T(t){\mathit{F}}_E^T(\tau ){\mathit{R}}_E{\mathit{F}}_E(\tau )\mathit{x}(t)-{\mathit{y}}^T(t){\mathit{F}}_D^T(\tau ){\mathit{R}}_D{\mathit{F}}_D(\tau )\mathit{y}(t)]\\ & +{\mathit{\lambda }}_E^T[\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{u}}_P(t)-\mathit{B}{\mathit{F}}_E(\tau )\mathit{x}(t)]+{\mathit{\lambda }}_D^T[\mathit{y}(t)+\mathit{B}{\mathit{F}}_D(\tau )\mathit{z}(t)-{\mathit{G}}_P(\tau ){\mathit{u}}_P(t)]\end{array}$

According to Pontryagin’s maximum principle (PMP), the optimal controls of the pursuer must minimize the Hamiltonian. Thus, the derivative of Hamiltonian with respect to the pursuer’s control inputs satisfies the following equation:

(37)$\frac{\partial H}{\partial {\mathit{u}}_P}={\mathit{R}}_P{\mathit{u}}_P(t)+{\mathit{B}}^T{\mathit{\lambda }}_E-{\mathit{B}}^T{\mathit{\lambda }}_D=0\Rightarrow {\mathit{u}}_P(t)=-{\mathit{R}}_P^{-1}{\mathit{B}}^T({\mathit{\lambda }}_E-{\mathit{\lambda }}_D)$

The evolution of the co-state vectors ${\mathit{\lambda }}_E$ and ${\mathit{\lambda }}_D$ follows the adjoint equations:

(38)${\dot{\mathit{\lambda }}}_E(t)=-\frac{\partial H}{\partial \mathit{x}(t)}={\mathit{F}}_E^T(\tau ){\mathit{L}}_3{\mathit{F}}_E(\tau )\mathit{x}(t)-{[\mathit{A}-\mathit{B}{\mathit{F}}_E(\tau )]}^T{\mathit{\lambda }}_E$

(39)${\dot{\mathit{\lambda }}}_D(t)=-\frac{\partial H}{\partial \mathit{y}(t)}={\mathit{F}}_D^T(\tau ){\mathit{L}}_4{\mathit{F}}_D(\tau )\mathit{y}(t)-{[\mathit{A}+\mathit{B}{\mathit{F}}_D(\tau )]}^T{\mathit{\lambda }}_D$

The transversality condition of the co-state satisfies the following conditions:

(40)${\mathit{\lambda }}_E(t_f)=\frac{\partial \mathbf{\Phi }(t_f)}{\partial \mathit{x}(t_f)}{\mathit{Q}}_f\mathit{x}(t_f)$

(41)${\mathit{\lambda }}_D(t_f)=\frac{\partial \mathbf{\Phi }(t_f)}{\partial \mathit{y}(t_f)}=-{\mathit{Q}}_f\mathit{y}(t_f)$

Substituting Equations (38) and (39) into Equation (37), and further substituting Equation (37) into Equation (32), then the following equation can be obtained:

(42)$\left\{\begin{array}{l}\dot{\mathit{x}}(t)=[\mathit{A}-\mathit{B}{\mathit{F}}_E(\tau )]\mathit{x}(t)-\mathit{B}{\mathit{R}}_P^{-1}\mathit{B}({\mathit{\lambda }}_E-{\mathit{\lambda }}_D)\\ \dot{\mathit{y}}(t)=[\mathit{A}+\mathit{B}{\mathit{F}}_D(\tau )]\mathit{y}(t)+\mathit{B}{\mathit{R}}_P^{-1}\mathit{B}({\mathit{\lambda }}_E-{\mathit{\lambda }}_D)\end{array}\right.$

Combining the equations of the state and the co-state, we have

(43)$\left[\begin{array}{c}\dot{\mathit{x}}\\ \dot{\mathit{y}}\end{array}\right]=\left[\begin{array}{cc}\mathit{A}-\mathit{B}{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& \mathit{A}+\mathit{B}{\mathit{F}}_D(\tau )\end{array}\right]\left[\begin{array}{c}\mathit{x}\\ \mathit{y}\end{array}\right]+\left[\begin{array}{cc}-\mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T& \mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T\\ \mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T& -\mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T\end{array}\right]\left[\begin{array}{c}{\mathit{\lambda }}_E\\ {\mathit{\lambda }}_D\end{array}\right]$

(44)$\left[\begin{array}{c}{\dot{\mathit{\lambda }}}_E\\ {\dot{\mathit{\lambda }}}_D\end{array}\right]=\left[\begin{array}{cc}{\mathit{F}}_E^T(\tau ){\mathit{R}}_E{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& {\mathit{F}}_D^T(\tau ){\mathit{R}}_D{\mathit{F}}_D(\tau )\end{array}\right]\left[\begin{array}{c}\mathit{y}\\ \mathit{z}\end{array}\right]+\left[\begin{array}{cc}-{\mathit{A}}^T+{\mathit{F}}_E^T(\tau ){\mathit{B}}^T& \mathbf{0}\\ \mathbf{0}& -{\mathit{A}}^T-{\mathit{F}}_D^T(\tau ){\mathit{B}}^T\end{array}\right]\left[\begin{array}{c}{\mathit{\lambda }}_E\\ {\mathit{\lambda }}_D\end{array}\right]$

Let $\mathit{X}={[{\mathit{x}}^T,{\mathit{y}}^T]}^T$, $\mathit{\Lambda }={[{\mathit{\lambda }}_E^T,{\mathit{\lambda }}_D^T]}^T$, and let

(45)$$\begin{gathered} \mathit{M}=\left[\begin{array}{cc}\mathit{A}-\mathit{B}{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& \mathit{A}+\mathit{B}{\mathit{F}}_D(\tau )\end{array}\right],\mathit{V}=\left[\begin{array}{cc}{\mathit{F}}_E^T(\tau ){\mathit{L}}_3{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& {\mathit{F}}_D^T(\tau ){\mathit{L}}_4{\mathit{F}}_D(\tau )\end{array}\right] \\ \mathit{N}=\left[\begin{array}{cc}-\mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T& \mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T\\ \mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T& -\mathit{B}{\mathit{R}}_P^{-1}{\mathit{B}}^T\end{array}\right],\mathit{W}=\left[\begin{array}{cc}-{\mathit{A}}^T+{\mathit{F}}_E^T(\tau ){\mathit{B}}^T& \mathbf{0}\\ \mathbf{0}& -{\mathit{A}}^T-{\mathit{F}}_D^T(\tau ){\mathit{B}}^T\end{array}\right] \end{gathered}$$

Therefore, Equation (43) can be rewritten in the matrix form as below:

(46)$\left[\begin{array}{c}\dot{\mathit{X}}(t)\\ \dot{\mathit{\Lambda }}(t)\end{array}\right]=\left[\begin{array}{cc}\mathit{M}(\tau )& \mathit{N}(\tau )\\ \mathit{V}(\tau )& \mathit{W}(\tau )\end{array}\right]\left[\begin{array}{c}\mathit{X}(t)\\ \mathit{\Lambda }(t)\end{array}\right]$

A linear relationship between the state and the co-state is obtained as below:

(47)$\mathit{\Lambda }(t)=\mathit{P}(t)\mathit{X}(t)$

where $\mathit{P}(t)$ represents the Riccati matrix. By substituting Equation (47) into Equation (46), a matrix Riccati differential equation is derived as follows:

(48)$\dot{\mathit{P}}(t)=-\mathit{P}(t)\mathit{M}-{\mathit{M}}^T\mathit{P}(t)-\mathit{P}(t)\mathit{N}\mathit{P}(t)+\mathit{V}$

Substituting Equation (48) into Equations (40) and (41), the boundary condition of Equation (48) is derived as

(49)$\mathit{P}(t_f)=\left[\begin{array}{cc}{\mathit{Q}}_f& \mathbf{0}\\ \mathbf{0}& -{\mathit{Q}}_f\end{array}\right]$

By integrating Equation (48) backwards, the matrix $\mathit{P}(t)$ can be solved. Then, the optimal control law of the pursuer in Equation (37) can be calculated by

(50)${\mathit{u}}_P(t)=-{\mathit{R}}_P^{-1}{\mathit{B}}^T{\mathit{S}}_P\mathit{P}(t)\mathit{X}(t)$

where ${\mathit{S}}_P=[{\mathbf{I}}_3,-{\mathbf{I}}_3]$.

Eventually, Equation (50) provides a two-sided optimal pursuit strategy for an offensive as well as defensive pursuer.

3.3. Cooperative Evasion–Defense Strategy

The evader and the defender can coordinate with each other to defeat the pursuer. With the help of the defender, the evader can operate without moving far away from the pursuer. Instead, it only needs to maintain a decent distance from the pursuer, preferably more than that between the defender and the pursuer. Based on this, the following payoff function is derived for the evader and the defender:

(51)$\begin{array}{l}{J}_{DE}=\frac{1}{2}{\mathit{Q}}_f\left({‖\mathit{x}(t_f)‖}^2-{‖\mathit{y}(t_f)‖}^2\right)+\frac{1}{2}{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}}{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)dt\\ =\frac{1}{2}\mathit{x}{(t_f)}^T{\mathit{Q}}_f\mathit{x}(t_f)-\frac{1}{2}\mathit{y}{(t_f)}^T{\mathit{Q}}_f\mathit{y}(t_f)+\frac{1}{2}{\displaystyle \underset{t_{\mathbf{0}}}{\overset{t_f}{\int }}}{\mathit{u}}_P^T(t){\mathit{R}}_P{\mathit{u}}_P(t)-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)dt\end{array}$

In the above equation, the square of the distance is utilized to replace the distance for maintaining a quadratic form of the payoff function, which is the same as that in Equation (33).

Based on this, the Hamiltonian is constructed as

(52)$\begin{array}{ll}H& =\frac{1}{2}[{\mathit{x}}^T(t){\mathit{F}}_P^T(\tau ){\mathit{R}}_P{\mathit{F}}_P(\tau )\mathit{x}(t)-{\mathit{u}}_E^T(t){\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{u}}_D^T(t){\mathit{R}}_D{\mathit{u}}_D(t)]\\ & +{\mathit{\lambda }}_E^T[\mathit{A}\mathit{x}(t)+\mathit{B}{\mathit{F}}_P(\tau )\mathit{x}(t)-\mathit{B}{\mathit{u}}_E(t)]+{\mathit{\lambda }}_D^T[\mathit{A}\mathit{y}(t)+\mathit{B}{\mathit{u}}_D(t)-\mathit{B}{\mathit{F}}_P(\tau )\mathit{x}(t)]\end{array}$

According to the PMP, the optimal controls of the evader and the defender must minimize the Hamiltonian. The derivatives of Hamiltonian with respect to their controls thus satisfy the following conditions:

(53)$\frac{\partial H}{\partial {\mathit{u}}_E}=-{\mathit{R}}_E{\mathit{u}}_E(t)-{\mathit{B}}^T{\mathit{\lambda }}_E=\mathbf{0}\Rightarrow {\mathit{u}}_E(t)=-{\mathit{R}}_E^{-1}{\mathit{B}}^T{\mathit{\lambda }}_E$

(54)$\frac{\partial H}{\partial {\mathit{u}}_D}=-{\mathit{R}}_D{\mathit{u}}_D(t)+{\mathit{B}}^T{\mathit{\lambda }}_D=\mathbf{0}\Rightarrow {\mathit{u}}_D(t)={\mathit{R}}_D^{-1}{\mathit{B}}^T{\mathit{\lambda }}_D$

The evolution of the co-state vectors ${\mathit{\lambda }}_E$ and ${\mathit{\lambda }}_D$ follows the adjoint equations:

(55)${\dot{\mathit{\lambda }}}_E(t)=-\frac{\partial H}{\partial \mathit{x}(t)}=-{\mathit{F}}_P^T(\tau ){\mathit{R}}_P{\mathit{F}}_P(\tau )\mathit{x}(t)-{[\mathit{A}+\mathit{B}{\mathit{F}}_P(\tau )]}^T{\mathit{\lambda }}_E+{\mathit{F}}_P^T{\mathit{B}}^T{\mathit{\lambda }}_D$

(56)${\dot{\mathit{\lambda }}}_D(t)=-\frac{\partial H}{\partial \mathit{y}(t)}=-\mathit{A}{\mathit{\lambda }}_D$

Substituting Equations (55) and (56) into the equations of the state (32), the following equation can be derived:

(57)$\left\{\begin{array}{l}\dot{\mathit{x}}(t)=[\mathit{A}+\mathit{B}{\mathit{F}}_P(\tau )]\mathit{x}(t)+\mathit{B}{\mathit{R}}_E^{-1}\mathit{B}{\mathit{\lambda }}_E\\ \dot{\mathit{y}}(t)=\mathit{A}\mathit{y}(t)-\mathit{B}{\mathit{F}}_P(\tau )\mathit{x}(t)+\mathit{B}{\mathit{R}}_D^{-1}\mathit{B}{\mathit{\lambda }}_D\end{array}\right.$

The equations of the state and the co-state are further combined as follows:

(58)$\left[\begin{array}{c}\dot{\mathit{x}}\\ \dot{\mathit{y}}\end{array}\right]=\left[\begin{array}{cc}\mathit{A}+\mathit{B}{\mathit{F}}_P(\tau )& \mathbf{0}\\ -\mathit{B}{\mathit{F}}_P(\tau )& \mathit{A}\end{array}\right]\left[\begin{array}{c}\mathit{x}\\ \mathit{y}\end{array}\right]+\left[\begin{array}{cc}\mathit{B}{\mathit{R}}_E^{-1}\mathit{B}& \mathbf{0}\\ \mathbf{0}& \mathit{B}{\mathit{R}}_D^{-1}\mathit{B}\end{array}\right]\left[\begin{array}{c}{\mathit{\lambda }}_E\\ {\mathit{\lambda }}_D\end{array}\right]$

(59)$\left[\begin{array}{c}{\dot{\mathit{\lambda }}}_E\\ {\dot{\mathit{\lambda }}}_D\end{array}\right]=\left[\begin{array}{cc}-{\mathit{F}}_E^T(\tau ){\mathit{R}}_E{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right]\left[\begin{array}{c}\mathit{y}\\ \mathit{z}\end{array}\right]+\left[\begin{array}{cc}-{\mathit{A}}^T-{\mathit{F}}_P^T(\tau ){\mathit{B}}^T& {\mathit{F}}_P^T(\tau ){\mathit{B}}^T\\ \mathbf{0}& -{\mathit{A}}^T\end{array}\right]\left[\begin{array}{c}{\mathit{\lambda }}_E\\ {\mathit{\lambda }}_D\end{array}\right]$

Performing a similar procedure in Equations (45)–(49), the following matrices can be obtained to form a matrix Riccati differential equation with the same form as that of Equation (48).

(60)$$\begin{gathered} \mathit{M}=\left[\begin{array}{cc}\mathit{A}+\mathit{B}{\mathit{F}}_P(\tau )& \mathbf{0}\\ -\mathit{B}{\mathit{F}}_P(\tau )& \mathit{A}\end{array}\right],\mathit{V}=\left[\begin{array}{cc}-{\mathit{F}}_E^T(\tau ){\mathit{R}}_E{\mathit{F}}_E(\tau )& \mathbf{0}\\ \mathbf{0}& \mathbf{0}\end{array}\right] \\ \mathit{N}=\left[\begin{array}{cc}\mathit{B}{\mathit{R}}_E^{-1}\mathit{B}& \mathbf{0}\\ \mathbf{0}& \mathit{B}{\mathit{R}}_D^{-1}\mathit{B}\end{array}\right],\mathit{W}=\left[\begin{array}{cc}-{\mathit{A}}^T-{\mathit{F}}_P^T(\tau ){\mathit{B}}^T& {\mathit{F}}_P^T(\tau ){\mathit{B}}^T\\ \mathbf{0}& -{\mathit{A}}^T\end{array}\right] \end{gathered}$$

After solving the matrix Riccati differential Equation (48), the optimal control laws of the evader and the defender are as follows:

(61)${\mathit{u}}_E(t)=-{\mathit{R}}_E^{-1}{\mathit{B}}^T{\mathit{S}}_E\mathit{P}(t)\mathit{X}(t)$

(62)${\mathit{u}}_D(t)={\mathit{R}}_D^{-1}{\mathit{B}}^T{\mathit{S}}_D\mathit{P}(t)\mathit{X}(t)$

where ${\mathit{S}}_E=[{\mathbf{I}}_3,{\mathbf{0}}_3]$ and ${\mathit{S}}_D=[{\mathbf{0}}_3,{\mathbf{I}}_3]$.

Consequently, Equations (61) and (62) constitute the optimal cooperative evasion–defense strategy for a coordinated evader–defender pair.

4. Simulation Results

To verify the performance of the proposed strategies, a three-player orbital pursuit–evasion–defense game with distinct strategies is simulated and compared in this section. Initially, the evader is moving in a circular orbit with a height of 400 km. To establish the LVLH coordinate frame, the initial orbit of the evader is chosen as the reference orbit and the point coinciding with the initial position of the evader is selected as the reference moving point. The initial states of the three players are chosen as ${\mathit{x}}_P$ = [−6 km, −16 km, 4 km, −9 m/s, 13.6 m/s, 0 m/s], ${\mathit{x}}_E$ = [0 km, 0 km, 0 km, 0 m/s, 0 m/s, 0 m/s], and ${\mathit{x}}_D$ = [−1 km, 3 km, 0 km, 0 m/s, 0 m/s, 0 m/s] under the LVLH frame. The permitted terminal miss distance in the game is 100 m. The payoff weightings of the game are tuned as ${\mathit{Q}}_f=2\times {10}^4$, ${\mathit{R}}_P=0.8\times {10}^{10}$, ${\mathit{R}}_E=3.6\times {10}^{10}$, and ${\mathit{R}}_{\mathrm{D}}=0.2\times {10}^{10}$. The above case (case 1 in Table 1) will be a typical representative which is simultaneously used in Section 4.1, Section 4.2, Section 4.3 for a vertical comparison. Some other cases with different initial positions are also provided in Table 1. (In case 2, the pursuer flies around the evader and the defender hovers; in case 3, the defender flies around the evader and the pursuer hovers.)

4.1. Results of the LQDA Strategy

First, we examine the LQDA strategy in the two-player pursuit–evasion game (On the basis of case 1, the defender is temporarily overlooked). As illustrated in Figure 4a, the pursuer can successfully intercept the evader. Compared with the results of the classic LQ strategy in Figure 4b, the LQDA strategy has higher interception accuracy. Then, a defender using the LQDA strategy is added to the case (namely, becoming case 1), the game results are illustrated in Figure 4c. It can be observed that the defender intercepts the pursuer before the pursuer reaches the evader. Compared with Figure 4a, the appearance of an unexpected defender dramatically affects the ending of the game, signifying the prominence of the defender to the evader. But if the defender uses the classic LQ strategy, the evader will lose the game, as shown in Figure 4d. Figure 4e,f further depict the control histories of the three players when the LQDA and LQ strategy are employed by the defender in case 1, respectively. Cases 2 and 3 further validate the significance of the defender. In these two cases, the pursuer is intercepted by the defender after 262 s and 242 s, respectively.

Figure 5 demonstrates the relationship between the miss distance and the time-to-go under different settings of $q_f$. A turning point at around 100 m can be observed, before which the curve drops sharply and after which the curve descends gradually. (If the miss distance is 100 m, the interception time will be 560 s. But if the miss distance is 10 m, the interception time will be 1018 s). This suggests that when the miss distance is smaller than a certain value, the required interception time will greatly increase with the decrease in the miss distance.

4.2. Results of the Two-Sided Optimal Pursuit Strategy

When the control law of the pursuer has no components against the defender, the defender can break the two-player equilibrium between the pursuer and the evader. Thus, if the pursuer can pursue the evader and avoid the defender simultaneously, the ending of the game may be in favor of the pursuer.

In this case, the parameters of the game are the same as those in Section 4.1, but the pursuer is employing the two-sided optimal pursuit strategy, which aims to chase the evader and dodge the defender simultaneously. Figure 6a depicts the trajectories of the game. Compared to the results of the LQDA strategy in Figure 4a, the pursuer shown in Figure 6a succeeds in avoiding the interception from the defender, leading to a sharp transition in the trajectory of the defender. As seen from the locally enlarged view in Figure 6a, the pursuer successfully intercepts the evader in the end.

The control histories are illustrated in Figure 6b. Several jumps can be observed in the three players’ controls, accompanied by the jumps of the time-to-go. This is because the control gain in optimal feedback strategy is a function of the time-to-go. To further demonstrate the self-defense ability of the pursuer under the two-sided optimal pursuit strategy, the defender is put in another initial position which is closer to the pursuer. Results in Figure 7a reflect that the pursuer can still safely complete the interception.

The results of the LQDA strategy and the two-sided optimal pursuit strategy under different ${\mathit{R}}_P$ settings are compared in Figure 7b. Given a lower ${\mathit{R}}_P$ (the importance of the energy consumption decreases for the pursuer), the pursuer still cannot succeed under the LQDA strategy, but the results under the two-sided optimal pursuit strategy are different from this. Through these simulations, it can be concluded that it is indeed crucial for the pursuer to be both offensive and defensive in such a PED game, and the designed two-sided optimal pursuit strategy can help the pursuer to achieve high self-defense performance in an offensive task.

4.3. Results of the Cooperative Evasion–Defense Strategy

For the evader and the defender, the performance of the LQDA strategy also possesses some drawbacks owing to ignorance exhibited during the information sharing and the potential cooperation between the two players. Under the LQDA strategy, the defender is solely focusing on intercepting the pursuer but is not concerned with indirectly increasing the distance between the pursuer and the evader. However, the evader does not need to maintain a considerable distance from the pursuer. The optimum scenario would be to maintain a distance between the pursuer and the evader larger than that between the pursuer and the defender.

An example is given in Figure 8, where the pursuer under the LQDA strategy succeeds in intercepting the evader before being intercepted by the defender. In this case, although the ${\mathit{R}}_E$ is smaller than that in Section 4.1 (the importance of the energy consumption decreases for the evader), the evader is still unable to get rid of the pursuer’s interception. However, if the evader and the defender adopt the cooperative evasion–defense strategy, the game ending will be quite different, as illustrated in Figure 8b. In the cooperation with the defender, the evader behaves like bait and flies to the defender to reduce the distance between the defender and the pursuer. Eventually, the defender achieves a head-on interception to the pursuer.

In Figure 9, the game results are depicted to demonstrate the cooperation effect between the evader and the defender when the evader or the defender has no control. In Figure 9a, the defender is a free-flying spacecraft with no external control force, and the evader is employing the proposed cooperation strategy. As the game goes on, the evader gradually flies to the defender. Similarly, when the evader has no control (see Figure 9b), the defender actively flies to the evader and intercepts the pursuer. From this case, it can be concluded that the proposed cooperative evasion–defense strategy can help the evader and the defender build cooperation and achieve a better ending in the orbital PED game.

5. Conclusions

This paper investigates the orbital pursuit–evasion–defense (PED) game that involves three spacecraft players. First, a linear-quadratic duration-adaptive (LQDA) guidance law is proposed for each player. In contrast to the LQ strategy, which requires a fixed known game duration, the LQDA strategy can adaptively determine the game duration based on the real-time miss distance, thereby assuring terminal interception. Numerical analysis suggests that if the permitted miss distance is smaller than a certain threshold, the required time cost to achieve the interception will significantly increase with the decrease in the permitted miss distance. Secondly, a two-sided optimal pursuit strategy is designed for the pursuer. Compared to the LQDA strategy, the designed pursuit strategy can notably improve the self-defense ability of the pursuer. Furthermore, a cooperative evasion–defense strategy for the evader and defender is presented through this research. Simulations proved that the cooperative strategy can help the evader and the defender build mutual coordination and achieve a better ending in the orbital PED game. A limitation of this study is the lack of control constraints. Saturation constraints for the control magnitude are commonly seen in the literature. Adding saturation constraints may prolong the interception time in the two-player pursuit–evasion game compared with that case without constraints, or it may alter the game ending in the three-player pursuit–evasion–defense game when different control magnitudes are given to the three players. However, it will not change the effectiveness of the proposed guidance strategies. The basic conclusions still hold with the control constraints considered. Future work may consider the mass change and saturation constraints for the control magnitude for more realistic scenarios.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.

Conflicts of Interest

The author declares that there are no conflicts of interest in this work.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Table

Figure 1. Illustration of the PED game in the LVLH frame.

Figure 2. Illustration of a fluctuating [Forumla omitted. See PDF.].

Figure 3. A critical point for the time-to-go.

Figure 4. Results of the game using the LQDA strategy. (a) The pursuer uses the LQDA strategy in the PE game; (b) pursuer uses the LQ strategy in the PE game; (c) the defender uses the LQDA strategy in the PED game; (d) the defender uses the LQ strategy in the PED game; (e) the controls of the defender using the LQDA strategy; (f) the ontrols of the defender using the LQ strategy.

Figure 5. m vs. t with different qf.

Figure 6. Results of the game under the two-sided optimal pursuit strategy for the pursuer. (a) The game trajectories; (b) the controls and time-to-go.

Figure 7. Results of the game under the two-sided optimal pursuit strategy for the pursuer. (a) The trajectories when the defender has a more favorable initial position; (b) the results of LQDA and two-sided optimal pursuit (TSOP) strategy under different parameters [Forumla omitted. See PDF.].

Figure 8. Results of the game when [Forumla omitted. See PDF.]. (a) the trajectories under the LQDA strategy; (b) the trajectories under the cooperation strategy.

Figure 9. Results of the game when the evader or the defender has no controls. (a) The game trajectories when [Forumla omitted. See PDF.]; (b) the game trajectories when [Forumla omitted. See PDF.].

Table 1

Initial states of the players in the game.

Case	Height	Player	x (km)	y (km)	z (km)	$\dot{\mathit{x}}(\mathbf{m}/\mathbf{s})$	$\dot{\mathit{y}}(\mathbf{m}/\mathbf{s})$	$\dot{\mathit{z}}(\mathbf{m}/\mathbf{s})$
1	400	Evader	0	0	0	0	0	0
		Pursuer	−6	−16	4	−9	13.6	0
		Defender	−1	3	0	0	0	0
2	500	Evader	0	0	0	0	0	0
		Pursuer	−6	−16	4	−9	13.6	0
		Defender	0	−5	0	0	0	0
3	600	Evader	0	0	0	0	0	0
		Pursuer	0	10	0	0	−2	0
		Defender	5	1	0	0.54	−10.8	0

References

1. Luo, Y.Z.; Li, Z.Y.; Zhu, H. Survey on spacecraft orbital pursuit-evasion differential games. Sci. Sin. Technol.; 2020; 50, pp. 1533-1545. [DOI: https://dx.doi.org/10.1360/SST-2019-0174]

2. Li, Z.Y.; Zhu, H.; Luo, Y.Z. Orbital inspection game formulation and epsilon-Nash equilibrium solution. J. Spacecraft Rocket.; 2024; 61, pp. 157-172. [DOI: https://dx.doi.org/10.2514/1.A35800]

3. Gong, B.C.; Li, W.D.; Li, S.; Ma, W.; Zheng, L. Angles-only initial relative orbit determination algorithm for noncooperative spacecraft proximity operations. Astrodynamics; 2018; 2, pp. 217-231. [DOI: https://dx.doi.org/10.1007/s42064-018-0022-0]

4. Spendel, D.F. Parameter Study of an Orbital Debris Defender Using Two Teams, Three Player Differential Game Theory; Air Force Institute of Technology: Wright-Patterson Air Force Base, OH, USA, 2018.

5. Pontani, M.; Conway, B.A. Numerical solution of the three-dimensional orbital pursuit-evasion game. J. Guid. Control Dyn.; 2009; 32, pp. 474-487. [DOI: https://dx.doi.org/10.2514/1.37962]

6. Storn, R. Sytem design by constraint adaptation and differential evolution. IEEE Trans. Evolut. Comput.; 1999; 3, pp. 22-34. [DOI: https://dx.doi.org/10.1109/4235.752918]

7. Dong, X.; Zhou, D.X. Learning gradients by a gradient descent algorithm. J. Math. Anal. Appl.; 2008; 341, pp. 1018-1027. [DOI: https://dx.doi.org/10.1016/j.jmaa.2007.10.044]

8. Hafer, W.T.; Reed, H.L.; Turner, J.D.; Pham, K. Sensitivity methods applied to orbital pursuit evasion. J. Guid. Control Dyn.; 2015; 38, pp. 1118-1126. [DOI: https://dx.doi.org/10.2514/1.G000832]

9. Shen, H.L. Casalino, Revisit of the three-dimensional orbital pursuit-evasion game. J. Guid. Control Dyn.; 2018; 41, pp. 1820-1828. [DOI: https://dx.doi.org/10.2514/1.G003127]

10. Li, Z.Y.; Zhu, H.; Yang, Z.; Luo, Y.-Z. Saddle point of orbital pursuit-evasion game under J2-perturbed dynamics. J. Guid. Control Dyn.; 2020; 43, pp. 1733-1739. [DOI: https://dx.doi.org/10.2514/1.G004459]

11. Jagat, A.; Sinclair, A.J. LQ Dynamic Optimization and Differential Games. Proceedings of the AIAA/AAS Astrodynamics Specialist Conference; San Diego, CA, USA, 3–7 January 2014; pp. 1-20.

12. Jagat, A.; Sinclair, A.J. Nonlinear control for spacecraft pursuit-evasion game using the state-dependent Riccati equation method. IEEE Trans. Aerosp. Electron. Syst.; 2017; 53, pp. 3032-3042. [DOI: https://dx.doi.org/10.1109/TAES.2017.2725498]

13. Li, Z.Y.; Zhu, H.; Luo, Y.Z. An escape strategy in orbital pursuit-evasion games with incomplete information. Sci. China Ser. E Technol. Sci.; 2021; 64, pp. 559-570. [DOI: https://dx.doi.org/10.1007/s11431-020-1662-0]

14. Yan, X.D.; Lyu, S. A two-side cooperative interception guidance law for active air defense with a relative time-to-go deviation. Aerosp. Sci. Technol.; 2020; 100, 105787. [DOI: https://dx.doi.org/10.1016/j.ast.2020.105787]

15. Shaferman, V.; Shima, T. Cooperative multiple-model adaptive guidance for an aircraft defending missile. J. Guid. Control Dyn.; 2010; 33, pp. 1801-1813. [DOI: https://dx.doi.org/10.2514/1.49515]

16. Ratnoo, A.; Shima, T. Guidance strategies against defended aerial targets. J. Guid. Control Dyn.; 2012; 35, pp. 1059-1068. [DOI: https://dx.doi.org/10.2514/1.56924]

17. Perelman, A.; Shima, T.; Rusnak, I. Cooperative differential games strategies for active aircraft protection from a homing missile. J. Guid. Control Dyn.; 2011; 34, pp. 761-773. [DOI: https://dx.doi.org/10.2514/1.51611]

18. Prokopov, O.; Shima, T. Linear quadratic optimal cooperative strategies for active aircraft protection. J. Guid. Control Dyn.; 2013; 36, pp. 753-764. [DOI: https://dx.doi.org/10.2514/1.58531]

19. Weiss, M.; Shima, T.; Castaneda, D. Minimum effort intercept and evasion guidance algorithms for active aircraft defense. J. Guid. Control Dyn.; 2016; 39, pp. 2297-2310. [DOI: https://dx.doi.org/10.2514/1.G000558]

20. Shalumov, V. Online launch-time selection using deep learning in a target–missile–defender engagement. J. Guid. Control Dyn.; 2019; 16, pp. 224-236. [DOI: https://dx.doi.org/10.2514/1.I010668]

21. Sun, Q.L.; Zhang, C.F.; Liu, N.; Zhou, W.; Qi, N. Guidance laws for attacking defended target. Chin. J. Aeronaut.; 2019; 32, pp. 2337-2353. [DOI: https://dx.doi.org/10.1016/j.cja.2019.05.011]

22. Qi, N.M.; Sun, Q.L.; Zhao, J. Evasion and pursuit guidance law against defended target. Chin. J. Aeronaut.; 2017; 30, pp. 1958-1973. [DOI: https://dx.doi.org/10.1016/j.cja.2017.06.015]

23. Shalumov, V. Cooperative online guide-launch-guide policy in a target-missile-defender engagement using deep reinforcement learning. Aerosp. Sci. Technol.; 2020; 104, 105996. [DOI: https://dx.doi.org/10.1016/j.ast.2020.105996]

24. Rubinsky, S.; Gutman, S. Three player pursuit and evasion conflict. J. Guid. Control Dyn.; 2014; 37, pp. 98-110. [DOI: https://dx.doi.org/10.2514/1.61832]

25. Liu, Y.F.; Li, R.F.; Wang, S.Q. Orbital three-player differential game using semi-direct collocation with nonlinear programming. Proceedings of the 2nd International Conference on Control Science and Systems Engineering; Singapore, 27–29 July 2016; 27–29 July pp. 217-222.

26. Liu, Y.F.; Li, R.F.; Hu, L.; Cai, Z.-Q. Optimal solution to orbital three-player defense problems using impulsive transfer. Soft Comput.; 2018; 22, pp. 2921-2934. [DOI: https://dx.doi.org/10.1007/s00500-017-2545-3]

27. Liang, H.Z.; Wang, J.Y.; Liu, J.Q.; Liu, P. Guidance strategies for interceptor against active defense spacecraft in two-on-two engagement. Aerosp. Sci. Technol.; 2020; 96, 105529. [DOI: https://dx.doi.org/10.1016/j.ast.2019.105529]

28. Clohessy, W.H.; Wiltshire, R.S. Terminal guidance system for satellite rendezvous. J. Aerosp. Sci.; 1960; 27, pp. 653-658. [DOI: https://dx.doi.org/10.2514/8.8704]

29. Engwerda, J. LQ Dynamic Optimization and Differential Games; John Wiley & Sons, Ltd.: Chichester, UK, 2005.

30. Xia, X.; Xu, Z. Effective finite horizon linear-quadratic continuous terminal control. Int. J. Comput. Method; 2013; 10, 1350038. [DOI: https://dx.doi.org/10.1142/S0219876213500382]

31. Rubinsky, S.; Gutman, S. Vector guidance approach to three-player conflict in exoatmospheric interception. J. Guid. Control Dyn.; 2015; 38, pp. 2270-2285. [DOI: https://dx.doi.org/10.2514/1.G000942]

32. Gutman, S.; Rubinsky, S. Exoatmospheric thrust vector interception via time-to-go analysis. J. Guid. Control Dyn.; 2019; 42, pp. 86-96. [DOI: https://dx.doi.org/10.2514/1.G001268]

33. Ye, D.; Shi, M.M.; Sun, Z.W. Satellite proximate interception vector guidance based on differential games. Chin. J. Aeronaut.; 2018; 31, pp. 1352-1361. [DOI: https://dx.doi.org/10.1016/j.cja.2018.03.012]

34. Behk, R.D.; Ho, Y.C. On a class of linear stochastic differentia1 games. IEEE Trans. Automat. Contr.; 1968; 13, pp. 227-240.

35. Ogata, K. Modern Control Engineering; Pearson Education: New Delhi, India, 2009.

36. Ypma, T.J. Historical development of the Newton–Raphson method. SIAM Rev.; 1995; 37, pp. 531-551. [DOI: https://dx.doi.org/10.1137/1037125]