Fault-tolerant control for nonlinear time-delay systems using neural network observers

Abstract

This article addresses the problem of fault-tolerant control in nonlinear time-delay systems using adaptive dynamic programming. An adaptive neural network observer is developed to estimate unknown dynamics, system states, and actuator faults. This observer is then transformed into an augmented structure for optimal fault-tolerant control problem. The gains of this observer are determined by solving a linear matrix inequality. A new value function index is introduced to account for time-delay states, and control law is derived associated with this novel value function. The Hamilton–Jacobi–Bellman equation for this value function is solved via a critic neural network. Lyapunov functional analysis demonstrates that the closed-loop system remains uniformly ultimately bounded. Simulation results validate the proposed fault tolerant approach. The key contribution of this paper lies in incorporating time-delay states into the adaptive dynamic programming value function in the presence of actuator faults.

Introduction

In contemporary industrial applications and technological systems, control approaches must consider system safety and reliability. This is especially critical in safety-focused applications, where mission success and the protection of lives and the environment are essential. Consequently, the importance of guaranteeing system safety and reliability has reached unprecedented levels [1, 2]. Since modern systems consist of many components, such as sensors and mechanical actuators, then they may fail over time. Failures can lead to system instability and degrade control performance in systems. Actuator failures, in particular, are especially critical because they can significantly impair control performance due to their unpredictable actions. To address this issue in controller design, fault-tolerant control (FTC) strategies should be implemented [3, 4].

The primary goal of a FTC approach is to adjust the controller structure so the system can manage component failures without compromising the mission or endangering users and the public. This often requires incorporating dynamic fault estimation into the nominal control input, which presents significant challenges in ensuring system stability with the modified control input [5].

FTC methods are generally divided into two categories: passive and active. Passive FTC methods rely on robust control theory, maintaining a constant controller structure and tolerating only a limited number of predefined conditions and assumptions. While passive approaches are more conservative, they are easier to design and implement. In contrast, active FTC methods adapt to faults by re-configuring the controller based on fault detection and isolation information. These approaches are less conservative and impose fewer conditions on the system, but proving stability and achieving acceptable control performance with active methods is more challenging. Great works on both active and passive fault approaches have been published; the reader can refer to [6, 7–8] for more details.

This paper introduces an active FTC approach for nonlinear time-delay systems using adaptive dynamic programming (ADP). Various methods for addressing active FTC problems have been published, including adaptive control approaches [9, 10–11], model predictive control [12, 13], and reinforcement learning with ADP algorithms [14, 15]. Among these various techniques, ADP algorithm is particularly effective for FTC challenges in nonlinear systems. An ADP approach is a powerful method for addressing optimization and optimal control problems by utilizing the principle of optimality [16]. For example, in work [17], a novel cost function is developed to address challenges in solving optimal tracking control problems for certain nonlinear systems with known dynamics using ADP. This innovative approach introduces a specific cost function related to tracking errors and their derivatives, eliminating the need for the traditional zero equilibrium assumption and simplifying the controller design process for nonlinear systems with nonzero equilibrium. ADP, an intelligent control method, has gained significant interest in both academia and industry since its proposal [18].

In [19], the authors proposed an online fault compensation control strategy using ADP algorithm to achieve optimal control in nonlinear systems with actuator faults. This method notably avoids using an action neural network. Expanding on this, [20] introduces an observer to enhance the value function. Unlike the approach in [19], the improved value function in [20] simultaneously addresses both faults and control, providing a more integrated solution.

Building on the findings from [19] and [20], the authors in [21] developed an FTC approach that tackles both actuator and sensor faults using neuro-dynamic programming. In [22], an optimal FTC method for a class of unknown nonlinear discrete-time systems with actuator faults was introduced, employing adaptive critic design. The study in [23] utilized a sliding-mode control technique integrated with a ADP algorithm to solve the optimal FTC problem for affine nonlinear systems. However, the works in [21, 22, 14, 19, 20], and [23] did not address nonlinear systems with time delays.

Consequently, a primary motivation of our work is to develop a fault tolerant method using ADP algorithm for nonlinear systems with time delays. Nearly all the previously mentioned works depend on information from nominal systems and assume the availability of system states. Observers, particularly learning observers, are crucial and play an important role in designing control approaches. Then, another motivation is to develop a fault-tolerant method using the ADP algorithm for nonlinear time-delay systems, based on a neural network observer.

In [24], a fault identification approach for partially unknown nonlinear systems is proposed using a combination of the deterministic learning method and the adaptive high gain observer. However, this paper does not consider time delay or adaptive dynamic programming. Building on these results, [25] proposes an adaptive neural observer-based FTC approach for a class of nonlinear systems using ADP. Most works utilizing neural network observers focus on fault estimation and detection, and methods for neural network observer-based FTC are scarce.

In all the aforementioned works, nonlinear time-delay systems were not considered. Time delays are critical in many engineering systems. Industrial processes, both linear and nonlinear, often involve time delays. In control terminology, these delays are categorized as constant time delays and time-varying delays. Constant time delays, common in slow-changing processes like chemical reactors, represent the maximum tolerable delay. Time-varying delays, typical in fast-paced systems like power systems and mobile robot teams, can cause instability or reduced performance. For example, in distributed optimization problems, delays can increase the time needed for systems to converge to an optimal solution [26]. To mitigate these effects, researchers incorporate delay considerations to ensure the stability of their controllers [27]. A widely used approach for this purpose is based on Lyapunov-Krasovskii methods [28].

Addressing time-delay systems in ADP is particularly challenging due to their infinite-dimensional nature, which complicates their solution. Analyzing systems with time delays is significantly more complex than analyzing systems without delays. Currently, there is no specific method for handling nonlinear time-delay systems. For more information, see chapters 4 and 5 of [29].

Based on the discussions and to the best of our knowledge, the issue of FTC based on neural network observer for nonlinear time-delay systems using an ADP algorithm has not been previously studied. The main contributions of this paper are as follows:

• An adaptive neural network observer for nonlinear time-delay systems is designed to learn the unknown system dynamics and actuator faults. This observer is integrated into a FTC approach, enhancing its adaptability to stochastic faults through active intervention.

• An novel value function for neural network observer-based FTC problem using ADP algorithm: Unlike existing ADP algorithms for fault tolerant in nonlinear systems, this paper specifically targets nonlinear time-delay systems. We introduce a new value function for our ADP algorithm, which integrates state time delays into the dynamics of nonlinear systems. Compared to [19, 24, 25] and [20], our method includes time delays in the control law updates and considers their impact to ensure the stability of the proposed approach.

• Stability analysis using bilinear matrix inequality (BMI): We introduce a BMI to consider the impact of time delay on the stability of the proposed method. This BMI helps analyze the effect of constant time delay on the stability of the FTC problem based on the ADP algorithm for nonlinear time-delay systems.

The remainder of this paper is structured as follows. Section 2 presents the problem statement and outlines the control objectives. It also discusses the construction of a neural network observer, the development of a performance index function, and the creation of a fault-tolerant approach using adaptive techniques to address the FTC problem for a class of partially unknown nonlinear time-delay systems. Section 3 provides simulation examples to demonstrate the effectiveness of the proposed scheme. Finally, Sect. 4 concludes the paper.

Notation: The symbols $\mathbb{R}$, ${\mathbb{R}}^n$, and ${\mathbb{R}}^{n\times m}$ denote the sets of all real numbers, n-dimensional real vectors, and $n\times m$ real matrices, respectively. The notation $∥.∥$ represents the Euclidean vector norm. The functions ${\overline{\lambda }}_{min}(.)$ and ${\overline{\lambda }}_{max}(.)$ indicate the minimum and maximum eigenvalues, respectively.

Mathematical formulation

Consider a class of partially unknown nonlinear time-delay systems represented as follows:

1

$$\begin{array}{cc}& \dot{z}(t)=\mathcal{A}z(t)+{\mathcal{A}}_{d}z(t-d)+\mathcal{B}u(t)+f(z(t))\hfill \\ \hfill & +{\mathcal{D}}_{f}a(t),\hfill \\ \hfill & y(t)=\mathcal{C}z(t),\hfill \\ \hfill & z(t)=h(t),t\in [-d,0],\hfill \end{array}$$the variables $z(t)\in {\mathbb{R}}^n$ and $z(t-d)\in {\mathbb{R}}^n$ denote the state vectors of the system at time $t$ and at a delayed time $t-d$, respectively. The vector $y(t)\in {\mathbb{R}}^p$ represents the measured output, while $u(t)\in {\mathbb{R}}^m$ signifies the control input vector. The term $a(t)\in {\mathbb{R}}^j$ denotes the unknown actuator fault. The matrices $\mathcal{A}\in {\mathbb{R}}^{n\times n}$, $\mathcal{B}\in {\mathbb{R}}^{n\times m}$, $\mathcal{C}\in {\mathbb{R}}^{p\times n}$, and ${\mathcal{D}}_f\in {\mathbb{R}}^{n\times j}$ are known real and constant. The term h(t) represents the historical state of the system and d is time-delay constant. The pair $(\mathcal{A},\mathcal{B})$ is controllable, and the pair $(\mathcal{A},\mathcal{C})$ is observable. The function $f(z(t))\in {\mathbb{R}}^n$ is an unknown nonlinear function with Lipschitz continuity.

The following assumption must be defined to proceed with the proposed method in this paper.

Assumption 1

The partial derivative of the approximation error in the critic neural network, denoted as $\mathrm{\nabla }{ϵ}_c(x)$, the actuator fault $a(t)$, and the function $f(z(t))$ are all constrained by their respective norms. Specifically, the norm of the gradient $\mathrm{\nabla }{ϵ}_c(z(t))$ is bounded by $\mathrm{\nabla }{\overline{ϵ}}_c$, the norm of the function $f(z(t))$ is bounded by $f_1$, and the norm of the actuator fault $a(t)$ is bounded by $f_2$. Here, $\mathrm{\nabla }{\overline{ϵ}}_c$, $f_1$, and $f_2$ are positive constants.

In Eq. (1), the dynamics $f(z(t))$ and the fault $a(t)$ are unknown. To handle these unknown nonlinear terms and the actuator fault, we will use a critic neural network approximation.

Radial basis function neural networks

Radial basis function (RBF) networks have the capability to approximate any function universally. This approximation can be expressed as ${\mathcal{F}}^T(\mathcal{Z})\mathcal{W}$, where $\mathcal{F}(\mathcal{Z})$ represents a vector of RBF neurons, $\mathcal{W}$ denotes a vector of network weights, and $\mathcal{Z}$ is the input vector. Let $\psi$ be a trajectory confined within a bounded set, and let ${\mathcal{C}}_{\zeta }$ denote the localized area along this trajectory $\psi$. According to [30], by leveraging the neurons within ${\mathcal{C}}_{\zeta }$, for any smooth nonlinear function $\varphi (\mathcal{Z})$ and for any given tolerance $ϵ>0$, there exists an optimal weight vector ${\mathcal{W}}_{\zeta }^{\ast }$ such that:

2

$$\begin{array}{c}\hfill \varphi (\mathcal{Z})={\mathcal{F}}_{\zeta }^T(\mathcal{Z}){\mathcal{W}}_{\zeta }^{\ast }-ϵ,\mathcal{Z}\in \psi \end{array}$$where ${\mathcal{F}}_{\zeta }(\mathcal{Z})$ denotes a vector of RBF neurons within ${\mathcal{C}}_{\zeta }$.

Control objective

The aim of this paper is to develop a control strategy using neural network observers and the ADP algorithm for system (1). This strategy is designed to ensure stability despite actuator faults and time delays. The neural network observer allows for the estimation of system states and unknown dynamics. We consider the unknown nonlinear function $f(z(t))$ and the fault $a(t)$ as a combined term, expressed as $\phi (z)=f(z(t))+{\mathcal{D}}_{f}a(t)$, and assume the detectable condition is met. The control laws are formulated to incorporate the influence of time delays, guaranteeing system stability. In particular, the control law accounts for the integral of the time-delay history. We propose a new value function that includes the history of time delays.

Adaptive neural network observer

Neural network observers excel in handling nonlinear dynamics, a key advantage over traditional linear observers. While linear observers are limited to small ranges around specific operating points, neural networks can approximate complex, nonlinear functions. This allows them to learn nonlinear dynamics from data, resulting in more accurate state estimation across a wider range of operating conditions. Such capability is particularly valuable in fault-tolerant control problems, where the ability to detect and isolate faults in nonlinear systems is critical for maintaining system stability and performance [24]. In [31], fuzzy logic systems are employed to model the unknown nonlinear dynamics of the pure-feedback system. These systems are particularly well-suited for this task because they can represent complex, nonlinear relationships through a set of simple "if-then" rules derived from expert knowledge or data. However, while fuzzy logic systems are flexible, their rule base can be relatively static unless specifically designed to adapt. This static nature can limit their ability to learn complex, high-dimensional patterns compared to neural networks, which can automatically learn representations from data.

Based on (2), an adaptive neural network observer for system (1) is designed as follows:

3

$$\begin{array}{cc}& \dot{\widehat{z}}(t)=\mathcal{A}\widehat{z}(t)+{\mathcal{A}}_d\widehat{z}(t-d)+\mathcal{B}u(t)+{\mathcal{F}}^T(y){\widehat{\mathcal{W}}}_o\hfill \\ \hfill & +\mathrm{\Lambda }(y(t)-\mathcal{C}\widehat{z}(t)),\hfill \\ \hfill & \widehat{y}(t)=\mathcal{C}\widehat{z}(t),\hfill \end{array}$$and the neural network identifier as

4

$$\begin{array}{c}\hfill {\dot{\widehat{\mathcal{W}}}}_o(t)=\mathrm{\Omega }\mathcal{F}(y(t))\mathcal{L}(y(t)-\mathcal{C}\widehat{z}(t))-\delta \mathrm{\Omega }{\widehat{\mathcal{W}}}_o(t),\end{array}$$in which $\widehat{z}(t)\in {\mathbb{R}}^n$ and $\widehat{y}(t)\in {\mathbb{R}}^n$ represent the estimates of the system state z(t) and the output measurements y(t), respectively. The observer gain matrices are $\mathrm{\Lambda }\in {\mathbb{R}}^{n\times p}$ and $\mathcal{L}\in {\mathbb{R}}^{j\times p}$. The parameters $\delta >0$ and $\mathrm{\Omega }\in {\mathbb{R}}^{j\times j}>0$ are defined. The term ${\mathcal{F}}^T(y(t))\in {\mathbb{R}}^{n\times k}$, and $\widehat{\mathcal{W}}\in {\mathbb{R}}^k$ is the estimate of $\mathcal{W}$. The RBF neural network ${\mathcal{F}}^T(y(t))\widehat{\mathcal{W}}$ is used to approximate the unknown function $\phi (z(t))$.

Remark 1

Existing NN-based control methods rely on adaptive techniques, with NN weight update laws designed using a Lyapunov function. However, this approach only guarantees convergence of NN weights, while other NN parameters like widths and centers remain fixed and cannot be updated online. In contrast, gradient descent (GD)-based NN techniques can achieve better approximation performance by updating all NN parameters to satisfactory values through cost function minimization via the GD algorithm. In [32], a novel control strategy for nonlinear systems with unknown dynamics is introduced, utilizing a neural network-based approach that combines gradient descent (GD) with the Barzilai-Borwein (BB) method. This GD-BB-based control algorithm enhances adaptability by updating neural network parameters and learning rates online, simplifying the design process and reducing the need for manual tuning. Thus, using the proposed method in [32] for estimating unknown nonlinear dynamics could be a very interesting approach for fault-tolerant control problems.

For simplicity, the notation t is omitted in the equations that follow.

Let’s define the state observation error as $\tilde{z}=\widehat{z}-z$ and the neural network identifier observation error as $\tilde{\mathcal{W}}=\widehat{\mathcal{W}}-\mathcal{W}$. Then, based on Eqs. (1) and (3), we obtain the following.

5

$$\begin{array}{c}\hfill \dot{\tilde{z}}=(\mathcal{A}-\mathrm{\Lambda }\mathcal{C})\tilde{z}+{\mathcal{A}}_d\tilde{z}(t-d)+{\mathcal{F}}^T(y){\tilde{\mathcal{W}}}_o+\epsilon .\end{array}$$The following theorem is presented to show that the state observation error (5) is uniformly ultimately bounded.

Theorem 1

Given the provided observer (3) and Assumption 1, if there exist symmetric positive definite matrices E and ${\mathcal{Q}}_d\in {\mathbb{R}}^{n\times n}$, positive scalar ${\lambda }_1$, and matrix $\mathcal{Y}$ such that the following LMIs hold:

6

$$\begin{array}{c}\hfill \mathrm{\Pi }=\left[\begin{array}{ccc}-{\mathrm{\Pi }}_{11}\hfill & -E{\mathcal{A}}_d^T& 0\\ \ast \hfill & {\mathcal{Q}}_d& 0\\ \ast \hfill & \ast & \delta \end{array}\right]>0,\end{array}$$where$$\begin{array}{cc}& {\lambda }_3=\delta {\overline{\mathcal{W}}}^T\overline{\mathcal{W}},\hfill \\ \hfill & {\mathrm{\Pi }}_{11}={\mathcal{A}}^{T}E-{\mathcal{C}}^T{\mathcal{Y}}^T+E\mathcal{A}-\mathcal{Y}\mathcal{C}+\frac{1}{{\lambda }_1}E{E}^T+{\mathcal{Q}}_d,\hfill \end{array}$$the observer gain $\mathrm{\Lambda }$ and $\mathcal{L}$ are obtained as $\mathrm{\Lambda }=E^{-1}\mathcal{Y}$ and $\mathcal{L}=E^T{\mathcal{C}}^{-1}$. Consequently, the state estimation error $\tilde{z}$ and the identifier estimation error $\tilde{\mathcal{W}}$ are uniformly ultimately bounded.

Proof

The following Lyapunov function candidate is proposed:

7

$$\begin{array}{c}\hfill \sum ={\sum }_1+{\sum }_2+{\sum }_3,\end{array}$$where

8

$$\begin{array}{cc}& {\sum }_1={\tilde{z}}^{T}E\tilde{z},\hfill \\ \hfill & {\sum }_2={\int }_{t-d}^t{\tilde{z}}^T(\tau )E\tilde{z}(\tau )d\tau \hfill \\ \hfill & {\sum }_3={\tilde{\mathcal{W}}}^T{\mathrm{\Omega }}^{-1}\tilde{\mathcal{W}}.\hfill \end{array}$$Given (5) and (7), we derive the following:

9

$$\begin{array}{cc}\hfill {\dot{\sum }}_1& ={\dot{\tilde{z}}}^{T}E\tilde{z}+{\tilde{z}}^{T}E\dot{\tilde{z}}\hfill \\ \hfill & ={\tilde{z}}^T({(\mathcal{A}-\mathrm{\Lambda }\mathcal{C})}^{T}E+E(\mathcal{A}-\mathrm{\Lambda }\mathcal{C}))\tilde{z}\hfill \\ \hfill & +{\tilde{z}}^T(t-d){\mathcal{A}}_d^{T}E\tilde{z}+2{\tilde{\mathcal{W}}}^T\mathcal{F}(y)E\tilde{z}+2{\tilde{z}}^{T}E\epsilon ,\hfill \end{array}$$

10

$$\begin{array}{cc}\hfill {\dot{\sum }}_2& ={\tilde{z}}^T{\mathcal{Q}}_d\tilde{z}-{\tilde{z}}^T(t-d){\mathcal{Q}}_d\tilde{z}(t-d),\hfill \end{array}$$

11

$$\begin{array}{cc}\hfill {\dot{\sum }}_3& =2{\tilde{\mathcal{W}}}^T{\mathrm{\Omega }}^{-1}\dot{\widehat{\mathcal{W}}}=2{\tilde{\mathcal{W}}}^T{\mathrm{\Omega }}^{-1}(\mathrm{\Omega }{\mathcal{F}}^T(y)\hfill \\ \hfill & \times \mathcal{L}(y-\mathcal{C}\widehat{z})-\delta \mathrm{\Omega }\widehat{\mathcal{W}})\hfill \\ \hfill & =-2{\tilde{\mathcal{W}}}^T{\mathcal{F}}^T(y)\mathcal{L}\tilde{z}-2{\tilde{\mathcal{W}}}^T\delta \widehat{\mathcal{W}},\hfill \end{array}$$In (9) and using Young’s inequality, we obtain the following:

12

$$\begin{array}{c}\hfill 2{\tilde{z}}^{T}E\epsilon \le \frac{1}{{\lambda }_1}{\tilde{z}}^{T}E{E}^T\tilde{z}+{\lambda }_1{\epsilon }^T\epsilon .\end{array}$$First, let us examine the scenario where the neural network approximation error $\epsilon =0$. Then, we derive the following result:

13

$$\begin{array}{cc}& \dot{\sum }={\dot{\sum }}_1+{\dot{\sum }}_2+{\dot{\sum }}_3\le {\tilde{z}}^T({(\mathcal{A}-\mathrm{\Lambda }\mathcal{C})}^{T}E\hfill \\ \hfill & +E(\mathcal{A}-\mathrm{\Lambda }\mathcal{C}))\tilde{z}+{\tilde{z}}^T(t-d){\mathcal{A}}_d^{T}E\tilde{z}\hfill \\ \hfill & +2{\tilde{\mathcal{W}}}^T\mathcal{F}(y)E\tilde{z}+\frac{1}{{\lambda }_1}{\tilde{z}}^{T}E{E}^T\tilde{z}+{\tilde{z}}^T{\mathcal{Q}}_d\tilde{z}\hfill \\ \hfill & -{\tilde{z}}^T(t-d){\mathcal{Q}}_d\tilde{z}(t-d)-2{\tilde{\mathcal{W}}}^T{\mathcal{F}}^T(y)\hfill \\ \hfill & \times \mathcal{L}\mathcal{C}\tilde{z}-2{\tilde{\mathcal{W}}}^T\delta \widehat{\mathcal{W}}.\hfill \end{array}$$Utilizing Young’s inequality and Assumption 1, we derive the following result:

14

$$\begin{array}{cc}& -2{\tilde{\mathcal{W}}}^T\delta \widehat{\mathcal{W}}=-2\delta {\tilde{\mathcal{W}}}^T(\tilde{\mathcal{W}}+\mathcal{W})\hfill \\ \hfill & =-2\delta {\tilde{\mathcal{W}}}^T\tilde{\mathcal{W}}-2\delta {\tilde{\mathcal{W}}}^T\mathcal{W}\le -\delta {\tilde{\mathcal{W}}}^T\tilde{\mathcal{W}}\hfill \\ \hfill & -\delta {\mathcal{W}}^T\mathcal{W}\le -\delta {\tilde{\mathcal{W}}}^T\tilde{\mathcal{W}}-\delta {\overline{\mathcal{W}}}^T\overline{\mathcal{W}},\hfill \end{array}$$here $\overline{\mathcal{W}}$ denotes the upper bound of $\mathcal{W}$, that is $∥\mathcal{W}∥\le ∥\overline{\mathcal{W}}∥$. By assuming $E\mathcal{F}(y)={\mathcal{C}}^T{\mathcal{L}}^T\mathcal{F}(y)$ and defining the variable ${\eta }^T=\left[{\tilde{z}}^T,,,,{\tilde{z}}^T,(t-d),,,,{\tilde{\mathcal{W}}}^T\right]$

15

$$\begin{array}{c}\hfill \dot{\mathrm{\Sigma }}\le -{\eta }^T\mathrm{\Pi }\eta -{\lambda }_3\end{array}$$where ${\lambda }_3=\delta {\overline{\mathcal{W}}}^T\overline{\mathcal{W}}$ and $\mathrm{\Pi }$ is defined in (6). According to (15), if $\mathrm{\Pi }>0$, then, it leads to $\dot{\mathrm{\Sigma }}\le 0$.

In other words, $\tilde{z}$ and $\tilde{\mathcal{W}}$ will remain bounded over time. Thus, the proof is concluded. $\square$

Inspiring [24], Theorem 1 is obtained and the estimation errors for the states converge to zero when $\epsilon \simeq 0$. Thus, the convergence of the state estimate error $z$ and the neural network weight estimate errors $\mathcal{W}$ can be assured. By using the result of Theorem 1, the identification error systems (3) and (4) are uniformly ultimately bounded, $z$ and $\mathcal{W}$ are also bounded.

Designing optimal controller based on neural network observer

From the designed adaptive neural network observer (5), it is clear that system (1) is replaced by (3), thus transforming (1) into:

16

$$\begin{array}{cc}& \dot{\widehat{z}}=\mathcal{A}\widehat{z}+{\mathcal{A}}_d\widehat{z}(t-d)+\mathcal{B}u(\widehat{z})+{\mathcal{F}}^T(y)\widehat{\mathcal{W}}+g(\widehat{z})\hfill \\ \hfill & \widehat{y}=\mathcal{C}\widehat{z},\hfill \end{array}$$where $g(\widehat{z})=\mathrm{\Lambda }(y-\mathcal{C}\widehat{z})$.

If an optimal control policy $u(\widehat{z})$ is found for Eq. (16), it guarantees the convergence of system (1). This paper presents an augmented system to reframe the problem of fault tolerant as an optimal regulation problem. By merging Eqs. (4) and (16), the following observer system is provided as follows:

17

$$\begin{array}{cc}& \left[\begin{array}{c}\dot{\widehat{z}}\\ {\dot{\widehat{\mathcal{W}}}}_o\\ \end{array}\right]=\left[\begin{array}{cc}\mathcal{A}\hfill & {\mathcal{F}(y)}^T\\ 0\hfill & -\delta \mathrm{\Omega }\end{array}\right]\left[\begin{array}{c}\widehat{z}\\ {\widehat{\mathcal{W}}}_o\\ \end{array}\right]+\left[\begin{array}{cc}{\mathcal{A}}_d\hfill & 0\\ 0\hfill & 0\end{array}\right]\times \hfill \\ \hfill & \left[\begin{array}{c}\widehat{z}(t-d)\\ {\widehat{\mathcal{W}}}_o(t-d)\\ \end{array}\right]+\left[\begin{array}{c}\mathcal{B}\\ 0\\ \end{array}\right]u(\widehat{z})+\left[\begin{array}{c}g(\widehat{z})\\ \mathrm{\Omega }\mathcal{F}(y)\mathcal{L}(y-\mathcal{C}\widehat{z})\\ \end{array}\right],\hfill \end{array}$$introduce the augmented variables and matrices as:$$\begin{array}{cc}& \eta =\left[\begin{array}{c}\widehat{z}\\ {\widehat{\mathcal{W}}}_o\\ \end{array}\right],\overline{\mathcal{A}}=\left[\begin{array}{cc}\mathcal{A}\hfill & {\mathcal{F}(y)}^T\\ 0\hfill & -\delta \mathrm{\Omega }\end{array}\right]\hfill \\ & {\overline{\mathcal{A}}}_d=\left[\begin{array}{cc}{\mathcal{A}}_d\hfill & 0\\ 0\hfill & 0\end{array}\right],\overline{\mathcal{B}}=\left[\begin{array}{c}\mathcal{B}\\ 0\\ \end{array}\right],\hfill \\ & \mathcal{G}(\overline{\mathcal{M}}\xi )=\left[\begin{array}{c}g(\widehat{z})\\ \mathrm{\Omega }\mathcal{F}(y)\mathcal{L}(y-\mathcal{C}\widehat{z})\\ \end{array}\right],\overline{\mathcal{M}}=\left[I,,,,,,0\right].\hfill \end{array}$$Then, system (17) can be represented as

18

$$\begin{array}{c}\hfill \dot{\eta }=\overline{\mathcal{A}}\eta +{\overline{\mathcal{A}}}_d\eta (t-d)+\mathcal{B}u(\eta )+\mathcal{G}(\overline{\mathcal{M}}\eta ).\end{array}$$Next, we will derive the control law, assuming that the control input $u(\eta )\in \mathrm{\Psi }(\mathrm{\Omega })$ is considered admissible.

Controller formulation

The proposed control strategy leverages the ADP framework to solve optimal control problems using neural network observer. We employ a critic neural network to approximate the value function associated with the Hamilton–Jacobi–Bellman equation, training it to minimize temporal difference error. This approach, supported by Lyapunov functional analysis, ensures the convergence of the system states and neural network weights, thereby guaranteeing the stability and optimality of the control strategy.

System (1) is controllable on the set $\phi \in {\mathbb{R}}^n$. Therefore, the goal is to find a feedback control policy $u(\eta )$, that minimizes the infinite horizon performance index function, expressed as:

19

$$\begin{array}{cc}\hfill \mathcal{V}(\eta )=& {\int }_t^{\infty }(\mathcal{P}(\eta ,u)+{\int }_{\tau -d}^{\tau }{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq)d\tau ,\hfill \end{array}$$here, the utility function $\mathcal{P}(\eta (\tau ),u(\tau ))$ is given by

$\mathcal{P}(\eta ,u)={\eta }^T\overline{\mathcal{Q}}\eta +u^T\mathcal{R}u$. $\overline{\mathcal{Q}}=\left[\begin{array}{cc}\mathcal{Q}\hfill & 0\\ 0\hfill & 0\end{array}\right],$$\mathcal{Q}\in {\mathbb{R}}^{n\times n},$${\overline{\mathcal{Q}}}_d=\left[\begin{array}{cc}{\mathcal{Q}}_d\hfill & 0\\ 0\hfill & 0\end{array}\right],{\mathcal{Q}}_d\in {\mathbb{R}}^{n\times n}$ and $\mathcal{R}\in {\mathbb{R}}^{m\times m}$ are positive definite matrices.

Consider $U(\eta ,u)$, defined as:$$\begin{array}{c}\hfill U(\eta ,u)=\mathcal{P}(\eta ,u)+{\int }_{\tau -d}^{\tau }{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq.\end{array}$$This function satisfies $U(0,0)=0$ and is nonnegative for all $\eta$ and $u$. The expression (19) takes into account time delays.

Next, we will prove that using the new value function (19) to design a controller can stabilize the closed-loop system.

Let ${\mathcal{V}}^{\ast }(\eta )$ be the optimal value function corresponding to $\mathcal{V}(\eta )$, defined as follows:

20

$$\begin{array}{c}\hfill \mathcal{V}_{}^{\ast }(\eta )=\underset{u\in \phi }{min}\mathcal{V}(\eta ).\end{array}$$By applying the Bellman optimality principle, we can determine the gradient of ${\mathcal{V}}^{\ast }(\eta )$. This gradient, denoted as $\mathrm{\nabla }{\mathcal{V}}^{\ast }(\eta )=\frac{\partial {\mathcal{V}}^{\ast }(\eta )}{\partial \eta }$, satisfies the following equation:

21

$$\begin{array}{c}\hfill \underset{u\in \phi }{min}H(\eta ,\mathrm{\nabla }{\mathcal{V}}^{\ast }(\eta ))=0,\end{array}$$where $H(\eta ,\mathrm{\nabla }{\mathcal{V}}^{\ast }(\eta ))$ is known as the Hamiltonian function [33, 34]. The Hamiltonian function associated with the cost function (19) is given by:

22

$$\begin{array}{cc}\hfill H(\eta ,& \mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ),u)=U(\eta ,u)+(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\dot{\eta }\hfill \\ \hfill & =U(\eta ,u)+(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T(\overline{\mathcal{A}}\widehat{\eta }+{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)\hfill \\ \hfill & +\mathcal{G}(\overline{\mathcal{M}}\eta ))+(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\mathcal{B}u(\widehat{\eta })\hfill \\ \hfill & =\mathcal{P}(\eta ,u)+{\int }_{\tau -d}^{\tau }{\eta }^T(q){\mathcal{Q}}_d\eta (q)dq\hfill \\ \hfill & +(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T(\overline{\mathcal{A}}\widehat{\eta }+{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)\hfill \\ \hfill & +\mathcal{G}(\overline{\mathcal{M}}\eta ))+(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\mathcal{B}u(\widehat{\eta }).\hfill \end{array}$$Our goal is to minimize Eq. (22) by combining Eqs. (22) and (21) to derive the optimal control law, denoted as $u^{\ast }$. This optimal control law can be obtained from $\frac{\partial H(\eta ,\mathrm{\nabla }{\mathcal{V}}^{\ast }(\eta ),u)}{\partial u}=0$ as follows:

23

$$\begin{array}{c}\hfill u_{}^{\ast }=-\frac{1}{2}{\mathcal{R}}^{-1}{\mathcal{B}}^T\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ).\end{array}$$From (23), a simple transformation yields:

24

$$\begin{array}{c}\hfill (\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\mathcal{B}=2u_{}^{\ast }_{}^T\mathcal{R}.\end{array}$$Equation (24) will be needed later.

By substituting Eq. (23) in Eq. (22), the HJB equation can be rewritten as follows:

25

$$\begin{array}{cc}& {\eta }^T\mathcal{Q}\eta -\frac{1}{4}(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T{\mathcal{B}}^T\mathcal{R}\mathcal{B}\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta )\hfill \\ \hfill & +{\int }_{\tau -d}^{\tau }{\eta }^T(q){\mathcal{Q}}_d\eta (q)dq+(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T(\overline{\mathcal{A}}\widehat{\eta }\hfill \\ \hfill & +{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)+\mathcal{G}(\overline{\mathcal{M}}\eta ))=0.\hfill \end{array}$$It is essential to show that the control law derived in Eq. (23) can stabilize the system described by Eq. (1). This is established in the following theorem.

Theorem 2

Consider system (1), Theorem 1, and the control protocol given by Eq. (23). The control strategy outlined in Eq. (23) can ensure that nonlinear time-delay system (1) remains uniformly ultimately bounded, provided that there exist positive definite matrices $\overline{\mathcal{Q}}$ and ${\overline{\mathcal{Q}}}_d$, along with free-weighting matrices $\mathcal{M}$ and $\mathcal{N}$, such that the following BMI is satisfied:

26

$$\begin{array}{c}\hfill \overline{\mathrm{{\rm Y}}}=\left[\begin{array}{cccc}\mathcal{R}& 0& 0& 0\\ \ast & \overline{\mathcal{Q}}-{\mathcal{M}}_d& -\mathcal{M}{\mathcal{N}}_d& -\mathcal{M}\\ \ast & \ast & -{\mathcal{N}}_d& -\mathcal{N}\\ \ast & \ast & \ast & 0\\ \end{array}\right]\ge 0,\end{array}$$in which$$\begin{array}{cc}& {\mathcal{M}}_d=d\mathcal{M}{\overline{\mathcal{Q}}}_d^{-1}{\mathcal{M}}^T,\hfill \\ \hfill & \mathcal{M}{\mathcal{N}}_d=d\mathcal{M}{\overline{\mathcal{Q}}}_d^{-1}{\mathcal{N}}^T,\hfill \\ \hfill & {\mathcal{N}}_d=d\mathcal{N}{\overline{\mathcal{Q}}}_d^{-1}{\mathcal{N}}^T.\hfill \end{array}$$

Proof

The following Lyapunov function is considered:

27

$$\begin{array}{c}\hfill L(\eta )=\mathcal{V}_{}^{\ast }(\eta ).\end{array}$$Based on the definition of $\mathcal{V}_{}^{\ast }(\eta )$, we have $\mathcal{V}_{}^{\ast }(\eta )>0$ when $\eta \ne 0$ and $\mathcal{V}_{}^{\ast }(\eta )=0$ when $\eta =0$. This indicates that $\mathcal{V}_{}^{\ast }(\eta )$ is positive definite, which consequently implies that $L(\eta )$ is also positive definite. Following this, the derivative of the Lyapunov function (27) is evaluated along the trajectory of $\dot{\eta }=\overline{\mathcal{A}}\widehat{\eta }+{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)+\mathcal{B}u(\widehat{\eta })+\mathcal{G}(\overline{\mathcal{M}}\eta )$, yielding the following expression:

28

$$\begin{array}{cc}\hfill \dot{L}(\eta )=& (\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\dot{\eta }\hfill \\ \hfill =& (\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T(\overline{\mathcal{A}}\widehat{\eta }+{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)+\mathcal{B}u(\widehat{\eta })\hfill \\ \hfill & +\mathcal{G}(\overline{\mathcal{M}}\eta )).\hfill \end{array}$$Calling (22), we obtain:

29

$$\begin{array}{cc}& (\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T(\overline{\mathcal{A}}\widehat{\eta }+{\overline{\mathcal{A}}}_d\widehat{\eta }(t-d)+\mathcal{G}(\overline{\mathcal{M}}\eta ))=-\mathcal{P}(\eta ,u)\hfill \\ \hfill & -{\int }_{t-d}^t{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq-(\mathrm{\nabla }\mathcal{V}_{}^{\ast }(\eta ))_{}^T\mathcal{B}u.\hfill \end{array}$$By substituting (29) and (23) in (28), Eq. (28) can be reformulated as follows:

30

$$\begin{array}{c}\hfill \dot{L}(\eta )=-{\eta }^T\overline{\mathcal{Q}}\eta -u^{{\ast }^T}\mathcal{R}u_{}^{\ast }-{\int }_{t-d}^t{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq.\end{array}$$By using Jensen’s inequality [28], let us define the free-weighting matrices $\mathcal{M}$ and $\mathcal{N}$. The term $-{\int }_{t-d}^t{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)$$dq$ in (30) can be expressed as follows:

31

$$\begin{array}{cc}& -{\int }_{t-d}^t{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq=2{\eta }^T\mathcal{M}{\int }_{t-d}^t{\eta }^T(q)dq\hfill \\ \hfill & +2{\eta }^T(t-d)\mathcal{N}{\int }_{t-d}^t{\eta }^T(q)dq-{\int }_{t-d}^t({\eta }^T(q)\mathcal{M}\hfill \\ \hfill & +{\eta }^T(t-d)\mathcal{N}+{\eta }^T(q){\overline{\mathcal{Q}}}_d){\overline{\mathcal{Q}}}_d^{-1}({\eta }^T(q)\mathcal{M}\hfill \\ \hfill & +{\eta }^T(t-d)\mathcal{N}+{\eta }^T(q){\overline{\mathcal{Q}}}_d{)}^{T}dq+{\int }_{t-d}^t({\eta }^T(q)\mathcal{M}\hfill \\ \hfill & +{\eta }^T(t-d)\mathcal{N}){\overline{\mathcal{Q}}}_d^{-1}({\eta }^T(q)\mathcal{M}+{\eta }^T(t-d)\mathcal{N}{)}^{T}dq\hfill \\ \hfill & \le {\xi }^T\mathrm{{\rm Y}}\xi ,\hfill \end{array}$$in which$$\begin{array}{cc}& {\xi }^T=\left[{\eta }^T,,,,,{\eta }^T,(t-d),,,,,{\int }_{t-d}^t,{\eta }^T(q)dq\right],\hfill \\ \hfill & \mathrm{{\rm Y}}=\left[\begin{array}{ccc}{\mathcal{M}}_d& \mathcal{M}{\mathcal{N}}_d& \mathcal{M}\\ \ast & {\mathcal{N}}_d& \mathcal{N}\\ \ast & \ast & 0\\ \end{array}\right].\hfill \end{array}$$Referring to (30) and applying (31), we get:

32

$$\begin{array}{cc}& \dot{L}(\eta )=-{\eta }^T\overline{\mathcal{Q}}\eta -u^{{\ast }^T}\mathcal{R}u_{}^{\ast }-{\int }_{t-d}^t{\eta }^T(q){\overline{\mathcal{Q}}}_d\eta (q)dq\hfill \\ \hfill & \le -{\eta }^T\overline{\mathcal{Q}}\eta -u^{{\ast }^T}\mathcal{R}u_{}^{\ast }+{\eta }^T\mathrm{{\rm Y}}\eta \hfill \\ \hfill & \le -({\overline{\xi }}^T\overline{\mathrm{{\rm Y}}}\overline{\xi }),\hfill \end{array}$$where$$\begin{array}{cc}& {\overline{\xi }}^T=\left[u^T,,,,,{\eta }^T,,,,,{\eta }^T,(t-d),,,,,{\int }_{t-d}^t,{\eta }^T(q)dq\right],\hfill \\ & \overline{\mathrm{{\rm Y}}}=\left[\begin{array}{cccc}\mathcal{R}& 0& 0& 0\\ \ast & \overline{\mathcal{Q}}-{\mathcal{M}}_d& -\mathcal{M}{\mathcal{N}}_d& -\mathcal{M}\\ \ast & \ast & -{\mathcal{N}}_d& -\mathcal{N}\\ \ast & \ast & \ast & 0\\ \end{array}\right].\hfill \end{array}$$According to (32), if $\overline{\mathrm{{\rm Y}}}\ge 0$, then $-({\overline{\xi }}^T\overline{\mathrm{{\rm Y}}}\overline{\xi })<0$ holds, leading to $\dot{L}(\eta )<0$. This proves that the system (1) is asymptotically stable, thus completing the proof. $\square$

It is essential to delve deeper into the discussion regarding the BMI (26). As we are aware, solving this BMI is a challenging task. To address this difficulty and analyze the permissible delays, we can employ the following expression to assess the feasibility of the BMI:

33

$$\begin{array}{c}\hfill {\overline{\lambda }}_{max}(\mathcal{R})∥u_{}^{\ast }∥_{}^2-{\overline{\lambda }}_{max}(\mathrm{\Pi }){∥\eta ∥}^2=0\end{array}$$where

34

$$\begin{array}{c}\hfill \mathrm{\Pi }=\left[\begin{array}{ccc}\overline{\mathcal{Q}}-{\mathcal{M}}_d\hfill & -\mathcal{M}{\mathcal{N}}_d& -\mathcal{M}\\ \ast \hfill & -{\mathcal{N}}_d& -\mathcal{N}\\ \ast \hfill & \ast & 0\end{array}\right]\end{array}$$

Remark 2

The Lyapunov-Krasovskii function, ${\int }_{t-d}^t{\eta }^T(q)$${\overline{\mathcal{Q}}}_d\eta (q)dq$, is used for analyzing the stability of time-delay systems and is incorporated into the value function (19). Its purpose in the cost function is to account for the effects of time delays on the system, allowing this information to be used to update the control law. As demonstrated in Theorem 2, integrating the Lyapunov-Krasovskii function with time delays into the cost function helps ensure the system’s stability.

According to (26), $\overline{\mathrm{{\rm Y}}}$ incorporates the time delay $d$. The BMI condition $\overline{\mathrm{{\rm Y}}}\ge 0$ helps identify the permissible delays. In other words, since the feasibility of $\overline{\mathrm{{\rm Y}}}\ge 0$ is dependent on $d$, we can determine the maximum allowable delays. In the subsequent stages, a single-layer neural network to tackle the task of solving the HJB equation is provided.

Neural network algorithm

We know that neural networks are great at handling complex functions. Since the performance index function (19) is usually quite complicated and does not have an exact mathematical expression, we will use a neural network to estimate it. In this paper, we use a simple one-layer neural network to do this. Then, following [35], $\mathcal{V}(\eta )$ is expressed as

35

$$\begin{array}{c}\hfill \mathcal{V}(\eta )={\mathcal{W}}_c^T\mathcal{S}(\eta )+\epsilon (\eta ),\end{array}$$here, consider $\mathcal{S}(\eta )$ as the activation function, where ${\mathbb{R}}^c$ signifies the real-valued space with dimension $c$. ${\mathcal{W}}_c$ denotes the optimal weight vector, and $c$ represents the number of neurons in the hidden layer. $\epsilon (\eta )$ stands for the approximation error of the neural network. The gradient of Eq. (35) concerning $\eta$ can then be expressed as:

36

$$\begin{array}{c}\hfill \mathrm{\nabla }\mathcal{V}(\eta )={(\mathrm{\nabla }\mathcal{S}(\eta ))}^T{\mathcal{W}}_c+\mathrm{\nabla }\epsilon (\eta ),\end{array}$$let $\mathrm{\nabla }\mathcal{S}(\eta )=\frac{\partial \mathcal{S}(\eta )}{\partial \eta }\in {\mathbb{R}}^{c\times n}$ denote the gradient of the activation function, and $\mathrm{\nabla }\epsilon (\eta )$ represent the gradient of the approximation error.

When we substitute Eq. (36) into (21), we get:

37

$$\begin{array}{c}\hfill 0=U(\eta ,u)+({(\mathrm{\nabla }\mathcal{S}(\eta ))}^T{\mathcal{W}}_c+\mathrm{\nabla }\epsilon (\eta ))\dot{\eta }.\end{array}$$So, the Hamiltonian can be stated as:

38

$$\begin{array}{cc}\hfill H(\eta ,u,W_c)& =U(\eta ,u)+({\mathcal{W}}_c^T\mathrm{\nabla }\mathcal{S}(\eta ))\dot{\eta }\hfill \\ \hfill & =-\mathrm{\nabla }\epsilon (\eta )\dot{\eta }\triangleq e_{\mathit{rH}}.\hfill \end{array}$$In this context, $e_{\mathit{rH}}$ signifies the residual error stemming from the neural network approximation. Since the ideal weight vector ${\mathcal{W}}_c$ is unknown, we utilize the critic neural network to estimate $\mathcal{V}(\eta )$ as:

39

$$\begin{array}{c}\hfill \widehat{\mathcal{V}}(\eta )={\mathcal{W}}_c^T\mathcal{S}(\eta ).\end{array}$$As a result, we can formulate the gradient of $\widehat{\mathcal{V}}(\eta )$ as:

40

$$\begin{array}{c}\hfill \mathrm{\nabla }\widehat{\mathcal{V}}(\eta )={(\mathrm{\nabla }\mathcal{S}(\eta ))}^T{\widehat{\mathcal{W}}}_c,\end{array}$$hence, we can derive the approximate Hamiltonian as:

41

$$\begin{array}{cc}\hfill H(\eta ,u,{\widehat{\mathcal{W}}}_c)& =U(\eta ,u)+({\mathcal{W}}_c^T\mathrm{\nabla }\mathcal{S}(\eta ))\dot{\eta }\hfill \\ \hfill & \triangleq e_r.\hfill \end{array}$$We define the weight approximation error as ${\tilde{\mathcal{W}}}_c={\mathcal{W}}_c-{\widehat{\mathcal{W}}}_c$. Utilizing (41) and (38), we obtain:

42

$$\begin{array}{c}\hfill e_r=e_{\mathit{rH}}-{\tilde{\mathcal{W}}}_c^T\mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta }.\end{array}$$We can revise the weight approximation error as:

43

$$\begin{array}{c}\hfill {\dot{\tilde{\mathcal{W}}}}_c=-{\dot{\widehat{\mathcal{W}}}}_c=L_r(e_{\mathit{rH}}-{\tilde{\mathcal{W}}}_c^T\mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta })\mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta }.\end{array}$$To update the weight vector ${\widehat{\mathcal{W}}}_c$ of the critic neural network, we minimize the objective function $E_c=\frac{1}{2}{e}_r^T{e}_r$ using the normalized gradient algorithm. The weight vector ${\widehat{\mathcal{W}}}_c$ should be updated as follows:

44

$$\begin{array}{c}\hfill {\dot{\widehat{\mathcal{W}}}}_c=-L_r{e}_r\mathrm{\nabla }S(\eta )\dot{\eta },\end{array}$$in this expression, $L_r>0$ denotes the learning rate of the critic neural network. Therefore, considering (23) and (35), we can describe the ideal control policy as:

45

$$\begin{array}{c}\hfill u(\eta )=-\frac{1}{2}{\mathcal{R}}^{-1}{\mathcal{B}}^T({(\mathrm{\nabla }\mathcal{S}(\eta ))}^T{\mathcal{W}}_c+\mathrm{\nabla }\epsilon (\eta )),\end{array}$$and it can be estimated as:

46

$$\begin{array}{c}\hfill \widehat{u}(\eta )=-\frac{1}{2}{\mathcal{R}}^{-1}{\mathcal{B}}^T{(\mathrm{\nabla }\mathcal{S}(\eta ))}^T{\widehat{\mathcal{W}}}_c.\end{array}$$The control policy (46) solely relies on the critic neural network. This means that updating the weight vector of the critic neural network using (44) eliminates the need to train the action neural network. As a result, this approach is not only feasible but also computationally efficient for implementation.

Theorem 3

If the weights of the critic neural network are updated according to (43) for the system described in (18), the error in weight approximation is uniformly ultimately bounded.

Proof

For this theorem, we used the following Lyapunov candidate:

47

$$\begin{array}{c}\hfill {\mathrm{\Gamma }}_2=\frac{1}{2L_r}{\tilde{\mathcal{W}}}_c^T{\tilde{\mathcal{W}}}_c.\end{array}$$The time derivative of (47) is

48

$$\begin{array}{cc}\hfill {\dot{\mathrm{\Gamma }}}_2& =\frac{1}{2L_r}{\tilde{\mathcal{W}}}_c^T{\dot{\tilde{\mathcal{W}}}}_c={\tilde{\mathcal{W}}}_c^T(e_{\mathit{rH}}-{\tilde{\mathcal{W}}}_c^T\mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta })\hfill \\ \hfill & \times \mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta }={\tilde{\mathcal{W}}}_c^T{e}_{\mathit{rH}}\mathrm{\nabla }\mathcal{S}(\eta )\dot{\eta }-{∥{\tilde{\mathcal{W}}}_c^T,\mathrm{\nabla },\mathcal{S},(\eta ),\dot{\eta }∥}^2\hfill \\ \hfill & \le \frac{1}{2}{e}_{\mathit{rH}}^2-\frac{1}{2}{∥{\tilde{\mathcal{W}}}_c^T,\mathrm{\nabla },\mathcal{S},(\eta ),\dot{\eta }∥}^2.\hfill \end{array}$$Hence, ${\dot{\mathrm{\Gamma }}}_2<0$ holds when ${\tilde{\mathcal{W}}}_c$ belongs to the compact set defined by $\Vert {\tilde{\mathcal{W}}}_c\Vert \le \Vert \frac{e_{\mathit{rH}}}{{\theta }_1}\Vert$, assuming $\Vert \mathrm{\nabla }\mathcal{S}(z)\dot{z}\Vert \le {\theta }_1$, where ${\theta }_1$ is a positive constant. Then, we can conclude that the weight approximation error is upper uniformly bounded, thus completing the proof. $\square$

The ADP algorithm based on neural network observer associated with the approach proposed in this paper is detailed in Algorithm 1.

[See PDF for image]

Algorithm 1

Fault tolerant for a class of nonlinear time delay systems using ADP

Remark 3

For the computational requirements and efficiency of using a critic neural network in real-time applications, several factors need to be considered. These include the complexity of the neural network, the hardware capabilities, and the specific requirements of the real-time system. Real-time applications require low-latency responses, so the neural network should be optimized to produce outputs within the application’s time constraints.

Simulation numerical examples

This section presents numerical simulation examples to illustrate the effectiveness of the proposed method.

The simulation results were obtained using Julia programming, and the reader can download them from this link

Example: Chemical Reaction Tank

The chemical reactor systems consist of multiple chemical reactors and stirring bars, which introduce delays into the systems. The state-space expression of the chemical reaction is given as follows:

49

$$\begin{array}{cc}& {\dot{z}}_1(t)=-(\frac{1}{T_{11}}+C_{11})z_1(t)+(\frac{1-F_{12}}{V_{11}})z_2(t)\hfill \\ \hfill & +M_{11}sin(z_1(t)),\hfill \\ \hfill & {\dot{z}}_2(t)=-(\frac{1}{T_{12}}+C_{12})z_2(t)+(\frac{F_{12}}{V_{12}})z_2(t-d)\hfill \\ \hfill & +u(t)+a(t),\hfill \end{array}$$The chemical reaction parameters have been set as follows: $T_{11}=50$, $T_{12}=50$, $C_{11}=3$, $C_{12}=3$, $M_{11}=0.1$, $M_{12}=0.1$, $F_{11}=0.5$, $F_{12}=0.5$, $V_{11}=0.5$, and $V_{12}=0.5$.

According to (49):$$\begin{array}{cc}& \mathcal{C}=\left[\begin{array}{cc}1\hfill & 0\\ 0\hfill & 1\end{array}\right],{\mathcal{D}}_f=\left[\begin{array}{c}0\\ 1\\ \end{array}\right],\mathcal{B}=\left[\begin{array}{c}0\\ 1\\ \end{array}\right],\hfill \end{array}$$We assume that the actuator fault dynamic is as follows:$$\begin{array}{c}\hfill a(t)=\left\{\begin{array}{cc}0,\hfill & 0s\le t<10s,\\ 0.5sin(t/2\pi ),\hfill & 10s\le t\le 20s.\end{array}\right)\end{array}$$In this example, the initial values are chosen as $z(0)=[0.95;-0.8]$, and other parameters are set as follows: $\text{learning rate}=0.01$, $\mathcal{R}=I$, and $\mathcal{Q}=I_2$. The observer’s parameters are obtained from (6) as follows:$$\begin{array}{c}\hfill \mathcal{L}=\left[\begin{array}{cc}0.4941& -0.0385\\ -0.0645& 0.0091\end{array}\right],\mathrm{\Lambda }=\left[\begin{array}{cc}0.026& 0.1695\\ 0.4417& 12.7558\end{array}\right]\end{array}$$An adaptive neural network observer is designed to estimate the system states and the unknown function. The RBF network employs $M=11\times 11=121$ nodes, with neuron centers distributed across the range $[-1.2,1.2]\times [-1.2,1.2]$. The initial weight matrix is set to ${\widehat{\mathcal{W}}}_o=0$. The time delay is assumed to be $d=0.3s$. The weight vector in critic network learning is denoted as ${\widehat{\mathcal{W}}}_c={[{\widehat{\mathcal{W}}}_{c1},{\widehat{\mathcal{W}}}_{c2},{\widehat{\mathcal{W}}}_{c3}]}^T$, with its initial values given by $[{\widehat{\mathcal{W}}}_{c1},{\widehat{\mathcal{W}}}_{c2}]={[1,1]}^T$. The dimension of ${\widehat{\mathcal{W}}}_{c3}$ is determined by ${\mathcal{F}}^T(y)\in {\mathbb{R}}^{n\times N^2}$. Additionally, the activation function for the critic neural network is specified as ${\sigma }_c(z)=[{\widehat{x}}_1^2,{\widehat{x}}_1{\widehat{x}}_2,{\widehat{x}}_1\widehat{\mathcal{W}},{\widehat{x}}_2^2,{\widehat{x}}_2\widehat{\mathcal{W}},{\widehat{\mathcal{W}}}^2]$. The network comprises $c=2+121=123$ neurons, and the critic weight vectors are represented by ${\widehat{\mathcal{W}}}_c=[{\mathcal{W}}_{c1},\dots ,{\mathcal{W}}_{c123}]$.

The simulation results are shown in Figs. 1, 2, 3, 4 and 5. Figure 1 illustrates that the states of the nonlinear system (1) are stabilized under the control policy (46). When an actuator fault happens at $t=10s$, there is a noticeable change in the system states, but the controller still performs well. This indicates that the system, with control policy (46), can handle actuator faults. Figure 2 displays the control input (46) during simulation time. In our theoretical development, we focused on ensuring robustness and fault tolerance, which may sometimes introduce oscillations due to the conservative nature of the design. However, in practical implementations, these oscillations can be mitigated through damping techniques, filtering, and careful parameter tuning to ensure smooth control signals.

Figure 3 shows the performance of fault estimation and unknown dynamics, demonstrating that the proposed ADP-based approach can accurately and effectively estimate faults. Furthermore, Fig. 4 presents the critic weight vectors, indicating their convergence. For plotting ${\widehat{\mathcal{W}}}_o$, we used the norm $\left|W\right|$ due to the large size of ${\widehat{\mathcal{W}}}_o$, which is a vector with 121 columns.

[See PDF for image]

Fig. 1

System states and observer estimations

[See PDF for image]

Fig. 2

The control input (46)

[See PDF for image]

Fig. 3

The fault and unknown dynamic estimation performance

[See PDF for image]

Fig. 4

The weights of the critic neural network

The value of the cost function (19) is shown in Fig. 5. The integral part, which is related to the time delay, is also depicted in Fig. 5, indicating that when the states of the system converge to the equilibrium point, the rate of change of the cost function becomes constant.

[See PDF for image]

Fig. 5

The value of cost function using the proposed method

Conclusion

In this paper, we have proposed a FTC method for partially unknown nonlinear systems with time delays, employing adaptive dynamic programming. The neural network observer was implemented to estimate states, unknown dynamics, and actuator fault of the system. The observer gains were determined by solving a linear matrix inequality. A new value function index was introduced to account for time-delay states, and control laws were derived based on this value function. The Hamilton–Jacobi–Bellman equation for this value function was solved using a critic neural network. Lyapunov functional analysis demonstrated that the closed-loop system is uniformly ultimately bounded.

A promising future direction would be to extend our adaptive neural network observer and time-delay inclusive ADP approach to decentralized control scenarios for interconnected nonlinear systems with multiple fault types, as investigated in [32, 36].

Acknowledgements

None.

Author Contributions

Farshad Rahimi contributed to formulation, writing, simulation, methodology, mathematics, review, and editing.

Funding

The authors claim that there are no funds, grants, or other support devoted to the preparation of this manuscript.

Data availability

The simulation results were obtained using Julia programming, and the reader can download them from [this link](https://farshad-rahimi.github.io/FarshadRahimi//Learning%20Julia/).

Declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

1. Ni, Q; Ding, F; Zhan, Z; Liu, J; Li, X; Zhao, Z. A fault diagnosis and measurement method for temperature measurement circuits in electric traction drive system. Measurement; 2024; 227, 114250. [DOI: https://dx.doi.org/10.1016/j.measurement.2024.114250]

2. Isapour, S; Tavakoli-Kakhki, M. Rough neural network based data-driven model-free adaptive fault-tolerant control for discrete-time nonlinear systems. Eng Appl Artif Intell; 2024; 134, 108651. [DOI: https://dx.doi.org/10.1016/j.engappai.2024.108651] 07964803

3. Gao, Z; Cecati, C; Ding, SX. A survey of fault diagnosis and fault-tolerant techniques-part i: Fault diagnosis with model-based and signal-based approaches. IEEE Trans Industr Electron; 2015; 62, 6 pp. 3757-3767. [DOI: https://dx.doi.org/10.1109/TIE.2015.2417501] 1365.94571

4. Li, H; Yang, C; Hu, Y; Zhao, B; Zhao, M; Chen, Z. Fault-tolerant control for current sensors of doubly fed induction generators based on an improved fault detection method. Measurement; 2014; 47, pp. 929-937. [DOI: https://dx.doi.org/10.1016/j.measurement.2013.10.021] 1313.03023

5. Amin, AA; Hasan, KM. A review of fault tolerant control systems: advancements and applications. Measurement; 2019; 143, pp. 58-68. [DOI: https://dx.doi.org/10.1016/j.measurement.2019.04.083] 1227.97065

6. Jia, F; Cao, F; He, X. Active fault-tolerant control against intermittent faults for state-constrained nonlinear systems. IEEE Trans Syst, Man, Cybern: Syst; 2024; 54, pp. 2389-2401. [DOI: https://dx.doi.org/10.1109/TSMC.2023.3344292] 07966208

7. Hu, J; Sha, Y; Yao, J. Dual neural networks based active fault-tolerant control for electromechanical systems with actuator and sensor failure. Mech Syst Signal Process; 2023; 182, 109558. [DOI: https://dx.doi.org/10.1016/j.ymssp.2022.109558] 1137.86312

8. Ke, C; Cai, K-Y; Quan, Q. Uniform passive fault-tolerant control of a quadcopter with one, two, or three rotor failure. IEEE Trans Robot; 2023; 39, pp. 4297-4311. [DOI: https://dx.doi.org/10.1109/TRO.2023.3297048] 1548.93088

9. Shen, Q; Jiang, B; Shi, P. Fault diagnosis and fault-tolerant control based on adaptive control approach; 2017; Berlin, Springer: 1417.9302491

10. Zhang, X-N; Li, X-J. Adaptive fault-tolerant control for a class of stochastic nonlinear systems with multiple sensor faults. Int J Syst Sci; 2020; 51, 12 pp. 2217-2237.4141434 [DOI: https://dx.doi.org/10.1080/00207721.2020.1793231] 1483.93328

11. Shen, Q; Shi, P; Lim, CP. Fuzzy adaptive fault-tolerant stability control against novel actuator faults and its application to mechanical systems. IEEE Trans Fuzzy Syst; 2024; 32, pp. 2331-2340. [DOI: https://dx.doi.org/10.1109/TFUZZ.2023.3343403] 07902394

12. Camacho EF, Alamo T, Peña DM (2010) Fault-tolerant model predictive control. In: 2010 IEEE 15th Conference on Emerging Technologies & Factory Automation (ETFA 2010), IEEE, pp. 1–8

13. Bavili, RE; Khosrowjerdi, MJ et al. Active fault tolerant controller design using model predictive control. J Control Eng Appl Inf; 2015; 17, 3 pp. 68-76.1312.93047

14. Wang, L; Qi, R; Jiang, B. Adaptive fault-tolerant optimal control for hypersonic vehicles with state constrains based on adaptive dynamic programming. J Franklin Inst; 2024; 361, 8 106833.4733861 [DOI: https://dx.doi.org/10.1016/j.jfranklin.2024.106833] 1539.93099

15. Zhao, Y; Wang, H; Xu, N; Zong, G; Zhao, X. Reinforcement learning-based decentralized fault tolerant control for constrained interconnected nonlinear systems. Chaos, Solitons & Fractals; 2023; 167, 113034.4525496 [DOI: https://dx.doi.org/10.1016/j.chaos.2022.113034] 1452.65068

16. Bardi, M; Dolcetta, IC et al. Optimal control and viscosity solutions of Hamilton–Jacobi–Bellman equations; 1997; Berlin, Springer: [DOI: https://dx.doi.org/10.1007/978-0-8176-4755-1] 0890.4901112

17. Wang, T; Wang, Y; Yang, X; Yang, J. Further results on optimal tracking control for nonlinear systems with nonzero equilibrium via adaptive dynamic programming. IEEE Trans Neural Netw Learn Syst; 2021; 34, 4 pp. 1900-1910.4573877 [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3105646] 07886811

18. Liu, D; Xue, S; Zhao, B; Luo, B; Wei, Q. Adaptive dynamic programming for control: a survey and recent advances. IEEE Trans Syst, Man, and Cybernet: Syst; 2020; 51, 1 pp. 142-160. [DOI: https://dx.doi.org/10.1109/TSMC.2020.3042876] 1480.35332

19. Zhao, B; Liu, D; Li, Y. Online fault compensation control based on policy iteration algorithm for a class of affine non-linear systems with actuator failures. IET Control Theory & Appl; 2016; 10, 15 pp. 1816-1823.3587316 [DOI: https://dx.doi.org/10.1049/iet-cta.2015.1105] 1197.35332

20. Zhao, B; Liu, D; Li, Y. Observer based adaptive dynamic programming for fault tolerant control of a class of nonlinear systems. Inf Sci; 2017; 384, pp. 21-33. [DOI: https://dx.doi.org/10.1016/j.ins.2016.12.016] 1432.93185

21. Zeng, C; Zhao, B; Liu, D. Fault tolerant control for a class of nonlinear systems with multiple faults using neuro-dynamic programming. Neurocomputing; 2023; 553, 126502. [DOI: https://dx.doi.org/10.1016/j.neucom.2023.126502] 1432.93185

22. Wang, Z; Liu, L; Wu, Y; Zhang, H. Optimal fault-tolerant control for discrete-time nonlinear strict-feedback systems based on adaptive critic design. IEEE Trans Neural Netw Learn Syst; 2018; 29, 6 pp. 2179-2191.3811407 [DOI: https://dx.doi.org/10.1109/TNNLS.2018.2810138] 1437.93129

23. Fan, Q-Y; Yang, G-H. Adaptive fault-tolerant control for affine non-linear systems based on approximate dynamic programming. IET Control Theory & Appl; 2016; 10, 6 pp. 655-663.3469069 [DOI: https://dx.doi.org/10.1049/iet-cta.2015.1081] 1412.47213

24. Chen, T; Zeng, C; Wang, C. Fault identification for a class of nonlinear systems of canonical form via deterministic learning. IEEE Trans Cybern; 2021; 52, 10 pp. 10957-10968. [DOI: https://dx.doi.org/10.1109/TCYB.2021.3072645] 1497.53121

25. Zeng C, Zhao B, Liu D (2023) Adaptive neural network observer-based fault tolerant control for partially unknown nonlinear systems via adaptive dynamic programming. In: 2023 IEEE 13th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), IEEE, pp. 212–216

26. Rahimi, F; Mahboobi Esfanjani, R. Estimating tolerable communication delays for distributed optimization problems in control of heterogeneous multi-agent systems. IET Control Theory & Appl; 2024; 18, 5 pp. 626-639.4736452 [DOI: https://dx.doi.org/10.1049/cth2.12595] 1323.93055

27. Zhao, W; Zhang, H; Yu, W. Distributed optimal output synchronization of heterogeneous multiagent systems with communication delays. Int J Robust Nonlinear Control; 2024; 34, pp. 7821-7836.4773657 [DOI: https://dx.doi.org/10.1002/rnc.7369] 1543.93021

28. Fridman E (2014) Introduction to time-delay systems: analysis and control. In: Systems & control: foundations & applications. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-09393-2. Website: https://link.springer.com/book/10.1007/978-3-319-09393-2

29. Zhang, H; Liu, D; Luo, Y; Wang, D. Adaptive dynamic programming for control: algorithms and stability; 2012; Berlin, Springer: 1279.49017

30. Wang, C; Hill, DJ. Deterministic learning and rapid dynamical pattern recognition. IEEE Trans Neural Networks; 2007; 18, 3 pp. 617-630. [DOI: https://dx.doi.org/10.1109/TNN.2006.889496] 1247.93001

31. Wang, Y; Wang, T; Li, C; Yang, J. Neural network-based optimal fault-tolerant control for interconnected nonlinear systems with actuator failures. IEEE Trans Emerg Top Comput Intell; 2024; 8, pp. 1828-1840. [DOI: https://dx.doi.org/10.1109/TETCI.2024.3358981] 1369.93680

32. Wang, Y; Wang, T; Yang, X; Yang, J. Gradient descent-Barzilai Borwein-based neural network tracking control for nonlinear systems with unknown dynamics. IEEE Trans Neural Netw Learn Syst; 2021; 34, 1 pp. 305-315.4551072 [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3093877] 1377.28003

33. Al-Tamimi, A; Lewis, FL; Abu-Khalaf, M. Discrete-time nonlinear HJB solution using approximate dynamic programming: convergence proof. IEEE Trans Syst, Man, and Cybernet, Part B (Cybernetics); 2008; 38, 4 pp. 943-949. [DOI: https://dx.doi.org/10.1109/TSMCB.2008.926614] 1210.90170

34. Lewis, FL; Vrabie, D; Syrmos, VL. Optimal control; 2012; Hoboken, John Wiley & Sons: [DOI: https://dx.doi.org/10.1002/9781118122631] 1284.49001

35. Zhang, L; Zhao, X; Zhao, N. Real-time reachable set control for neutral singular Markov jump systems with mixed delays. IEEE Trans Circuits Syst II Express Briefs; 2021; 69, 3 pp. 1367-1371.1369.05221

36. Rahimi, F; Rezaei, H. A distributed fault estimation approach for a class of continuous-time nonlinear networked systems subject to communication delays. IEEE Control Syst Lett; 2021; 6, pp. 295-300.4454081 [DOI: https://dx.doi.org/10.1109/LCSYS.2021.3071478] 1476.37013