Longitudinal Hierarchical Control of Autonomous Vehicle Based on Deep Reinforcement Learning and PID Algorithm

Abstract

Longitudinal control of autonomous vehicles (AVs) has long been a prominent subject and challenge. A hierarchical longitudinal control system that integrates deep deterministic policy gradient (DDPG) and proportional–integral–derivative (PID) control algorithms was proposed in this paper to ensure safe and efficient vehicle operation. First, a hierarchical control structure was employed to devise the longitudinal control algorithm, utilizing a Carsim-based model of the vehicle’s longitudinal dynamics. Subsequently, an upper controller algorithm was developed, combining DDPG and PID, wherein perceptual information such as leading vehicle speed and distance served as input state for the DDPG algorithm to determine PID parameters and output the desired acceleration of the vehicle. Following this, a lower controller was designed employing a PID-based driving and braking switching strategy. The disparity between the desired and actual accelerations was fed into the PID, which calculated the control acceleration to enact the driving and braking switching strategy. Finally, the effectiveness of the designed control algorithm was validated through simulation scenarios using Carsim and Simulink. Results demonstrate that the longitudinal control method proposed herein adeptly manages vehicle speed and following distance, thus satisfying the safety requirements of AVs.

This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

1. Introduction

The surge in vehicles has intensified the sustained trend of increasing traffic accident rates, highlighting the enduring importance of road traffic problems. It is worth noting that in recent years, leading companies including Baidu, Tesla, and Google have conducted extensive research work in the field of autonomous driving. AVs have the potential to contribute to the reduction of such traffic accidents and are poised to become a pivotal factor in transforming road traffic conditions. The design of precise control algorithms of AVs holds practical and theoretical significance.

Up to this point, a range of traditional and intelligent control algorithms for AVs have been developed. Traditional control methods rely on mathematical models of the controlled object to achieve closed-loop control, rendering them the preferred choice for practical vehicle applications. These methods encompass widely adopted approaches such as proportional–integral–derivative (PID) control, optimal control (OC), and model predictive control (MPC). For instance, Simorgh et al. [1] considered aerodynamics and employed a control algorithm based on the PID model reference adaptive control to manage the longitudinal speed of vehicles. HosseinNia et al. [2] proposed an optimized fractional order control approach and designed two distinct PI controllers based on the PID control theory to regulate throttle and brake systems, enhancing safety performance in traffic congestion scenarios. Schrdel et al. [3] derived OC laws based on dynamic programming principles, utilizing OC for tackling speed control challenges involving multiple objectives. Graffione et al. [4] developed a MPC methodology to oversee the spacing and speed of vehicle platoons, ultimately enhancing road safety. Moon et al. [5] introduced an efficient method for identifying target vehicles within road traffic flow environments, exploring the system’s smoothness and safety when tracking target vehicles through optimal controllers. Mekala et al. [6] proposed a longitudinal motion speed control strategy based on MPC for AV, aiming to achieve vehicle stability and safe driving.

In comparison to traditional control approaches, intelligent control methods offer distinct advantages. They operate without the need for fundamental mathematical models, are not reliant on precise control architectures, and possess the capability to autonomously learn from extensive data sources [7]. Intelligent control methods exhibit the ability to adapt to diverse and complex environments, making them invaluable in the realm of autonomous driving. Prominent examples of intelligent control methods include fuzzy control (FC), sliding mode control (SMC), and neural network control methods. For instance, Liangyi et al. [8] considered the integration of vehicle internal information and devised a fuzzy proportional–integral (PI) controller to govern longitudinal speed. This approach effectively ensures the safety of vehicle tracking and maintains stable fleet operation. Jin et al. [9] designed a longitudinal controller using the SMC method, which proves highly effective in tracking vehicle speed. Yuan et al. [10] proposed an end-to-end control methodology based on 3D convolutional neural networks (3DCNNs) and long short-term memory (LSTM) for coupled lateral and longitudinal lane-keeping control. PérezGil et al. [11] explored the use of deep Q-networks (DQN) for learning intelligent vehicle tracking control problems and tested these methods within the simulation environment, CARLA. He et al. [12] introduced an innovative entropy-constrained reinforcement learning scheme for multiobjective longitudinal decision-making of electric AV. Coppola et al. [13] also demonstrated the effectiveness of adaptive cruise control for autonomous electric vehicles using a Q-learning algorithm.

The longitudinal control system of AV presents a typical nonlinear discrete state with time-varying uncertain parameters and multiple disturbances. Therefore, it becomes imperative to consider multiple factors in the actual driving process. Singular reliance on either traditional or intelligent control modes falls short of meeting the precise control requirements for the longitudinal motion of AV, jeopardizing tracking accuracy. Consequently, numerous studies have integrated control methodologies to attain safer and more efficient longitudinal vehicle control. For instance, Sun et al. [14] formulated an objective function based on tracking performance and driving characteristics during the tracking process. They designed an upper-level optimal controller and created a lower-level low-speed tracking controller employing the fuzzy PID control method to track the desired speed. Yao [15] introduced an intelligent control method based on Q-learning with variable domain fuzzy PID. Nie et al. [16], in pursuit of enhanced speed tracking accuracy and safety management for AV, adopted a control approach based on adaptive radial basis function neural networks (RBFNNs) coupled with PID control, employing the vehicle dynamics model to establish the longitudinal speed control model.

In summary, researchers have successively engaged in the study of autonomous driving control technology, yielding numerous research achievements. However, concerning the control methods for autonomous vehicles (AVs), traditional and intelligent longitudinal control methods differ in principles and applicability for distinct scenarios, resulting in significant variations in their application outcomes. In previous longitudinal control research, reinforcement learning can be directly used to calculate the desired control inputs, but its effectiveness is not significantly improved compared to the PID algorithm, and training and testing processes require a lot of time [17]. In addition, the effectiveness of reinforcement learning is sometimes unstable [18], and the training process is easily affected by data bias and variance, so it is not optimal [19]. On the contrary, real vehicles predominantly employed traditional control algorithms, with PID control being suitable for simpler scenarios. It is a classic traditional method that requires three key parameters to be determined. When faced with complex systems, PID parameters are challenging to tune, requiring manual adjustment of controller parameters and posing difficulties in adapting to various operational conditions.

Therefore, this paper adopts a hierarchical control approach to investigate the longitudinal control of AVs. The upper-level control module utilizes a combination of PID control and DDPG algorithm to analyze environmental state information during vehicle operation and generate desired signals. The lower-level control is established based on PID control with drive-brake switching control logic. A joint simulation model is employed to evaluate the system’s responsiveness and stability across various scenarios.

2. Modeling of Vehicle Longitudinal Dynamics System

Carsim is a specialized software tailored for dynamic modeling and simulation of medium and small vehicles [20]. In this paper, the C-Class Hatchback model was used to construct a longitudinal dynamics system model for AVs, which involved several parameters within Carsim (such as vehicle body parameters) and integrating the engine, transmission, hydraulic torque converter, as well as other power and braking components into the control system.

To achieve a closer approximation of the actual vehicle control performance, the primary parameters of the vehicle body in Carsim are calibrated according to the Volkswagen Tanyue model, while the remaining parameters are configured utilizing the built-in parameters of the C-Class model. The specific settings for vehicle and environmental parameters are detailed in Table 1.

Table 1

Vehicle parameter settings.

Parameters	Symbol	Value	Units
Vehicle mass	$m$	1616	kg
Tire radius	$r$	324	mm
Final drive ratio	$i_0$	4.1	—
Air density	$\rho$	1.293	g/L
Air resistance coefficient	$C_D$	0.3	—
Rolling resistance coefficient	$f$	0.018	—
Tire-road friction coefficient	$\mu$	0.85	—
Transmission efficiency	$\eta$	0.92	—
Windward area	$A$	2.2	m2

To valid the effectiveness of the simulation, this paper conduct modeling tests in conjunction with Carsim and MATLAB/Simulink platforms. The signals ultimately fed back to the vehicle model by the lower controller are throttle and brake pressure. Consequently, the selected input signals comprise the throttle and brake master cylinder pressure under the current state. The output signals are the longitudinal speed $V_x$, longitudinal acceleration $A_x$, engine speed $AV\_Eng$, relative speed between two vehicles $SpdS1\_1$, relative distance between two vehicles $DisS1\_1$, and transmission gear $Rgear\_Tr$.

3. Upper Controller of AV Based on DDPG-PID

3.1. PID Controller Based on DDPG Algorithm

Given the continuous nature of both the input and output spaces in the domain of AV’s motion control, the PID controller emerges as the prevalent choice in industrial applications, owing to its straightforward design and robust performance. Nevertheless, the selection of the three pivotal parameters in the traditional PID controller significantly impacts system control but proves challenging to determine accurately. The conventional tuning process is not only time-consuming and labor-intensive but also lacks adaptability to diverse operational conditions. Deep reinforcement learning (DRL) stands out for its exceptional learning and adaptive capabilities. When employed for parameter adjustment of PID, it facilitates dynamic tuning in response to system variations, thereby enhancing control accuracy. In this study, the DDPG algorithm, integrated within the actor-critic (AC) framework suitable for handling continuous input and output spaces, is utilized to perform real-time PID parameter adjustments. Subsequently, these optimized parameters are input into the PID controller to enable precise target control, as shown in Figure 1.

[figure(s) omitted; refer to PDF]

3.2. PID Control Theory

PID controllers represent the most prevalent control algorithms employed in industrial processes [21]. The outcomes of the PID control algorithm are intricately linked to three key hyperparameters and are typically applied to regulate the state of the controlled system within diverse operational scenarios [22]. The adjustment of PID controller parameters $K_p,K_i,K_d$ and $K_p,K_i,K_d$ typically relies on specific research objectives to attain the desired control outcomes. It involves computing the disparity between the system input (setpoint) and the feedback (process variable) from the controlled system, denoted as the error $t$. The error $et$ between the system input $rt$ and the feedback $yt$ from the controlled object is formed, which is $et=rt-yt$. Following the calculations, the control output $ut$ is generated [23].

This study employs a positional PID control strategy to derive the actual control output, which can be expressed using the following formula:$$\begin{array}{}\text{(1)}& ut=k_{p}et+k_i{\int }_0^{t}etdt+k_d\frac{det}{dt},\end{array}$$where $ut$ is control signal of the PID controller regulating system; $K_p$ is proportional coefficient; $K_i$ is integral coefficient; $K_d$ is differential coefficient; $T_i$ integral time constant; $T_d$ is differential time constant.

3.3. DDPG Algorithm

DRL represents a convergence of deep learning and reinforcement learning, capable of addressing challenges that arise in complex state spaces. Among the model-free DRL approaches, the DDPG algorithm has the ability to produce continuous actions. Consequently, this study utilizes the DDPG algorithm for optimizing the PID controller [24].

In general, the DRL method typically models Markov decision processes (MDPs) relevant to specific research challenges [25]. This modeling process involves the integration of the current state $s_t$, action $a_t$, reward $r_t$, and next state $s_{t+1}$ into a set of elements $s_t,a_t,r_t,s_{t+1}$. These smaller sets of elements are amalgamated to create a larger set $S,A,R,S^{\prime }$, with optimization objectives $\pi :s⟶a$. The objective is to ensure that the system’s optimization strategy maximizes the cumulative reward value at time t. To achieve this, the Q function is defined as $Q^{\pi }=E{R}_t{s}_t,a_t$. The ultimate objective of the continuous optimization within the DRL algorithm is to identify an optimal strategy ${\pi }^{\mathrm{\ast }}$ that maximizes the cumulative reward value (R) while establishing as follows:$$\begin{array}{}\text{(2)}& Q^{{\pi }^{\mathrm{\ast }}}s,a\ge Q^{\pi }s,a,\forall s,a\in S,A.\end{array}$$

This study offers a comprehensive explanation of the reward function developed in future sections. Regarding the network structure, DDPG relies on the AC framework to facilitate the optimization process, which comprises the actor network $\mu s{\theta }^{\mu }$, critic network $Qs,a{\theta }^Q$, target actor network $\mu s{\theta }^{{\mu }^{\prime }}$, and target critic network $Qs,a{\theta }^{Q^{\prime }}$.

The actor network employs a set of parameters ${\theta }^{\mu }$ to depict the deterministic strategy in the current state, and it generates corresponding actions based on the current strategy. Taking into account the relationship between cumulative rewards $Q^{\pi }=E{R}_t{s}_t,a_t$ and actions, as depicted in formula (3), a gradient ascent method is employed to update the parameters.$$\begin{array}{c}\text{(3)}& {\nabla }_{{\theta }^{\mu }}J\approx E_{s_t-{\rho }^{\beta }}{\nabla }_{{\theta }^{\mu }}Q{s,a{\theta }^Q\text{⁣}}_{s=s^{\prime },a=\mu s_t{\theta }^{\mu }}=E_{s_t-{\rho }^{\beta }}{\nabla }_{a}Qs,a{\theta }^{Q}s=s_t\\ a=\mu s_t\cdot {\nabla }_{{\theta }^{\mu }}\mu {s{\theta }^{\mu }\text{⁣}}_{s=s_t}.\end{array}$$

The input for the actor network proposed in this paper is $s_t$, and the output is $a_t$, which are three parameters of a set of PID controllers. The actor network comprises two hidden layers. The first layer in the network is a fully connected (FC) layer with 300 neurons, utilizing the rectified linear unit (ReLU) activation function. The second layer of the network is also an FC layer with 600 neurons, and it employs the ReLU activation function.

The critic network utilizes a set of parameters ${\theta }^Q$ to compute the Q value based on the current action, and the precision of the Q value plays a vital role in network convergence. The parameters are updated by minimizing the loss function described in formulas (4) and (5), thereby enhancing the accuracy of the Q value [26].$$\begin{array}{c}\text{(4)}& L{\theta }^Q={\mathrm{E}}_{s_t\sim {\rho }^{\beta },a_t\sim \beta ,r_t\sim \mathrm{E}}Q{s_t,a_t{\theta }^Q-Q^{\wedge }}^2=\frac{1}{N_1}{\displaystyle \sum }_t{Q{s}_t,a_t{\theta }^Q-Q^{\wedge }}^2,\\ \text{(5)}& Q^{\wedge }=r_t+\gamma Q^{\prime }{s}_{t+1},{\mu }^{\prime }{s}_{t+1}{\theta }^{{\mu }^{\prime }}{\theta }^{Q^{\prime }}.\end{array}$$

The input layer of the critic network in this study comprises two components: the vehicle state $s_t$ and the vehicle’s output actions $a_t$, which correspond to the parameters of the PID controller. The critic network’s hidden layer is structured as a multilayer network. The first hidden layer consists of 300 hidden units and employs the ReLU activation function. The second layer uses the linear function as the activation function and has 600 hidden units to extract features $l_s$. Additionally, after passing through the hidden layers following the vehicle action input $a_t$, there is a FC layer that utilizes the linear activation function with 600 hidden units to obtain further features $l_a$. Subsequently, these features are combined and fused, and then they are output through another FC layer with a ReLU activation function, which contains 600 hidden units. The purpose of the critic network’s output layer is to provide the value function for environmental state behavior, yielding a scalar $Q{s}_t,a_t$, and it does not require any activation functions.

The target strategy network employs parameters ${\theta }^{{\mu }^{\prime }}$ to estimate the desired action of the target, while the target Q network estimates the target Q value through parameters ${\theta }^{Q^{\prime }}$. The target network updates the respective parameter values using formulas (6) and (7), where α represents the sliding average coefficient [27].$$\begin{array}{}\text{(6)}& {\theta }^{Q^{\prime }}⟵\tau {\theta }^Q+1-\tau {\theta }^{Q^{\prime }},\text{(7)}& {\theta }^{{\mu }^{\prime }}⟵\tau {\theta }^{\mu }+1-\tau {\theta }^{{\mu }^{\prime }}.\end{array}$$

The structural design of the target actor network in this paper mirrors that of the actor network.

3.4. MDP Modeling

In this section, the longitudinal upper control problem of AV is expressed as an MDP model [28]. In the system involved in this study, it is assumed that the environment is completely observable, and there is a good communication foundation between the vehicle and the environment. The definition of MDP’s state space, action space, and reward function is as follows.

In a road traffic setting, when contemplating the longitudinal motion of vehicles, it becomes essential to account for the present driving state of the AV and its interaction with the preceding vehicle. If all the gathered information is directly integrated into the design of the state set, the significant volume of data can introduce complexities in the vehicle’s driving status, especially given the strong causal relationship among the data. The chosen state space encompasses the following variables: the vehicle’s speed, relative speed, vehicle acceleration, the acceleration of the preceding vehicle, and relative distance, denoted as $S=v_x,v_{rel},a_x,a_f,d_{rel}$

In the longitudinal controller designed for the AV in this study, a DRL algorithm is employed to perform real-time adjustments to the parameters of the PID controller. These adjusted parameters are subsequently fed into the PID controller, which calculates the desired acceleration value as the output. Consequently, the action space in this article comprises three gain coefficients $=K_p,K_i,K_d$.

This study takes into account various factors, including driving safety, efficiency, and comfort when designing reward functions. It assesses the significance of each factor by incorporating different weight coefficients. During the training process, AVs receive a notably substantial negative reward in the event of a collision, as illustrated in the following equation:$$\begin{array}{}\text{(8)}& r_1=\begin{array}{cc}0,& \Delta x\ge L,\\ -100,& \Delta x<L,\end{array}\end{array}$$where $r_1$ represents the reward function of safety, $\Delta x$ represents the distance between ego vehicle and the front vehicle, and $L$ represents the safe distance and the value is five in this paper.

In the desired state, it is essential to maintain a speed error of 0 between the vehicle and the preceding vehicle while driving. To attain this desired state, the reward function takes into account the corresponding rewards when the speed error between the current vehicle and the preceding vehicle is less than a certain threshold. The reward and penalty functions for speed errors are detailed in the following equations:$$\begin{array}{}\text{(9)}& r_2=C_1\cdot \Delta V_i+C_2,\text{(10)}& C_1=\begin{array}{cc}-4,& \Delta V_i\le 0.5,\\ -3,& 0.5<\Delta V_i<3,\\ 0,& \text{else},\end{array}\text{(11)}& C_2=\begin{array}{cc}2,& \Delta V_i\le 1,\\ 1.5,& 1<\Delta V_i<10,\\ -7.5,& \text{else},\end{array}\end{array}$$where $r_2$ represents the reward function of the speed error and $\Delta V_i$ represents the speed error between the ego vehicle and the front vehicle.

When the speed error between the ego vehicle and the front vehicle is exactly 0, the intelligent agent receives the maximum reward value. If the speed error equals 0.5 m/s, the feedback value is set to 0. To prevent excessive penalties, if the speed error exceeds a certain threshold, the reward and penalty values are set to fixed values.

Abrupt changes in the vehicle’s motion state within a short period of time can lead to considerable discomfort for both drivers and passengers. The key factors influencing comfort are the magnitude of deceleration and the rate of change in deceleration. This paper computes the rate of acceleration change $r_3$ using the following equation:$$\begin{array}{}\text{(12)}& r_3=\underset{\Delta t⟶0}{\lim }\frac{\Delta a}{\Delta t},\end{array}$$where the change in acceleration per unit of time is represented as $\Delta a$. $r_3$ represents the reward function of abrupt accelerations.

When an AV is on the road, it needs to respond promptly to changes in the motion state of the preceding vehicle, such as sudden acceleration or abrupt deceleration. To assess the vehicle’s responsiveness, a reward function, denoted as $r_4$, is designed to express the vehicle response sensitivity index as described in the following equation:$$\begin{array}{}\text{(13)}& r_4=\begin{array}{cc}2-a_i,& a_i>2,\\ 0,& -3.5\le a_i\le 2,\\ 3.5,& a_i<-3.5.\end{array}\end{array}$$

In summary, the reward function used in the experiment comprises a total of four components, as presented in the following equation:$$\begin{array}{}\text{(14)}& r_{\text{total}}={\omega }_1{r}_1+{\omega }_2{r}_2+{\omega }_3{r}_3+{\omega }_4{r}_4,\end{array}$$where ${\omega }_1$, ${\omega }_2$, ${\omega }_3$, and ${\omega }_4$ represent the weight coefficients of the reward function. In this study, the weight coefficients were set to 1, 0.2, 0.2, and 0.1, respectively.

In this study, it is assumed that the vehicle controllers are distributed within each vehicle, and there is a communication foundation between vehicles. Each controller can use the obtained preceding vehicle information to obtain environmental status information based on vehicle communication and body sensors. Therefore, the desired acceleration during the car following process is as follows:$$\begin{array}{c}\text{(15)}& ut=K_p{v}_{n-1}t-v_{n}t+K_i\Delta x_{n}t-v_{n}t-L+K_d{a}_{n-1}t-a_{n}t,\end{array}$$where $ut$ is the desired acceleration.

The training data for DDPG-PID consist of experiences collected by the agent interacting with the environment, including the current state, action taken, reward received, next state, and a terminal flag indicating whether the episode has ended. These experiences are stored in a replay buffer. During training, mini-batches of experiences are randomly sampled from the replay buffer to update the policy network and value network. This process optimizes the agent’s behavior (parameters of PID), enabling it to perform effectively in continuous action space tasks.

3.5. Comparison of Simulation Results

This chapter primarily focuses on the design of the upper controller for longitudinal control of AVs. To assess the actual tracking performance of the upper control algorithm under varying input speeds and acceleration values, it is validated through the use of different operating conditions. To showcase the complexity, an enhanced speed curve is constructed, which includes uniform linear motion, rapid acceleration, and sudden deceleration. The complex desired speed curve developed is depicted in Figure 2.

[figure(s) omitted; refer to PDF]

The control performance of the DDPG-PID control algorithm proposed in this paper was evaluated under these operating conditions using a complex expected speed curve and compared the control performance with traditional PID controllers and adaptive PID algorithms (gain coefficients have been added to the coefficients of PID [29]). The simulation results are depicted in Figures 3, 4, and 5.

[figure(s) omitted; refer to PDF]

The figures display the longitudinal speed, longitudinal speed tracking error, and longitudinal acceleration tracking outcomes during the testing of the desired speed curve. Based on the analysis of the simulation results for the aforementioned operating conditions, it is evident that, under this specific working condition, the DDPG-PID controller exhibits superior control performance throughout the entire longitudinal tracking process of the AV. In addition, it can be seen from the acceleration graph that DDPG-PID is more sensitive to changes in acceleration and has smaller fluctuations compared to other models, resulting in better driving stability.

To compare the errors of models above more intuitively, this paper selects root mean squared error (RMSE), mean absolute error (MAE), and $R^2$ coefficient, to measure the tracking effect. The calculation formulas are shown in formulas (15) to (17). In addition, the maximum, minimum, and mean values of the errors shown in Figure 4 were statistically analyzed, and the results are shown in Tables 2, 3, 4, and 5.$$\begin{array}{}\text{(16)}& RMSE=\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{{\widehat{y}}_i-y_i^{\prime }}^2,}\text{(17)}& MAE=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{\widehat{y}}_i-y_i^{\prime },\text{(18)}& R^2=1-\frac{{{\displaystyle \sum }}_i{y_i-y_i^{\prime }}^2}{{{\displaystyle \sum }}_i{{\overline{y}}_i-y_i^{\prime }}^2},\end{array}$$where $\widehat{y_i}$ represents the predicted value of the model, ${y_i}^{\prime }$ represents the desired value, $\overline{y_i}$ represents the average value of desired value, and $n$ represents the amount of data.

Table 2

Comparison of speed tracking errors.

Model	RMSE (m/s)	MAE (m/s)	R2
PID	0.47	0.38	0.995
Adaptive PID	0.35	0.28	0.997
DDPG-PID	0.23	0.18	0.998

Abbreviations: MAE, mean absolute error; RMSE, root mean squared error.

Table 3

Comparison of acceleration tracking errors.

Model	RMSE (m/s2)	MAE (m/s2)	MAPE (m/s2)
PID	0.46	0.28	0.849
Adaptive PID	0.25	0.14	0.955
DDPG-PID	0.24	0.12	0.957

Abbreviations: DDPG, deep deterministic policy gradient; MAE, mean absolute error; PID, proportional–integral-derivative; RMSE, root mean squared error.

Table 4

Numerical characteristics of speed tracking errors.

Model	Max (m/s)	Min (m/s)	Average (m/s)
PID	0.58	−1.16	−0.07
Adaptive PID	0.60	−0.90	−0.08
DDPG-PID	0.40	−0.90	0.01

Abbreviations: DDPG, deep deterministic policy gradient; Max, maximum error; Min, minimum error; PID, proportional–integral-derivative.

Table 5

Numerical characteristics of acceleration tracking errors.

Model	Max (m/s2)	Min (m/s2)	Average (m/s2)
PID	1.30	−1.20	−0.046
Adaptive PID	1.20	−0.94	0.033
DDPG-PID	1.10	−0.80	0.010

Abbreviations: DDPG, deep deterministic policy gradient; Max, maximum error; Min, minimum error; PID, proportional–integral-derivative.

By reviewing the table above, it is evident that compared to other models, DDPG-PID has improved RMSE, MAE, and $R^2$ by 34.3%, 35.7%, and 0.1%, respectively, in terms of speed tracking error. Regarding acceleration tracking error, compared with alternative models, DDPG-PID has improved by 4.0%, 14.3%, and 0.2% in three indicators, respectively. In addition, when examining the statistical values of speed and acceleration tracking errors, DDPG-PID significantly reduces both maximum and average errors compared to other models.

4. Lower Controller of AV

The lower controller manages the vehicle’s acceleration based on desired acceleration output from the upper controller. It is essential to design an efficient lower controller that converts the desired acceleration input into the necessary accelerator or brake pedal positions for the vehicle. The structure is shown in Figure 6.

[figure(s) omitted; refer to PDF]

For the PID controller, the error of the acceleration $a_{r}t$ generated by the desired acceleration of the upper control input $a_{des}t$ and the actual acceleration feedback $a_{act}t$ from the vehicle model is used as the input, and the output is the control acceleration $a_{con}t$. Therefore, the driving and braking switching strategy will be designed based on the control acceleration. The input acceleration error can be calculated according to the following equation:$$\begin{array}{}\text{(19)}& a_{r}t=a_{des}t-a_{act}t.\end{array}$$

The output acceleration can be calculated as follows:$$\begin{array}{}\text{(20)}& a_{\text{con}}t=k_p{a}_{r}t+k{\displaystyle \sum }{a}_{r}t+k_d{a}_{r}t-a_{r}t-1.\end{array}$$

4.1. Reverse Drive System Model

During the driving process of the vehicle, while being driven by the driving force $F_t$, it also receives resistance including air resistance $F_k$, rolling friction $F_f$, slope resistance $F_p$, and acceleration resistance $F_j$, which can be calculated as follows:$$\begin{array}{}\text{(21)}& F_f=f{F}_n=fmg\cos \theta ,\text{(22)}& F_k=\frac{1}{2}{C}_D\rho A{V}^2,\text{(23)}& F_p=G\sin \theta =mg\sin \theta ,\text{(24)}& F_j=\delta ma=\delta m\frac{du}{dt},\end{array}$$where $C_D$ is the air resistance coefficient; $\rho$ is the air density, normal can be taken as 1.293 $g/\mathrm{L}$; $A$ is the windward area of the automatic vehicle; $V$ is the speed; $F_N$ is positive pressure; $f$ is the rolling resistance coefficient. $G$ is the gravity of the car; $m$ is the curb weight of the vehicle; $\theta$ is the angle of the slope of the driving road, where $\theta =\arctan 1/s$. $\delta$ is the rotational mass conversion coefficient, generally greater than 1; $du/dt$ is the acceleration of the vehicle.

According to Newton’s laws of motion, the force acting on the car in the x-direction is as follows:$$\begin{array}{}\text{(25)}& F_t-F_k-F_f-F_p-F_j=m{a}_{des}.\end{array}$$

Among them, $\sum f$ represents the sum of resistances.

Without considering the deformation of the vehicle’s transmission system, the driving force can be calculated as follows:$$\begin{array}{}\text{(26)}& F_t=\frac{T_{tq}{i}_g{i}_o{\eta }_T}{r}\tau \frac{n_t}{n_e}.\end{array}$$

By using constants $k$ to facilitate the observation of the overall parameters, the following formulas can be obtained:$$\begin{array}{}\text{(27)}& F_t=\frac{T_{tq}{i}_g{i}_o{\eta }_T}{r}\tau \frac{n_t}{n_e}=k{T}_{tq},\text{(28)}& k=\frac{i_g{i}_o{\eta }_T}{r}\tau \frac{n_t}{n_e},\end{array}$$where $T_{tq}$ is the output torque of the engine; $i_g$ is the transmission gear ratio; $i_0$ is main reducer reduction ratio; ${\eta }_T$ is mechanical efficiency of the transmission system; $r$ is the radius of the wheel; $\tau n_t/n_e$ is the torque characteristic coefficient of the hydraulic torque converter.

According to the drive/brake switching control logic, when the throttle is used for control and is not affected by the braking system, equation (29) for the desired engine torque $T_{tq}$ can be obtained as:$$\begin{array}{}\text{(29)}& T_{tq}=\frac{\sum f+m{a}_{des}}{k}=\frac{1/2C_D\rho A{V}^2+fmg\cos \theta +mg\sin \theta +\delta mdu/dt}{k}.\end{array}$$

The relationship between the output torque $T_{tq}$, throttle opening $thr$, and engine speed $n$ of the engine can be calculated as follows:$$\begin{array}{}\text{(30)}& thr=f{T}_{tq},n.\end{array}$$

The throttle opening can be obtained based on the output torque and speed value of the engine.

4.2. Reverse Braking System Model

When regulating the AV’s braking system, it is influenced by multiple factors, including air resistance, rolling resistance, slope resistance, braking force, and the counteractive forces generated by conventional systems. Under the collective influence of these resistances, the desired acceleration signal is utilized to determine the brake master cylinder pressure based on the longitudinal dynamics model, thereby achieving vehicle deceleration. Throughout the braking procedure, the vehicle’s throttle remains closed at zero. In accordance with Newton’s laws of motion, the force equilibrium relationship in the axial direction is as follows:$$\begin{array}{}\text{(31)}& -F_z-F_1-F_k-F_f-F_p=m{a}_{des},\end{array}$$where $F_z$ is the braking force; $F_1$ is the counteractive force.

When subjected to braking pressure, the wheels that typically rotate forward experience a braking force acting in the opposite direction to the road surface. From the torque balance relationship, equation (32) can be obtained:$$\begin{array}{}\text{(32)}& F_z=\frac{T_{\mu }}{r}=\frac{T_{bf}+T_{br}}{r},\end{array}$$where $T_{bf}$ is the friction torque of the front wheel brake; $T_{br}$ is the friction torque of the rear wheel brake.

When the desired acceleration $a_{des}$ output from the upper layer, the friction torque acting on the wheel brake can be calculated as follows:$$\begin{array}{}\text{(33)}& T_{\mu }=-F_1-F_k-F_f-F_p-m{a}_{des}\cdot r.\end{array}$$

When the road adhesion limit is less than or equal to unity, the relationship between vehicle braking force and braking pressure is presented as follows:$$\begin{array}{}\text{(34)}& F_z=K_b{P}_b.\end{array}$$

Moreover, the relationship between braking torque and brake master cylinder pressure is as follows:$$\begin{array}{}\text{(35)}& T_{\mu }=K_b{P}_{b}R,\end{array}$$where $K_b$ is the efficiency factor; $P_b$ is the braking pressure of the vehicle; $R$ is the radius of action of the vehicle brake.

The relationship between desired acceleration and brake master cylinder pressure can be obtained as follows:$$\begin{array}{}\text{(36)}& P_b=\frac{-F_1-F_k-F_f-F_p-m{a}_{des}\cdot r}{K_{b}R}.\end{array}$$

By establishing a reverse braking system model in the lower controller, the conversion of brake master cylinder pressure can be obtained, which serves as the output of the lower controller for AVs during braking control.

4.3. Drive Brake Switching Logic

Frequently toggling between throttle and brake control pedals can result in severe damage to the vehicle’s transmission system and other automotive components, which can also lead to vibrations and performance discrepancies within the power system, ultimately causing a suboptimal driving experience for passengers. To mitigate these issues, a mode switch with a width of $\Delta h$ is introduced, aimed at reducing the need for frequent transitions between braking and driving modes. The width of this switch band is set at 0.2, with upper and lower limits corresponding to intermediate control acceleration values.

The force of the vehicle during driving without being affected by the driving brake system is as follows:$$\begin{array}{}\text{(37)}& F-\sum Fv=m{a}_1,\text{(38)}& a_1=\frac{F-\sum Fv}{m},\end{array}$$where $F$ is the driving force exerted by the road surface on the vehicle; $a_1$ is the acceleration of the vehicle; $m$ is the mass; $\sum Fv$ is the sum of various resistances experienced by an AV.

When the desired acceleration is greater than $a_1+0.1$, the lower controller should adopt drive control; When the desired acceleration output by the upper controller at the current vehicle speed is less than $a_1-0.1$, the lower controller should adopt braking control; When the desired acceleration value output by the upper controller $a_1$ meets the relationship $a_1-0.1<a<a_1+0.1$, the control system does not use either drive control or brake control mode. The AV decelerates under the combined action of system reverse drag, rolling resistance, air resistance, and slope resistance.

4.4. Model Simulation Verification

To verify the actual tracking ability and response delay of the designed lower controller to the desired acceleration output from the upper controller, a step operating condition was designed and simulation verification was conducted with Carsim and Simulink. The lower level PID control parameters were set to $k_p=0.5;k_i=0.22;k_d=0.01$. Set the initial speed to 40 km/h and the input desired acceleration to 0. When t = 10 s, the input desired acceleration value is 1 m/s². When t = 20 s, the desired acceleration value is 0. Therefore, the overall desired acceleration is a step function with an amplitude of 1 m/s² and a duration of 10 s. The results are shown in Figures 7 and 8.

[figure(s) omitted; refer to PDF]

Analysis of the step response acceleration simulation results reveals that at 10 s, there is a deviation in the actual acceleration. However, this deviation quickly recovers to a normal tracking state within a very short time. Around 15 s, a sudden change occurs in the actual acceleration, attributed to abrupt engine speed variations during a gear shift in the vehicle’s acceleration process. Overall, the controller exhibits relative stability in the tracking of desired acceleration and speed.

5. Joint Simulation Verification of Control System

In this paper, Simulink and Carsim platforms are combined to carry out the overall control algorithm and model verification. The effectiveness and reasonableness of the designed control algorithm are tested on a good road surface with a road adhesion coefficient of 0.85, with a simulation step size of 0.01 s. Typical traffic scenarios in actual road traffic (steady-state following conditions and start-up following conditions) are selected for simulation testing to verify the effectiveness of the algorithm, and the specific design is as follows:

The steady-state following scenario set in this section is intended to start with the vehicle cruising at a constant speed of 60 km/h. There is a front vehicle traveling at 50 km/h 100 m ahead of this lane, and its speed varies from 40 km/h to 60 km/h. During this period, ego vehicle always follows the front vehicle.

Figures 9, 10, and 11 show that after the speed change of the front vehicle, ego vehicle also undergoes acceleration and deceleration to follow the front vehicle, and the changes of the motion state of ego vehicle are in line with the simulated driving scenario. At the beginning, the vehicle was traveling at a constant speed of 60 km/h with the acceleration kept at zero. At this time, it was far away from the front vehicle. After monitoring the front vehicle at 20 s, it starts to do deceleration. During this period, the speed changes gently, and after a small fluctuation of acceleration, it immediately returns to its original tracking state. At about 40 s, the two vehicles have the same speed, and the acceleration is 0. At this time, the distance between the cars remains unchanged. In the subsequent deceleration and acceleration, although there is a delay in ego vehicle, the amplitude is small and there is no extreme value. Ultimately, it can maintain the same speed as the front vehicle, indicating that the controller has a good effect on the speed control. The acceleration of ego vehicle is basically consistent with the desired acceleration, and both positive and negative changes in acceleration correspond to the changes in vehicle speed, with its maximum peak value reaching 0.37 m/s² and minimum peak value −0.45 m/s², indicating that the acceleration and deceleration of the vehicle are relatively comfortable. Moreover, the control of the distance between the two vehicles is relatively gentle, greater than the safe distance, which indicates that the controller designed in this paper has good following performance.

[figure(s) omitted; refer to PDF]

The driving scenario proposed in this section is when the vehicle is about to start at the beginning, or when the vehicle is restarted after braking in a traffic congested road. Assuming that at the beginning of the state, ego vehicle and the front vehicle remain stationary, with a distance of 6 m between the two vehicles. At the initial moment, the front vehicle starts to start and accelerates to 30 km/h within 5 s. At the same time, ego vehicle also reacts accordingly and follows the front vehicle to start and accelerate, ultimately maintaining the state of driving at a constant speed with a certain distance between the vehicles.

According to the results shown in Figures 12, 13, and 14, at the initial moment, ego vehicle and the front vehicle remained stationary, with a distance of 6 m between the two vehicles. The front vehicle suddenly accelerates, and the acceleration process lasts for 5 s, with the speed rising from 0 to 30 km/h. Afterward, the speed of the front vehicle remains stable at 30 km/h and travels at a constant speed. After monitoring a change in the state of the front vehicle, the vehicle reacts quickly and immediately does acceleration in 1–2 s. At this point, the acceleration suddenly surges to 1.54 m/s². Although there is a small fluctuation, the amplitude is small, and the response time was short, with an immediate deceleration to 1.44 m/s². After that, the acceleration stabilizes, and then the workshop distance continues to increase. Until 7.2 s, the speed of ego vehicle and the front vehicle becomes the same, after that the speed remains unchanged. At this time, the distance between the vehicles reaches a maximum of 21.63 m, and the acceleration also returns to zero. The actual acceleration and desired acceleration curves are basically consistent, and their magnitude changes are in line with the trend of vehicle speed. The overall switching is relatively smooth, which shows that the controller is more effective in starting the vehicle to follow the driving.

[figure(s) omitted; refer to PDF]

6. Conclusion

This paper established a hierarchical longitudinal control system based on the DDPG and PID control algorithms, aiming to address challenges in the current state of longitudinal control technology for AVs. The conclusions can be drawn as follows:

1. A longitudinal control algorithm was designed under a hierarchical control structure. We use Carsim to model the longitudinal dynamics of the vehicle and generate the required acceleration through the upper layer DDPG and PID algorithms. The research results indicate that compared with other models, DDPG-PID has improved RMSE, MAE, and R2 by 34.3%, 35.7%, and 0.1%, respectively, demonstrating excellent performance in handling sudden changes in perception information in driving scenarios.

2. The lower level control adopts a PID-based driving and braking switching control strategy. Through joint simulation verification using Carsim and Simulink, the longitudinal control method has demonstrated robustness and adaptability in various simulation scenarios, affirming its reliability under different road conditions.

3. The designed method effectively controls the speed and following distance of the vehicle, provides a feasible longitudinal control solution for the AV, and provides effective technical support for the safe operation of autonomous driving.

Future research directions include further optimizing algorithm performance, considering more complex traffic scenarios and vehicle behavior models, to enhance the robustness of the system in complex environments. In addition, exploring real-world experiments under actual road conditions to validate simulation results will help improve the reliability of research results. These efforts will help further promote and apply autonomous driving technology.

Funding

This work was supported by the National Natural Science Foundation of China (72371019).

Acknowledgments

This work was supported by the National Natural Science Foundation of China (72371019). We did not use AI in our paper, including text and images generated by artificial intelligence.

References

[1] A. Simorgh, A. Marashian, A. Razminia, "Adaptive PID Control Design for Longitudinal Velocity Control of Autonomous Vehicles," Proceedings of the 2019 International Conference on Control, Instrumentation and Automation (ICCIA),DOI: 10.1109/iccia49288.2019.9030856, .

[2] S. H. HosseinNia, I. Tejado, B. M. Vinagre, V. Milanés, J. Villagrá, "Low Speed Control of an Autonomous Vehicle Using a Hybrid Fractional Order Controller," Proceedings of the 2nd International Conference on Control, Instrumentation and Automation, pp. 116-121, DOI: 10.1109/icciautom.2011.6356641, .

[3] S. Frank, P. Herrmann, N. Schwarz, "An Improved Multi-Object Adaptive Cruise Control Approach," Proceedings of the 10th IFAC Symposium on Intelligent Autonomous Vehicles, .

[4] S. Graffione, C. Bersani, R. Sacile, E. Zero, "Model Predictive Control of a Vehicle Platoon," Proceedings of the 2020 IEEE 15th International Conference of System of Systems Engineering (SoSE), pp. 513-518, .

[5] S. Moon, H. J. Kang, K. Yi, "Multi-vehicle Target Selection for Adaptive Cruise Control," vol. no. 11, pp. 1325-1343, DOI: 10.1080/00423114.2010.499952, .

[6] G. K. Mekala, N. R. Sarugari, A. Chavan, "Speed Control in Longitudinal Plane of Autonomous Vehicle Using MPC," Proceedings of the 2020 IEEE International Conference for Innovation in Technology (INOCON),DOI: 10.1109/inocon50539.2020.9298213, .

[7] J. Zhao, Z. Yu, X. Yang, Z. Gao, W. Liu, "Short Term Traffic Flow Prediction of Expressway Service Area Based on STL-OMS," Physica A: Statistical Mechanics and Its Applications, vol. 595,DOI: 10.1016/j.physa.2022.126937, 2022.

[8] L. Yang, D. Sun, F. Xie, J. Zhu, "Study of Autonomous Platoon Vehicle Longitudinal Modeling," Proceedings of the IET International Conference on Intelligent and Connected Vehicles (ICV 2016), .

[9] M. Jin, "Improvement of Road-Following Intelligent Speed Control Based on Road Curvature," Proceedings of the 2013 Third International Conference on Intelligent System Design and Engineering Applications, pp. 870-873, .

[10] W. Yuan, M. Yang, C. Wang, B. Wang, "Longitudinal and Lateral Coupling Model Based End-To-End Learning for Lane Keeping of Self-Driving Car," Proceedings of the 4th International Conference on Cognitive Systems and Information Processing (ICCSIP), pp. 425-436, .

[11] Ó. Pérez-Gil, R. Barea, E. López-Guillén, "Dqn-Based Deep Reinforcement Learning for Autonomous Driving," Proceedings of the 21st International Workshop of Physical Agents (WAF 2020), .

[12] X. He, C. Fei, Y. Liu, K. Yang, X. Ji, "Multi-Objective Longitudinal Decision-Making for Autonomous Electric Vehicle: A Entropy-Constrained Reinforcement Learning Approach," Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece,DOI: 10.1109/itsc45102.2020.9294736, .

[13] A. Coppola, A. Petrillo, R. Rizzo, S. Santini, "Adaptive Cruise Control for Autonomous Electric Vehicles Based on Q-Learning Algorithm," 2021 AEIT International Annual Conference (AEIT),DOI: 10.23919/aeit53387.2021.9627059, .

[14] Z. Sun, R. Wang, Q. Ye, Z. Wei, B. Yan, "Investigation of Intelligent Vehicle Path Tracking Based on Longitudinal and Lateral Coordinated Control," IEEE Access, vol. 8, pp. 105031-105046, DOI: 10.1109/access.2020.2994437, 2020.

[15] Y. Yao, N. Ma, C. Wang, Z. Wu, C. Xu, J. Zhang, "Research and Implementation of Variable-Domain Fuzzy PID Intelligent Control Method Based on Q-Learning for Self-Driving in Complex Scenarios," Mathematical Biosciences and Engineering, vol. 20 no. 3, pp. 6016-6029, DOI: 10.3934/mbe.2023260, 2023.

[16] L. Nie, J. Guan, C. Lu, H. Zheng, Z. Yin, "Longitudinal Speed Control of Autonomous Vehicle Based on a Self-Adaptive PID of Radial Basis Function Neural Network," IET Intelligent Transport Systems, vol. 12 no. 6, pp. 485-494, DOI: 10.1049/iet-its.2016.0293, 2018.

[17] M. Fayyazi, M. Abdoos, D. Phan, "Real-Time Self-Adaptive Q-Learning Controller for Energy Management of Conventional Autonomous Vehicles," Expert Systems with Applications, vol. 222,DOI: 10.1016/j.eswa.2023.119770, 2023.

[18] C. Desjardins, B. Chaib-draa, "Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach," IEEE Transactions on Intelligent Transportation Systems, vol. 12 no. 4, pp. 1248-1260, DOI: 10.1109/tits.2011.2157145, 2011.

[19] L. Song, J. Li, Z. Wei, K. Yang, E. Hashemi, H. Wang, "Longitudinal and Lateral Control Methods from Single Vehicle to Autonomous Platoon," Green Energy and Intelligent Transportation, vol. 2,DOI: 10.1016/j.geits.2023.100066, 2023.

[20] L. Guo, Z. Ren, P. Ge, J. Chang, "Advanced Emergency Braking Controller Design for Pedestrian Protection Oriented Automotive Collision Avoidance System," The Scientific World Journal, vol. 2014, 2014.

[21] L. Wang, X. Fang, S. Duan, X. Liao, "PID Controller Based on Memristive CMAC Network," Abstract and Applied Analysis, vol. 2013,DOI: 10.1155/2013/510238, 2013.

[22] Y. Chen, D. Li, H. Zhong, R. Zhao, "The Method for Automatic Adjustment of AGV’s PID Based on Deep Reinforcement Learning," Journal of Physics: Conference Series, vol. 2320 no. 1,DOI: 10.1088/1742-6596/2320/1/012008, 2022.

[23] L. Yu, Y. Chen, S. Chen, Y. Zhang, H. Zhang, C. Liu, "Numerical Analysis of the Performance of a PID-Controlled Air Curtain for Fire-Induced Smoke Confinement in a Tunnel Configuration," Fire Safety Journal, vol. 141,DOI: 10.1016/j.firesaf.2023.103930, 2023.

[24] S. Wang, Y. Hu, Z. Liu, L. Ma, "Research on Adaptive Obstacle Avoidance Algorithm of Robot Based on DDPG-DWA," Computers & Electrical Engineering, vol. 109,DOI: 10.1016/j.compeleceng.2023.108753, 2023.

[25] Y. Qian, S. Feng, W. Hu, W. Wang, "Obstacle Avoidance Planning of Autonomous Vehicles Using Deep Reinforcement Learning," Advances in Mechanical Engineering, vol. 14 no. 12,DOI: 10.1177/16878132221139661, 2022.

[26] P. Kuo, J. Hu, K. Chen, W. Chang, X. Chen, C. Huang, "Sequential Sensor Fusion-Based W-DDPG Gait Controller of Bipedal Robots for Adaptive Slope Walking," Advanced Engineering Informatics, vol. 57,DOI: 10.1016/j.aei.2023.102067, 2023.

[27] D. Li, O. Okhrin, "Modified DDPG Car-Following Model With a Real-World Human Driving Experience With CARLA Simulator," Transportation Research Part C: Emerging Technologies, vol. 147,DOI: 10.1016/j.trc.2022.103987, 2023.

[28] L. Ye, K. Ling, Q. Han, "DIMDP: A Driving Intention-Based MDP Service Migration Model Under MEC/MSCN Architecture," Mobile Information Systems, vol. 2022,DOI: 10.1155/2022/4988266, 2022.

[29] K. J. Åström, T. Hägglund, Advanced PID Control, 2005.