Model Predictive Control with Variational Autoencoders for Signal Temporal Logic Specifications

Abstract

This paper presents a control strategy synthesis method for dynamical systems with differential constraints, emphasizing the prioritization of specific rules. Special attention is given to scenarios where not all rules can be simultaneously satisfied to complete a given task, necessitating decisions on the extent to which each rule is satisfied, including which rules must be upheld or disregarded. We propose a learning-based Model Predictive Control (MPC) method designed to address these challenges. Our approach integrates a learning method with a traditional control scheme, enabling the controller to emulate human expert behavior. Rules are represented as Signal Temporal Logic (STL) formulas. A robustness margin, quantifying the degree of rule satisfaction, is learned from expert demonstrations using a Conditional Variational Autoencoder (CVAE). This learned margin is then applied in the MPC process to guide the prioritization or exclusion of rules. In a track driving simulation, our method demonstrates the ability to generate behavior resembling that of human experts and effectively manage rule-based dilemmas.

1. Introduction

Robotics is increasingly permeating diverse sectors, spanning both civilian and industrial applications, and is becoming integral to everyday life. Service robots are now prevalent in public spaces, interacting with individuals and delivering services. Within the field of robotics, autonomous driving emerges as a particularly dynamic area, garnering extensive research attention.

In robotics, adherence to rules varies from basic collision avoidance in navigation scenarios to compliance with complex traffic regulations in autonomous driving. These rules, established primarily for safety, must generally be upheld by robots while executing their tasks. However, it is essential to acknowledge that not all rules carry equal importance. Depending on the context, some rules may need to be prioritized over others or even disregarded. For example, in autonomous driving, scenarios may necessitate breaching certain rules—such as lane changes in dense traffic, decisions at yellow traffic lights, or crossing double yellow lines to avoid obstacles. These situations compel robots to make intricate decisions regarding rule compliance, presenting significant challenges in determining appropriate control inputs.

Model Predictive Control (MPC) stands out as a robust approach for autonomous control, recognized for its capabilities in online trajectory optimization [1]. The core principle of MPC involves identifying optimal control inputs to minimize a predefined cost function, considering both inputs and anticipated future outputs. This method integrates an objective function characterizing the desired robot behavior and constraints mitigating undesirable actions. The efficacy of MPC is well-documented across diverse applications, such as the full-body control of humanoid robots [2,3,4].

Designing effective MPC controllers remains a significant challenge. Experienced operators can adeptly manage robots, yet encoding such expertise into MPC parameters is complex. For instance, expert drivers in autonomous driving must make continuous, complex decisions, such as whether to decelerate or change lanes in response to slow-moving vehicles. However, finding the appropriate MPC parameters to handle such varied scenarios is complex and computationally intensive.

Recently, imitation learning has emerged as a promising solution for robotic learning challenges [5,6]. This approach derives near-optimal control strategies directly from human expert demonstrations, eliminating the need for manual policy or cost function design. Imitation learning excels in capturing complex policy functions that balance multiple considerations [7], learning the importance of various factors from expert behaviors to enable robots to replicate human actions. However, despite its advantages, imitation learning does not inherently ensure performance reliability. In scenarios where safety rules like collision avoidance are crucial, imitation learning may not consistently yield control actions that comply with essential safety norms, underscoring the paramount importance of rule adherence for robot safety and human protection.

In this paper, we address a control synthesis problem within a framework of prioritized rules, building on our previous research [8]. We assumed inherent rule priorities and aimed to design a controller that accounts for these priorities to manage dilemmas effectively. Our methodology is grounded in the MPC framework, which, unlike purely deep learning-based approaches, integrates each rule as a constraint, thereby enhancing performance reliability.

We represent these rules using Signal Temporal Logic (STL) [9,10], a formalism allowing the precise specification of desired system behaviors, commonly applied in robotic task specifications [11,12,13,14,15]. STL is particularly suited for describing properties of real-valued signals in dense time scenarios, making it ideal for real-world robotic applications.

Instead of explicitly determining rule priorities, we adopted a learning approach to identify minimal acceptable levels of rule satisfaction, informed by expert demonstrations. This approach diverges from our earlier work [8] by employing a Conditional Variational Autoencoder (CVAE) [16]. This technique helps discern essential rules and decide on adherence levels, facilitating selective compliance rather than strict obedience to all rules. The use of a CVAE is justified by its efficiency in handling uncertainties within data, providing a more effective solution compared to Gaussian process regression methods used in previous work [8].

Our hybrid approach combines deep learning with traditional MPC, guiding robots to emulate expert human behaviors in complex decision-making scenarios.

2. Related Work

Extensive research has explored trajectory optimization and Model Predictive Control (MPC) within the framework of temporal logic specifications, particularly Linear Temporal Logic (LTL). Mixed-integer linear programming (MILP) has been employed to generate trajectories for continuous systems subject to finite-horizon LTL specifications [17,18]. Wolff et al. [19] extended this approach by encoding general LTL formulas into MILP constraints, accommodating infinite runs with periodic structures. Additionally, Cho et al. [20] investigated optimal path planning under synthetically co-safe LTL specifications, utilizing a sampling-tree and two-layered structure.

Recent advancements have integrated Signal Temporal Logic (STL) within MPC frameworks. Raman et al. [21] structured MPC to facilitate control synthesis from STL specifications using MILP, allowing for the calculation of open-loop control signals that adhere to both finite and infinite horizon STL properties while maximizing robust satisfaction. Sadigh et al. [22] introduced a novel STL variant incorporating probabilistic predicates to address uncertainties in predictive models, thereby enhancing safety assessments under uncertainty. Mao et al. [23] proposed a solution to handle complex temporal requirements formalized in STL specifications within the Successive Convexification algorithmic framework. This approach retains the expressiveness of encoding mission requirements with STL semantics while avoiding combinatorial optimization techniques such as MILP.

The integration of MPC with machine learning techniques has been pursued to address system identification challenges within MPC contexts [24,25,26]. Lenz et al. [24] applied deep learning within MPC to derive task-specific controls for complex activities such as robotic food cutting. Carron et al. [25] presented a model-based control approach that utilizes data gathered during operation to improve the model of a robotic arm and thereby enhance the tracking performance. Their scheme is based on inverse dynamic feedback linearization and a data-driven error model, integrated into an MPC formulation. Lin et al. [26] compared deep reinforcement learning (DRL) and MPC for Adaptive Cruise Control (ACC) design in car-following scenarios.

Efforts have also been made to address the types of dilemmas introduced in our work. Tumova et al. [27] and Castro et al. [28] examined scenarios where not all LTL rules can be satisfied in path planning, seeking paths that minimally violate these rules. However, their approaches require predetermined weights among rules, contrasting with our method that learns directly from expert demonstrations. Urban driving dilemmas were specifically addressed by Lee et al. [29], who applied inverse reinforcement learning to capture expert driving strategies.

Imitation learning is emerging as a promising approach to robotic learning problems and has been widely applied to autonomous driving. Policies for autonomous vehicles have been learned from image or video datasets through Convolutional Neural Networks (CNNs) [30,31]. Schmerling et al. [32] utilized a Conditional Variational Autoencoder (CVAE) framework to reason about interactions between vehicles in traffic-weaving scenarios, producing multimodal outputs. Additionally, some studies have applied learning approaches to MPC, where certain parameters of MPC are learned from data [8,33]. Reinforcement learning has also been considered for autonomous driving, using CNNs to encode visual information [34].

3. Preliminaries

3.1. System Model

We consider a continuous-time dynamical system described by the following differential equation:

(1)${\dot{\mathrm{x}}}_t=f({\mathrm{x}}_t,{\mathrm{u}}_t),$

where ${\mathrm{x}}_t\in \mathcal{X}\subset {\mathbb{R}}^{n_{\mathrm{x}}}$ represents the state vector, ${\mathrm{u}}_t\in \mathcal{U}\subset {\mathbb{R}}^{n_{\mathrm{u}}}$ denotes the control input, and f is a smooth (continuously differentiable) function with respect to its arguments. Through employing a predefined time step $dt$, the continuous system in Equation (1) can be discretized as follows:

(2)${\mathrm{x}}_{n+1}=f({\mathrm{x}}_n,{\mathrm{u}}_n),$

where n represents the discrete time step, defined as $n=\lfloor t/dt\rfloor$, and ${\mathrm{x}}_0$ denotes the initial state. For a fixed horizon H, let $\mathbf{x}({\mathrm{x}}_n,{\mathbf{u}}^{H,n})$ denote a trajectory generated from the state ${\mathrm{x}}_n$ with the control inputs ${\mathbf{u}}^{H,n}=\{{\mathrm{u}}_n,\dots ,{\mathrm{u}}_{n+H-1}\}$.

A signal is defined as a sequence of states and control inputs:

(3)$\xi ({\mathrm{x}}_n,{\mathbf{u}}^{H,n})=({\mathrm{x}}_n,{\mathrm{u}}_n),\dots ,({\mathrm{x}}_{n+H-1},{\mathrm{u}}_{n+H-1}).$

In addition to the definition provided in Equation (3), we use the notation $\xi \left(n\right)$ to represent a signal starting from the discrete time step n, with a slight abuse of notation.

3.2. Signal Temporal Logic

Signal Temporal Logic (STL) is a formalism used to specify properties of real-valued, dense-time signals, and is extensively applied in the analysis of continuous and hybrid systems [9,10]. A predicate within an STL formula is defined as an inequality of the form $\mu \left(\xi \right(t\left)\right)>0$, where $\mu$ is a function of the signal $\xi$ at time t. The truth value of the predicate $\mu$ is determined by the condition $\mu \left(\xi \right(t\left)\right)>0$.

An STL formula is composed of boolean and temporal operations on these predicates. The syntax of STL formulae $\phi$ is defined recursively as follows:

(4)$\phi ::=\mu |\neg \mu |\phi \wedge \psi |{\mathrm{G}}_{[a,b]}\psi |\phi {\mathrm{U}}_{[a,b]}\psi ,$

where $\phi$ and $\psi$ are STL formulas, $\mathrm{G}$ denotes the globally operator, and $\mathrm{U}$ represents the until operator.

The validity of an STL formula $\phi$ with respect to a signal $\xi$ at time t is defined inductively as follows:

(5)$(\xi ,t)\vDash \mu \iff \mu \left(\xi \right(t\left)\right)>0$

(6)$(\xi ,t)\vDash \neg \mu \iff \neg \left(\right(\xi ,t)\vDash \mu )$

(7)$(\xi ,t)\vDash \phi \wedge \psi \iff (\xi ,t)\vDash \phi \wedge (\xi ,t)\vDash \psi$

(8)$(\xi ,t)\vDash \phi \vee \psi \iff (\xi ,t)\vDash \phi \vee (\xi ,t)\vDash \psi$

(9)$(\xi ,t)\vDash {\mathrm{G}}_{[a,b]}\phi \iff \forall t^{\prime }\in [t+a,t+b],(\xi ,t^{\prime })\vDash \phi$

(10)$(\xi ,t)\vDash \phi {\mathrm{U}}_{[a,b]}\psi \iff \exists t^{\prime }\in [t+a,t+b]\mathrm{s}.\mathrm{t}.(\xi ,t^{\prime })\vDash \psi \wedge \forall t^{″}\in [t,t^{\prime }],(\xi ,t^{″})\vDash \phi .$

The notation $(\xi ,t)\vDash \phi$ indicates that the signal $\xi$ satisfies the STL formula $\phi$ at time t. For example, $(\xi ,t)\vDash {\mathrm{G}}_{[a,b]}\phi$ implies that $\phi$ holds for the signal $\xi$ throughout the interval from $t+a$ to $t+b$. In discrete-time systems, STL formulas are evaluated over discrete time intervals.

One significant advantage of Signal Temporal Logic (STL) is its associated metric, known as the robustness degree, which quantifies how well a given signal $\xi$ satisfies an STL formula $\phi$. The robustness degree is defined as a real-valued function of the signal $\xi$ and time t, calculated recursively using the following quantitative semantics:

(11)$\begin{array}{cccc}\hfill & {\rho }^{\mu }(\xi ,t)\hfill & \hfill =& \mu \left(\xi \right(t\left)\right),\hfill \end{array}$

(12)$\begin{array}{cccc}\hfill & {\rho }^{\neg \mu }(\xi ,t)\hfill & \hfill =& -\mu \left(\xi \right(t\left)\right),\hfill \end{array}$

(13)$\begin{array}{cccc}\hfill & {\rho }^{\phi \wedge \psi }(\xi ,t)\hfill & \hfill =& min({\rho }^{\phi }(\xi ,t),{\rho }^{\psi }(\xi ,t)),\hfill \end{array}$

(14)$\begin{array}{cccc}\hfill & {\rho }^{\phi \vee \psi }(\xi ,t)\hfill & \hfill =& max({\rho }^{\phi }(\xi ,t),{\rho }^{\psi }(\xi ,t)),\hfill \end{array}$

(15)$\begin{array}{cccc}\hfill & {\rho }^{{\mathrm{G}}_{[a,b]}\phi }(\xi ,t)\hfill & \hfill =& \underset{t^{\prime }\in [t+a,t+b]}{min}{\rho }^{\phi }(\xi ,t^{\prime }),\hfill \end{array}$

(16)$\begin{array}{cccc}\hfill & {\rho }^{\phi {\mathrm{U}}_{[a,b]}\psi }(\xi ,t)\hfill & \hfill =& \underset{t^{\prime }\in [t+a,t+b]}{max}(min({\rho }^{\psi }(\xi ,t^{\prime }),\hfill \end{array}$

(17)$\begin{array}{cccc}\hfill & \hfill & & \underset{t^{″}\in [t,t^{\prime }]}{min}{\rho }^{\phi }(\xi ,t^{″}))).\hfill \end{array}$

Following our previous study [8], we introduce the notation $(\xi ,t)\vDash (\phi ,r)$ to indicate that the signal $\xi$ satisfies the STL formula $\phi$ at time t with a robustness slackness r, defined as

(18)$(\xi ,t)\vDash (\phi ,r)\equiv {\rho }^{\phi }(\xi ,t)>r.$

Equation (18) asserts that the signal $\xi$ satisfies $\phi$ with at least the minimum robustness degree r. The robustness slackness r serves as a margin for the satisfaction of the STL formula $\phi$. As r increases, the constraints on the signal $\xi$ to satisfy $\phi$ at time t become more stringent, while smaller values of r imply more relaxed constraints. Notably, when $r<0$, it allows for the violation of $\phi$.

4. Problem Formulation

This study aimed to solve a control synthesis problem using Signal Temporal Logic (STL) formulas [8]. Let $\mathit{\phi }=[{\phi }_1,\dots ,{\phi }_N]$ represent a set of STL formulas, with their conjunction denoted as $\overline{\mathit{\phi }}={\phi }_1\wedge \dots \wedge {\phi }_N$. We define a cost function J over the state and control spaces, where $J(\mathbf{x},\mathbf{u})$ measures the cost associated with a trajectory $\mathbf{x}$ and control sequence $\mathbf{u}$. The control synthesis problem under STL for Model Predictive Control (MPC) is formulated as follows.

Problem 1.

Given a system model as described in (2) and an initial state ${\mathrm{x}}_0$, with a planning horizon of length H, determine the control input sequence ${\mathrm{u}}^{H,t}$ at each time step t that minimizes the cost function $J(\mathbf{x}({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),{\mathbf{u}}^{H,t})$ while ensuring that the conjunction of STL formulas $\overline{\mathit{\phi }}$ is satisfied:

(19) $\begin{array}{cccc}\hfill & \underset{{\mathbf{u}}^{H,t}}{\mathit{minimize}}\hfill & \hfill & J(\mathbf{x}({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),{\mathbf{u}}^{H,t})\hfill \\ \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & (\xi ({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),t)\vDash \overline{\mathit{\phi }}.\hfill \end{array}$

While this strict formulation ensures compliance with the STL formulas, our primary objective is to develop a control sequence that incorporates flexibility in rule compliance. To this end, we introduce robustness slackness values, denoted by $\mathit{r}=[r_1,\dots ,r_N]$, which quantify the degree to which each STL formula is satisfied. In incorporating these robustness values, the MPC problem can be reformulated as follows [8].

Problem 2.

Given the system model specified in (2), an initial state ${\mathrm{x}}_0$, and a horizon length H, compute the control input sequence ${\mathrm{u}}^{H,t}$ at each time step t by solving the following optimization problem:

(20) $\begin{array}{cccc}\hfill & \underset{{\mathbf{u}}^{H,t}}{\mathit{minimize}}\hfill & \hfill & J(\mathbf{x}({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),{\mathbf{u}}^{H,t})\hfill \\ \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & (\xi ({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),t)\vDash ({\phi }_1,r_1),\hfill \\ \hfill & \hfill & & ⋮\hfill \\ \hfill & \hfill & & (\xi ({\mathrm{x}}_t,{\mathbf{u}}^{H,t}),t)\vDash ({\phi }_N,r_N).\hfill \end{array}$

This enhanced formulation allows for a more flexible management of STL constraints, effectively addressing scenarios where it is not feasible to fully satisfy all STL formulas. The robustness slackness values are derived from expert demonstrations, based on the assumption that these experts have accurately assessed the priority and required compliance level of each rule. This learning is achieved through a deep learning approach.

5. Proposed Method

The proposed framework, illustrated in Figure 1, synergizes learning techniques with STL constraints to refine MPC, enabling it to more accurately mimic human expert behavior. By leveraging expert demonstrations, we learn robustness slackness values, which define the margins of rule compliance. A Conditional Variational Autoencoder (CVAE) [16] is utilized to estimate these robustness slackness values in novel scenarios.

In incorporating the robustness slackness values obtained through the learning process, the MPC method, designed under STL constraints, generates control sequences that respect the specified rules with a certain level of flexibility. To manage the nonlinear differential constraints characteristic of dynamical systems, we employ linearized models. Although this approach may introduce some approximation errors, it remains effective for practical applications.

Figure 1 presents an overview of the proposed learning-based MPC framework. Expert demonstrations are used to learn the lower bounds of robustness, referred to as robustness slackness, through a deep learning approach. These learned values inform the MPC method, which then calculates control sequences that take into account the STL rules.

5.1. Feature Description

We introduce a feature function, denoted as $\varphi$, which transforms a signal into a feature vector, mapping from the combined state and control spaces into the feature space: $\varphi :{\mathbb{R}}^{n_x+n_u}\to {\mathbb{R}}^{n_f}$.

As illustrated in Figure 2, the control of the ego vehicle, $V_{ego}$, is influenced by six nearby vehicles located in adjacent lanes. These vehicles are collectively referred to as $V_{near}=\{V_{lf},V_{lr},V_{cf},V_{cr},V_{rf},V_{rr}\}$, where the subscripts denote the relative position to the ego vehicle: left-front (lf), left-rear (lr), center-front (cf), center-rear (cr), right-front (rf), and right-rear (rr).

The feature vector $\varphi$ includes the following components:

• Distances to each of the nearby vehicles ($d_{lf},d_{lr},d_{cf},d_{cr},d_{rf},d_{rr}$);

• Lateral deviation from the lane center ($d_{dev}$);

• Heading angle relative to the lane direction (${\theta }_{dev}$).

5.2. Learning Robustness Slackness from Demonstration

We consider a set of M demonstrated signals, denoted by $\Xi ={\left\{{\xi }^i\right\}}_{i=1}^M$, where each signal ${\xi }_n^i=({\mathrm{x}}_n^i,{\mathrm{u}}_n^i)$ comprises the state ${\mathrm{x}}_n^i$ and control input ${\mathrm{u}}_n^i$ at time step n. The robustness degree $r^{i,j}$ is defined as the minimum value observed from the current time step n to the future time step $n+H-1$ for the demonstration ${\xi }^i$:

(21)$r_n^{i,j}=\underset{m\in [n,n+H-1]}{min}{\rho }^{{\phi }_j}({\xi }^i,m),$

where H denotes the control horizon length. The robustness degree $r^{i,j}$ serves as the robustness slackness for the signal over the horizon length H, starting from ${\xi }_n^i$, indicating the minimum permissible lower bound of robustness within this timeframe.

Figure 3 illustrates a demonstrated trajectory in a track driving scenario, depicting both the robustness degree and its lower bound for the time horizon H. The rule considered involves maintaining the first (lowest) lane, defined by the STL formula ${\phi }_{\mathrm{lane}}=(y\le y_{\mathrm{upper}})\wedge (y\ge y_{\mathrm{lower}})$, where y represents the vertical position of the vehicle, and $y_{\mathrm{upper}}$ and $y_{\mathrm{lower}}$ are the upper and lower lane boundaries, respectively. An obstacle (or other vehicle), depicted as a striped black box, necessitates a lane change to proceed. The figure illustrates the variance between the robustness degree values and their corresponding lower bounds.

From the demonstrated signals $\Xi$, let $D^j$ represent the outputs from Equation (21) corresponding to the STL formula ${\phi }_j$. We define $\Phi$ as the set of feature vectors derived from the demonstrated signals $\Xi$ (see Figure 2). For a new input feature $\varphi$, the CVAE network, depicted in Figure 4, predicts the lower bound of the robustness degree for the horizon H, representing the learned robustness slackness $\mathit{r}=[r_1,\dots ,r_N]$.

Our CVAE model comprises the following three parameterized functions:

• The recognition model $q_{\nu }\left(Z\right|\varphi )$ approximates the distribution of the latent variable Z based on the input features. This is modeled as a Gaussian distribution, $\mathcal{N}({\mu }_{\nu }\left(\varphi \right),{\mathsf{\Sigma }}_{\nu }\left(\varphi \right))$, where ${\mu }_{\nu }$ and ${\mathsf{\Sigma }}_{\nu }$ represent the mean and covariance determined by the network.

• The prior model $p_{\theta }\left(Z\right|\varphi )$ assumes a standard Gaussian distribution, $\mathcal{N}(0,I)$, simplifying the structure of the latent space.

• The generation model $p_{\theta }\left(\mathit{r}\right|Z,\varphi )$ calculates the likelihood of robustness slackness based on the latent variable Z and the input feature $\varphi$.

Both the recognition model $q_{\nu }\left(Z\right|\varphi )$ and the generation model $p_{\theta }\left(\mathit{r}\right|Z,\varphi )$ are implemented as multi-layer perceptrons.

The training of our CVAE is guided by the Evidence Lower Bound (ELBO) loss function, initially formulated as

(22)${\mathbb{E}}_{q_{\nu }\left(Z\right|\varphi )}[log{p}_{\theta }\left(r\right|Z,\varphi )]-{\mathcal{D}}_{KL}(q_{\nu }\left(Z\right|\varphi )\parallel p_{\theta }\left(Z\right|\varphi )).$

To better accommodate the specific requirements of our application, we adapted the ELBO function and define the loss function as follows:

(23)$-\sum _{i=1}^{N}log\left(p_{\theta }\left(r_i\right|Z,\varphi )\right)+\lambda ·{\mathcal{D}}_{KL}\left(\mathcal{N}({\mu }_{\nu }\left(\varphi \right),{\mathsf{\Sigma }}_{\nu }\left(\varphi \right))\parallel \mathcal{N}(0,I)\right),$

where $r_i$ represents an element of the robustness slackness $\mathit{r}$, and $\lambda$ is a scaling factor used to balance the terms. The Kullback–Leibler divergence (${\mathcal{D}}_{KL}$) measures the divergence between two probability distributions. We set $\lambda =1$ and optimize parameters $\nu$ and $\theta$ by minimizing this loss function.

5.3. Model Predictive Control Synthesis

Previous work, such as that by Raman et al. [21], has shown that MPC optimization with STL constraints can be formulated as a mixed-integer linear program (MILP). This method introduces two encoding strategies: one that focuses on satisfying STL formulas and another, termed ‘robustness-based encoding’, that considers the robustness degree of the STL formulas. In our problem formulation, we manage each STL formula according to its defined robustness slackness using the robustness-based encoding method.

Let $C_{{\phi }_j,r_j}$ denote the encoded constraints for the STL formula ${\phi }_j$ with robustness slackness $r_j$. The combined encoded constraints are formulated as follows:

(24)$\begin{array}{cc}\hfill z^{\phi }=\underset{j=1}{\overset{N}{\bigwedge }}{z}^{{\phi }_j}⟺& z^{\phi }\le z^{{\phi }_j},\hfill \end{array}$

(25)$\begin{array}{cc}\hfill & z^{\phi }\ge 1-N+\sum _j^N{z}^{{\phi }_j},\hfill \end{array}$

where $z^{\phi },z^{{\phi }_j}\in [0,1]$ are Boolean variables, with $z^{\phi }$ representing the satisfaction of all STL constraints and $z^{{\phi }_j}$ representing the satisfaction of an individual STL formula ${\phi }_j$. Note that $z^{{\phi }_j}=1$ only if ${\rho }^{{\phi }_j}-r_j>0$; otherwise, $z^{{\phi }_j}=0$.

The proposed algorithm is outlined in Algorithm 1. We extended our previous work [8] by incorporating a deep learning network approach. Inputs to the algorithm include a set of STL formulas ${\phi }_1,\dots ,{\phi }_N$, the time of interest $\tau =[t_0,t_1]$, the discretization time step $dt$, a control horizon H, an initial signal state ${\xi }_{init}$, and demonstrated signals $\Xi$.

Initially, feature vectors and robustness slackness values (the lowest robustness degree for the horizon H) are pre-computed from demonstrations (line 1). The closed-loop algorithm, which determines the optimal strategy at each time step, runs over the time interval $\tau =[t_0,t_1]$. Nonlinear dynamics are linearized with respect to the current signal state (line 4). The robustness slackness of the STL formula ${\phi }_j$ for the input feature $\varphi \left({\xi }_{cur}\right)$ is predicted using the trained CVAE network (line 6). The predicted robustness slackness for each STL formula is denoted as $r_j$. Based on the updated robustness slackness $r_j$, each STL formula ${\phi }_j$ is converted into mixed-integer programming constraints $C_{{\phi }_j,r_j}$ using the robustness-based encoding method (line 7), where $C_{{\phi }_j,r_j}$ consists of binary variables and linear predicates. In considering all STL constraints, dynamic constraints, and past trajectories, the optimal control sequence is computed over the time horizon H using a user-defined cost function (line 11). This procedure is repeated for the entire time interval $\tau$.

Algorithm 1 Variational Autoencoder-based Controller Synthesis under STL Constraints

1:. $\Phi ,D^j\leftarrow$ Initialize($\Xi$)   2:. ${\xi }_{cur}\leftarrow {\xi }_{init},{\xi }_{past}\leftarrow \varnothing$   3:. for $t=t_0:dt:t_1$ do   4:.      $f_{lin}\leftarrow$ Linearize($f,{\xi }_{cur}$)   5:.      for $j=1:1:N$ do   6:.           $r_j\leftarrow$ CVAE($\varphi \left({\xi }_{cur}\right)$)   7:.           $C_{{\phi }_j,r_j}\leftarrow$ EncodeSTLConstraints(${\phi }_j,r_j$)   8:.      end for   9:.      $C_{STL}\leftarrow C_{{\phi }_1,r_1}\wedge \dots \wedge C_{{\phi }_N,r_N}$ 10:.      $C\leftarrow C_{STL}\wedge f_{lin}\wedge [\xi (t_0,\cdots ,t-dt)={\xi }_{past}]$ 11:.      ${\mathbf{u}}^{H,t}\leftarrow$ Optimize($J\left({\xi }^H\right),C$) 12:.      ${\mathrm{x}}_{next}=f({\mathrm{x}}_{cur},{\mathbf{u}}^{H,t}\left(t\right))$ 13:.      ${\xi }_{past}\leftarrow \left[{\xi }_{past}{\xi }_{cur}\right]$ 14:.      ${\xi }_{cur}\leftarrow ({\mathrm{x}}_{next},{\mathbf{u}}^{H,t}\left(t\right))$ 15:. end for

6. Experimental Results

The proposed algorithm was implemented in a Python (version 3.10) environment, utilizing PyTorch (version 2.2.1) [35] for the deep learning components and Gurobi [36] as the optimization engine for MPC. Simulation experiments were conducted on a system equipped with an AMD R7-7700 processor and an RTX 4080 Super GPU. The Gurobi tool enabled solving the proposed MPC problem in approximately 0.11 s.

We conducted realistic simulations using the Next Generation Simulation (NGSIM) dataset [37] and the highD dataset [38], assuming that the drivers in these datasets possessed a certain level of expertise, making them suitable for “expert driver” demonstrations in our proposed approach. In the proposed method, obstacles were set as nearby vehicles. For generating training data, we utilized a combination of 70% data from the highD dataset and 30% from the NGSIM dataset. Data points from the NGSIM dataset that involved vehicles deviating from the track or causing collisions were excluded or modified. Additionally, data with normal speeds but no lane changes were partially removed to ensure a diverse set of training scenarios.

The CVAE network was trained with the following hyperparameters: a batch size of 64, a learning rate of 0.001, and 100 epochs. The future time horizon H was set to 16.

6.1. System Description

We modeled the dynamics of the vehicles on the track using a unicycle model. The state of the system at time t is described by ${\mathrm{x}}_t={[x_t,y_t,{\theta }_t,v_t]}^T$, where $x_t$ and $y_t$ represent the vehicle’s position, ${\theta }_t$ denotes the heading angle, and $v_t$ indicates the linear velocity. The control inputs are ${\mathrm{u}}_t={[w_t,a_t]}^T$, with $w_t$ as the angular velocity and $a_t$ as the acceleration. The vehicle dynamics are expressed as follows:

$\begin{array}{c}\hfill {\dot{x}}_t=v_{t}cos\left({\theta }_t\right),{\dot{y}}_t=v_{t}sin\left({\theta }_t\right),{\dot{\theta }}_t={\kappa }_1{w}_t,{\dot{v}}_t={\kappa }_2{a}_t,\end{array}$

where ${\kappa }_1$ and ${\kappa }_2$ are constants. To facilitate the optimization process, we linearize the dynamics around a reference point $\widehat{\mathrm{x}}={[\widehat{x},\widehat{y},\widehat{\theta },\widehat{v}]}^T$. The resulting linear system is derived as a first-order Taylor approximation of the nonlinear dynamics, given by

$\begin{array}{c}\hfill {\mathrm{x}}_{n+1}=A_n{\mathrm{x}}_n+B_n{\mathrm{u}}_n+C_n,\end{array}$

where the matrices $A_n$, $B_n$, and $C_n$ are defined as

$\begin{array}{cc}\hfill A_n& =\left[\begin{array}{cccc}1& 0& -\widehat{v}sin\left(\widehat{\theta }\right)dt& cos\left(\widehat{\theta }\right)dt\\ 0& 1& \widehat{v}cos\left(\widehat{\theta }\right)dt& sin\left(\widehat{\theta }\right)dt\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],\hfill \\ \hfill B_n& =\left[\begin{array}{cc}0& 0\\ 0& 0\\ {\kappa }_1\widehat{v}dt& 0\\ 0& {\kappa }_{2}dt\end{array}\right],\hfill \\ \hfill C_n& =\left[\begin{array}{c}\widehat{v}sin\left(\widehat{\theta }\right)\widehat{\theta }dt\\ -\widehat{v}cos\left(\widehat{\theta }\right)\widehat{\theta }dt\\ 0\\ 0\end{array}\right].\hfill \end{array}$

6.2. Rule Description

We formulated five distinct rules as STL formulas. The definitions of the rules $\mathit{\phi }=[{\phi }_1,\dots ,{\phi }_5]$ are as follows:

• Lane keeping (right): ${\phi }_1=y_t\ge y_{l,min}$;

• Lane keeping (left): ${\phi }_2=y_t\le y_{l,max}$;

• Collision avoidance (front vehicle):

  ${\phi }_3=(x_t\le x_{c,min})\vee (x_t\ge x_{c,max})\vee (y_t\le y_{c,min})\vee (y_t\ge y_{c,max});$

• Speed limit: ${\phi }_4=v_t\le v_{th}$;

• Slow down before the front vehicle:

  ${\phi }_5=(v_t\le v_u){\mathrm{U}}_{[t_a,t_b]}(x_t\le x_{c,min}).$

In these formulations, $t_a$ and $t_b$ are set to 6 and 12, respectively.

Figure 5 illustrates the driving environment used to describe these STL rules. Note that in this figure, the ego vehicle is depicted in blue, the preceding vehicle in orange, and other vehicles in gray. The positions $x_t$ and $y_t$ and velocity $v_t$ correspond to the ego vehicle. The boundaries of the preceding vehicle at the x-y coordinates are denoted by $x_{c,min}$, $x_{c,max}$, $y_{c,min}$, and $y_{c,max}$. Similarly, $x_{o,min}$, $x_{o,max}$, $y_{o,min}$, and $y_{o,max}$ represent the boundaries of other vehicles except the preceding one. The lane boundaries are denoted by $y_{l,min}$ and $y_{l,max}$, while the track boundaries are represented by $y_{t,min}$ and $y_{t,max}$.

Here, $v_{th}$ represents the speed limit threshold for rule ${\phi }_4$. The final rule, ${\phi }_5$, mandates that the ego vehicle decelerate when approaching a preceding vehicle in the same lane. Parameters $v_u$, $t_a$, and $t_b$ are specific to rule ${\phi }_5$.

6.3. Simulation Results

Figure 6 presents the predicted robustness slackness $\mathit{r}$ generated by the proposed CVAE network, alongside the control sequence produced by the MPC based on these predicted values. In the left subfigures indicating robustness slackness, negative degrees of satisfaction are marked with a red box.

In Figure 6a, the predicted robustness slackness suggests that rules ${\phi }_2$ and ${\phi }_5$ may be violated. It can be observed that the control sequence generated by the MPC results in the vehicle moving to the left lane (violating ${\phi }_2$) and accelerating in the presence of a preceding vehicle (violating ${\phi }_5$).

Figure 7 demonstrates the application of the proposed method in the NGSIM road environment. The figure illustrates four different scenes, showing the predicted robustness slackness and the corresponding vehicle movements for each situation. For the lane-keeping rules ${\phi }_1$ and ${\phi }_2$, if the robustness slackness value is less than or equal to a specified threshold (indicated by ‘threshold’ in the figure), it is evident that the ego vehicle attempts to change lanes. Conversely, if the robustness slackness value for ${\phi }_1$ and ${\phi }_2$ is greater than the threshold value, the proposed method may not initiate a lane change, depending on the specific situation (as illustrated in scene 4). Overall, the proposed method demonstrates the ability to drive efficiently—allowing the violation of some rules in certain situations—while maintaining safety in complex traffic conditions.

Collision experiments using the proposed approach were conducted across five test scenarios: two from the NGSIM dataset and three from the highD dataset. We compared the proposed method against five methods: LBMPC_STL [8], LSTM, TFN [39], and DQN [40]. The LSTM method employs a naive LSTM encoder–decoder framework for imitation learning, while TFN utilizes a Transformer network for imitation learning. In the DQN method, the Q-network is modeled as a four-layer multi-layer perceptron with 12 discrete actions and receives input features. The DQN model was trained until convergence was achieved (1,000,000 episodes).

A total of 100 experiments were conducted on various tracks and starting positions. Figure 8 and Figure 9 illustrate single trials from the collision experiments, demonstrating the performance of each algorithm under identical track conditions, time, and starting positions. The distance traveled by each method is indicated below each subfigure. Methods incorporating MPC (Proposed and LBMPC_STL) successfully arrived at the target area (the end of the track), whereas other methods failed due to collisions. Compared to the proposed method, LBMPC_STL exhibited deficiencies, such as the ego vehicle ending up on the lane border and being too close to the preceding vehicle.

Table 1 presents the number of successful trials for each method. The two methods with the highest number of successes for each scenario are highlighted in bold. The results clearly demonstrate that the proposed approach outperforms other methods in most test scenarios. DQN(1/2) refers to cases where half of the episodes (500,000 episodes) are used in the reinforcement learning stage. The average time steps for successful cases are shown in parentheses.

In the results presented in Table 1, reinforcement learning techniques (specifically DQN) exhibit a longer average time step compared to other methods due to the emphasis on stability in the design of the reward function. Additionally, there was no significant difference in average time steps (for successful cases) between the MPC techniques, including the proposed method, and the supervised learning techniques. Notably, the proposed method demonstrated a slightly shorter average time step compared to the other MPC technique, LBMPC_STL.

While the “average time step” cannot be an absolute criterion for evaluating the superiority of an algorithm’s performance, when combined with the “collision rate”, it indicates that the proposed method enables more stable and efficient autonomous driving compared to other methods.

Two key observations can be made from these results:

• Model Predictive Control (MPC) demonstrates superior safety performance compared to reinforcement learning (DQN) and imitation learning approaches (LSTM, TFN).

• The deep learning approach employed in the proposed method yields a better performance than the Gaussian process regression approach used in LBMPC_STL.

7. Conclusions

In this paper, we present a Model Predictive Control (MPC) method designed to manage dynamic systems while adhering to a set of Signal Temporal Logic (STL) rules. Unlike traditional approaches that enforce strict compliance with all rules, our method efficiently balances rule adherence by selectively disobeying certain rules to resolve dilemma situations where not all rules can be simultaneously satisfied.

The proposed method introduces the concept of robustness slackness, which represents the lower bound of the robustness degree, learned from expert demonstrations or data. By employing a Conditional Variational Autoencoder (CVAE) network, the controller adapts its behavior to prioritize different rules based on the context, emulating the decision-making processes of human experts.

Our contribution lies in the innovative approach of learning the satisfaction measure of rules using a deep-learning network, enabling robots to internalize and replicate the value systems of humans. This approach allows for more flexible and context-aware control, which is crucial for operating in complex and dynamic environments.

Author Contributions

Conceptualization, E.I., M.C. and K.C.; methodology, E.I. and M.C.; validation, K.C.; data curation, E.I. and K.C.; writing—original draft preparation, E.I. and M.C.; writing—review and editing, K.C.; visualization, E.I. and M.C.; supervision, K.C. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data available in a publicly accessible repository.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Table

Figure 1. Overview of the proposed learning-based MPC framework. Expert demonstrations are utilized to learn the lower bounds of robustness, referred to as robustness slackness, through a deep learning approach. The learned values inform the MPC method, which then computes control sequences considering the STL rules.

Figure 2. Description of the ego vehicle and nearby vehicles in a track driving scenario. The ego vehicle ([Forumla omitted. See PDF.]) is shown in blue. The diagram includes up to six nearby vehicles positioned in front and behind, across the left, center, and right lanes relative to the ego vehicle.

Figure 3. Demonstration in a track driving environment showing (a) the robustness degree and (b) its predicted lower bound. Trajectories with values less than zero are shaded in red.

Figure 4. The CVAE network used to predict the robustness slackness.

Figure 5. Driving environment illustrating the defined STL rules [Forumla omitted. See PDF.].

Figure 6. Snapshots of the proposed method applied to the NGSIM dataset.

Figure 7. Illustration of the proposed method’s performance in NGSIM road environments. The figure depicts four different scenes, showing the predicted robustness slackness and the corresponding vehicle movements for each situation.

Figure 8. Snapshots of the collision experiments in the highD environment (Proposed, LBMPC_STL, LSTM).

Figure 9. Snapshots of the collision experiments in the highD environment (TFN, DQN).

Table 1

Number of successful trials in collision experiments.

	Proposed	LBMPC_STL	LSTM	TFN	DQN	DQN(1/2)
testset 1 (NGSIM-US101)	88 (180.1)	84 (189.6)	75 (186.5)	80 (183.4)	82 (199.5)	77 (205.5)
testset 2 (NGSIM-I80)	86 (175.4)	81 (180.8)	71 (176.6)	76 (174.2)	79 (187.1)	74 (193.2)
testset 3 (highD)	89 (124.3)	86 (129.1)	75 (125.5)	80 (119.9)	83 (147.3)	74 (156.8)
testset 4 (highD)	91 (131.6)	90 (131.6)	76 (130.8)	86 (135.2)	88 (148.8)	74 (159.2)
testset 5 (highD)	95 (136.1)	93 (139.8)	82 (138.7)	90 (134.3)	91 (159.3)	81 (165.7)

References

1. Camacho, E.F.; Alba, C.B. Model Predictive Control; Springer Science & Business Media: Berlin, Germany, 2013.

2. Dantec, E.; Naveau, M.; Fernbach, P.; Villa, N.; Saurel, G.; Stasse, O.; Taix, M.; Mansard, N. Whole-body model predictive control for biped locomotion on a torque-controlled humanoid robot. Proceedings of the IEEE-RAS International Conference on Humanoid Robots; Ginowan, Japan, 28–30 November 2022; pp. 638-644.

3. Kong, N.J.; Li, C.; Council, G.; Johnson, A.M. Hybrid iLQR model predictive control for contact implicit stabilization on legged robots. IEEE Trans. Robot.; 2023; 39, pp. 4712-4727. [DOI: https://dx.doi.org/10.1109/TRO.2023.3308773]

4. Le Cleac’h, S.; Howell, T.A.; Yang, S.; Lee, C.Y.; Zhang, J.; Bishop, A.; Schwager, M.; Manchester, Z. Fast contact-implicit model predictive control. IEEE Trans. Robot.; 2024; 40, pp. 1617-1629. [DOI: https://dx.doi.org/10.1109/TRO.2024.3351554]

5. Jang, E.; Irpan, A.; Khansari, M.; Kappler, D.; Ebert, F.; Lynch, C.; Levine, S.; Finn, C. Bc-z: Zero-shot task generalization with robotic imitation learning. Proceedings of the Conference on Robot Learning; London, UK, 8–11 November 2021; pp. 991-1002.

6. Zare, M.; Kebria, P.M.; Khosravi, A.; Nahavandi, S. A survey of imitation learning: Algorithms, recent developments, and challenges. arXiv; 2023; arXiv: 2309.02473

7. Abbeel, P.; Ng, A.Y. Apprenticeship learning via inverse reinforcement learning. Proceedings of the International Conference on Machine Learning; Banff, AB, Canada, 4–8 July 2004.

8. Cho, K.; Oh, S. Learning-based model predictive control under signal temporal logic specifications. Proceedings of the IEEE International Conference on Robotics and Automation; Brisbane, QLD, Australia, 21–25 May 2018; pp. 7322-7329.

9. Maler, O.; Nickovic, D. Monitoring temporal properties of continuous signals. Proceedings of the FORMATS/FTRTFT; Grenoble, France, 22–24 September 2004; Volume 3253, pp. 152-166.

10. Donzé, A.; Maler, O. Robust satisfaction of temporal logic over real-valued signals. Proceedings of the FORMATS; Klosterneuburg, Austria, 8–10 September 2010; Volume 6246, pp. 92-106.

11. Fainekos, G.E.; Kress-Gazit, H.; Pappas, G.J. Temporal logic motion planning for mobile robots. Proceedings of the IEEE International Conference on Robotics and Automation; Barcelona, Spain, 18–22 April 2005.

12. Karaman, S.; Frazzoli, E. Complex mission optimization for multiple-UAVs using linear temporal logic. Proceedings of the IEEE American Control Conference; Seattle, WA, USA, 11–13 June 2008.

13. Wongpiromsarn, T.; Topcu, U.; Murray, R.M. Receding horizon temporal logic planning for dynamical systems. Proceedings of the IEEE Conference on Decision and Control; Shanghai, China, 15–18 December 2009.

14. Pant, Y.V.; Abbas, H.; Mangharam, R. Distributed Trajectory Planning for Multi-rotor UAVs with Signal Temporal Logic Objectives. Proceedings of the IEEE Conference on Control Technology and Applications; Trieste, Italy, 23–25 August 2022; pp. 476-483.

15. Meng, Y.; Fan, C. Signal temporal logic neural predictive control. IEEE Robot. Autom. Lett.; 2023; 8, pp. 7719-7726. [DOI: https://dx.doi.org/10.1109/LRA.2023.3315536]

16. Sohn, K.; Lee, H.; Yan, X. Learning structured output representation using deep conditional generative models. Proceedings of the Advances in Neural Information Processing Systems; Montreal, QC, Canada, 7–12 December 2015; Volume 28.

17. Karaman, S.; Sanfelice, R.G.; Frazzoli, E. Optimal control of mixed logical dynamical systems with linear temporal logic specifications. Proceedings of the IEEE Conference on Decision and Control; Cancun, Mexico, 9–11 December 2008.

18. Kwon, Y.; Agha, G. LTLC: Linear temporal logic for control. Proceedings of the Hybrid Systems: Computation and Control; St. Louis, MO, USA, 22–24 April 2008; pp. 316-329.

19. Wolff, E.M.; Topcu, U.; Murray, R.M. Optimization-based control of nonlinear systems with linear temporal logic specifications. Proceedings of the IEEE International Conference on Robotics and Automation; Hong Kong, China, 31 May–7 June 2014; pp. 5319-5325.

20. Cho, K. Learning-based path planning under co-safe temporal logic specifications. IEEE Access; 2023; 11, pp. 25865-25878. [DOI: https://dx.doi.org/10.1109/ACCESS.2023.3257185]

21. Raman, V.; Donzé, A.; Maasoumy, M.; Murray, R.M.; Sangiovanni-Vincentelli, A.; Seshia, S.A. Model predictive control with signal temporal logic specifications. Proceedings of the IEEE Conference on Decision and Control; Los Angeles, CA, USA, 15–17 December 2014; pp. 81-87.

22. Sadigh, D.; Kapoor, A. Safe control under uncertainty. arXiv; 2015; arXiv: 1510.07313

23. Mao, Y.; Acikmese, B.; Garoche, P.L.; Chapoutot, A. Successive convexification for optimal control with signal temporal logic specifications. Proceedings of the ACM International Conference on Hybrid Systems: Computation and Control; Milan, Italy, 4–6 May 2022; pp. 1-7.

24. Lenz, I.; Knepper, R.A.; Saxena, A. DeepMPC: Learning Deep Latent Features for Model Predictive Control. Proceedings of the Robotics: Science and Systems; Rome, Italy, 13–17 July 2015.

25. Carron, A.; Arcari, E.; Wermelinger, M.; Hewing, L.; Hutter, M.; Zeilinger, M.N. Data-driven model predictive control for trajectory tracking with a robotic arm. IEEE Robot. Autom. Lett.; 2019; 4, pp. 3758-3765. [DOI: https://dx.doi.org/10.1109/LRA.2019.2929987]

26. Lin, Y.; McPhee, J.; Azad, N.L. Comparison of deep reinforcement learning and model predictive control for adaptive cruise control. IEEE Trans. Intell. Veh.; 2020; 6, pp. 221-231. [DOI: https://dx.doi.org/10.1109/TIV.2020.3012947]

27. Kong, Z.; Jones, A.; Medina Ayala, A.; Aydin Gol, E.; Belta, C. Temporal logic inference for classification and prediction from data. Proceedings of the International Conference on Hybrid Systems: Computation and Control; Berlin, Germany, 15–17 April 2014; pp. 273-282.

28. Castro, L.I.R.; Chaudhari, P.; Tumova, J.; Karaman, S.; Frazzoli, E.; Rus, D. Incremental sampling-based algorithm for minimum-violation motion planning. Proceedings of the IEEE Conference on Decision and Control; Firenze, Italy, 10–13 December 2013; pp. 3217-3224.

29. Lee, S.; Seo, S. A learning-based framework for handling dilemmas in urban automated driving. Proceedings of the IEEE Conference on Robotics and Automation; Singapore, 29 May–3 June 2017; pp. 1436-1442.

30. Xu, H.; Gao, Y.; Yu, F.; Darrell, T. End-to-end learning of driving models from large-scale video datasets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Honolulu, HI, USA, 21–26 July 2017; pp. 2174-2182.

31. Codevilla, F.; Miiller, M.; López, A.; Koltun, V.; Dosovitskiy, A. End-to-end driving via conditional imitation learning. Proceedings of the IEEE International Conference on Robotics and Automation; Brisbane, QLD, Australia, 21–25 May 2018; pp. 1-9.

32. Schmerling, E.; Leung, K.; Vollprecht, W.; Pavone, M. Multimodal probabilistic model-based planning for human-robot interaction. Proceedings of the IEEE International Conference on Robotics and Automation; Brisbane, QLD, Australia, 21–25 May 2018; pp. 1-9.

33. Drews, P.; Williams, G.; Goldfain, B.; Theodorou, E.A.; Rehg, J.M. Aggressive deep driving: Model predictive control with a cnn cost model. arXiv; 2017; arXiv: 1707.05303

34. Jaritz, M.; de Charette, R.; Toromanoff, M.; Perot, E.; Nashashibi, F. End-to-end race driving with deep reinforcement learning. Proceedings of the IEEE International Conference on Robotics and Automation; Brisbane, QLD, Australia, 21–25 May 2018; pp. 2070-2075.

35. Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; Lerer, A. Automatic differentiation in PyTorch. Proceedings of the NIPS-W; Long Beach, CA, USA, 4–9 December 2017.

36. Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual. 2024; Available online: http://www.gurobi.com (accessed on 3 March 2024).

37. Alexiadis, V.; Colyar, J.; Halkias, J.; Hranac, R.; McHale, G. The next generation simulation program. Inst. Transp. Eng.; 2004; 74, 22.

38. Krajewski, R.; Bock, J.; Kloeker, L.; Eckstein, L. The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems. Proceedings of the IEEE International Conference on Intelligent Transportation Systems; Maui, HI, USA, 4–7 November 2018.

39. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems; Long Beach, CA, USA, 4–9 December 2017.

40. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; Riedmiller, M. Playing atari with deep reinforcement learning. arXiv; 2013; arXiv: 1312.5602