Integrated Sensing and Communication Target Detection Framework and Waveform Design Method Based on Information Theory

Abstract

Target detection is a core function of integrated sensing and communication (ISAC) systems. The traditional likelihood ratio test (LRT) target detection algorithm performs inadequately under low signal-to-noise ratio (SNR) conditions, and the performance of mainstream orthogonal frequency division multiplexing (OFDM) waveforms declines sharply in high-speed scenarios. To address these issues, an information-theory-based orthogonal time frequency space (OTFS)-ISAC target detection processing framework is proposed. This framework adopts the OTFS waveform as its fundamental signal. The target detection is implemented through a relative entropy test (RET) comparing echo signals against target presence/absence hypotheses. Furthermore, to enhance the system’s target detection capability, the iterative OTFS-ISAC waveform design (I-OTFS-WD) method which maximizes the relative entropy is proposed. This method utilizes the minorization-maximization (MM) algorithm framework and semidefinite relaxation (SDR) technique to transform the non-convex optimization problem into an iterative convex optimization problem for resolution. The simulation results demonstrate that, under sufficient sample conditions, the RET algorithm achieves a 9.12-fold performance improvement over LRT in low-SNR scenarios; additionally, the optimized waveform reduces the sample requirements of the RET algorithm by 40%, further enhancing the target detection capability of the OTFS-ISAC system.

1. Introduction

Modern communication and sensing systems tend to integrate functionalities to share infrastructure, improve resource utilization, and reduce consumption. Against this backdrop, the integrated sensing and communication (ISAC) system, which combines communication and sensing functionalities while sharing resources such as energy, spectrum, waveforms, and antennas, has garnered significant research attention in recent years [1,2,3,4,5,6]. The research on ISAC encompasses various aspects, including physical-layer system design, networking considerations, ISAC applications, waveform design, performance trade-offs, signal processing techniques, and other related topics [4,5].

Waveform is a key component of ISAC systems [5]. ISAC systems tend to transmit a single signal to simultaneously achieve communication and sensing functionalities, thereby improving spectral efficiency. Existing studies on ISAC waveforms are mostly focused on OFDM due to its simplicity in processing. However, under high Doppler conditions, both the communication and sensing performance of OFDM deteriorate sharply, making it difficult to meet the requirements of modern ISAC systems [7]. Orthogonal time frequency space (OTFS) modulation, a recently proposed scheme, operates in the Doppler-delay (DD) domain for modulation and demodulation [8,9,10]. Unlike OFDM, OTFS maps the transmitted signal to the DD domain. During the modulation phase, the DD domain-transmitted signal is transformed into the time–frequency (TF) domain via the Heisenberg transform and then transmitted. At the receiver, the received TF signal undergoes the Wigner transform to map it back to the DD domain, thereby recovering the transmitted signal. By doing so, OTFS minimizes the impact of time and frequency dispersion caused by high mobility, leading to more reliable communications in high-mobility conditions, such as those seen in vehicular networks and high-speed train communications. The characteristics of OTFS make fast time-varying channels sparse and separable in the DD domain, offering significant advantages for ISAC signal processing. Studies have shown that OTFS outperforms OFDM in high-mobility scenarios with lower error rates [9,10]. OTFS waveforms have also been applied to sensing systems, achieving error-free Doppler frequency estimation for high-speed targets [7,11,12,13,14]. Therefore, OTFS is considered an ideal waveform for ISAC systems.

Target detection is a core sensing functionality in ISAC systems. To enhance the target detection capability of ISAC systems, waveform design techniques can be employed to optimize transmitted signals. Traditional sensing waveform optimization methods typically aim to enhance specific aspects of system performance. For instance, ref. [15,16] designed waveforms with superior autocorrelation properties to improve target detection, and ref. [17] developed local ambiguity function shaping algorithms to minimize the weighted integral sidelobe level of the ambiguity function. Although these traditional optimization methods can enhance sensing systems’ detection performance, they do not fundamentally optimize the information retrieval capability.

Information theory is a critical tool in sensing waveform design [18,19]. The use of information-theoretic approaches in sensing waveform design can be traced back to [18], where conditional mutual information between random target responses and received signals served as the objective function. Building on this foundation, ref. [19] extended information-theoretic waveform design to multiple-input multiple-output (MIMO) radar systems to address extended targets. The primary advantage of using information theory in waveform design lies in its ability to evaluate the sensing systems’ information acquisition capability, which is directly linked to improving target recognition performance. Although information theory offers advantages for sensing waveform design, existing research applying information theory to target detection is limited. Ref. [20] employed relative entropy as the objective function for radar waveform optimization and developed a relative entropy-based detection framework. Despite this, no existing studies have specifically investigated information-theoretic waveform design tailored for OTFS-ISAC systems.

Given the advantages of ISAC systems, OTFS, and information theory, this work proposes an OTFS-ISAC target detection framework. To further improve the detection perforamce of this framework, a waveform design method is proposed to optimize the transmitting OTFS signal. The main contributions of this work are as follows:

• An OTFS-ISAC target detection framework is proposed. This framework employs OTFS waveform as its fundamental waveform, and relative entropy is the adopted criterion criterion for hypothesis testing in target detection. In this framework, the relative entropy test (RET) algorithm, which uses relative entropy to measure the divergence between echo signal distributions under target-present and target-absent hypotheses, is proposed. Comparing to the traditional likelihood ratio test (LRT) target detection algorithm, RET demonstrates significant advantages under low signal-to-noise ratio (SNR) conditions. The proposed RET algorithm is suitable for application in ISAC scenarios, particularly in low-SNR environments, such as long-range target detection and the monitoring of weak targets (e.g., small unmanned aerial vehicles).

• An iterative OTFS-ISAC waveform design (I-OTFS-WD) method is proposed. I-OTFS-WD extends the work of [20] to OTFS with additional quality of service (QoS) constraints for communication. To address the non-convex optimization problem in I-OTFS-WD, the minorization-maximization (MM) algorithm framework is utilized to convert the problem into an iterative generalized quadratically constrained quadratic program (QCQP) [21]. Additionally, the semidefinite relaxation (SDR) technique is employed to solve the generalized non-convex QCQP optimization problem [22]. The waveform optimized by I-OTFS-WD further enhances the target detection capability of the OTFS-ISAC framework. The proposed I-OTFS-WD algorithm can be applied to scenarios with even lower target echo SNR, such as unmanned aerial vehicles with stealth structures or coatings. In such cases, waveform optimization is required to enhance the performance of the OTFS-ISAC system.

The structure of the paper is outlined as follows: Section 2.1 and Section 2.2 introduce the principles of OTFS-ISAC. Section 2.3 describes the RET algorithm. Section 2.4 describes the details of I-OTFS-WD. Section 3 presents simulation results and corresponding discussions. Finally, Section 4 concludes with a summary and outlines future work prospects.

2. Materials and Methods

2.1. Basic Concepts of OTFS

In the OTFS model, the TF plane is discretized into a grid as follows:

(1)$\varpi =(nT,m\Delta f),n=0,\dots ,N-1,m=0,\dots ,M-1$

where T is the time sampling interval, and $\Delta f$ is the frequency sampling interval.

The modulated TF symbols ${\mathit{X}}_{tf}$ are transmitted over an OTFS frame with a duration of $\widehat{T}=NT$, occupying a bandwidth of $B=M\Delta f$. The sampling frequency of the signal is $f_s=B$.

Similarly, the DD plane is discretized into a grid as follows:

(2)$\vartheta =\left({\displaystyle \frac{k}{NT}},{\displaystyle \frac{l}{M\Delta f}}\right),k=0,\dots ,N-1,l=0,\dots ,M-1$

2.2. The Input–Output Relationship of OTFS-ISAC

The modulation and demodulation process of the OTFS signal is illustrated in Figure 1. The transmitted symbols ${\mathit{X}}_{dd}\in {\mathbb{C}}^{N\times M}$ in the DD domain are first converted into the TF symbols ${\mathit{X}}_{tf}\in {\mathbb{C}}^{N\times M}$ via the inverse symplectic finite Fourier transform (ISFFT). The ISFFT transformation formula is:

(3)${\mathit{X}}_{tf}[n,m]={\displaystyle \frac{1}{\sqrt{NM}}}\sum _{k^{\prime }=0}^{N-1}\sum _{l^{\prime }=0}^{M-1}{\mathit{X}}_{dd}[k,l]e^{j2\pi ({\displaystyle \frac{kn}{N}}-{\displaystyle \frac{lm}{M}})}$

${\mathit{X}}_{tf}\in {\mathbb{C}}^{N\times M}$ is transformed into the time-domain transmitted signal $s\left(t\right)$ through the Heisenberg transform. The Heisenberg transform formula is:

(4)$s\left(t\right)={\displaystyle \frac{1}{M}}\sum _{n=0}^{N-1}\sum _{m=0}^{M-1}{\mathit{X}}_{tf}[n,m]g_{tx}(t-nT)e^{j2\pi \Delta f(t-mT)}$

where $g_{tx}$ represents the rectangular pulse shape of the transmitted signal. In OTFS-ISAC, a cyclic prefix (CP) is added to ensure the integrity of the signal symbols. The signal $s\left(t\right)$, after undergoing RF front-end processing, is converted into an RF signal and transmitted via the antenna.

The target scattering point model in the DD domain is as follows:

(5)$h(\tau ,f_d)=\sum _{p=1}^P{\alpha }_p\delta (\tau -{\tau }_p)\delta (f_d-f_{d_p})$

where P, ${\alpha }_p$, ${\tau }_p$, and $f_{d_p}$ represent the total number of target scattering points, the scattering coefficient, the delay, and the Doppler frequency p-th scattering point, respectively. Based on $h(\tau ,f_d)$, the expression for the target echo signal is as follows:

(6)$\begin{array}{cc}\hfill r\left(t\right)& =\int \int h(\tau ,f_d)s(t-\tau )e^{j2\pi f_{d}t}d\tau d{f}_d+w\left(t\right)\hfill \\ \hfill & =\sum _{p=1}^P{\alpha }_{p}s(\tau -{\tau }_p)e^{j2\pi f_{d_p}t}+w\left(t\right)\hfill \end{array}$

where $w\left(t\right)$ is the additive white Gaussian noise (AWGN).

The transmitted RF signal, after being reflected by the target, returns to the receiver. After undergoing RF front-end processing, the received signal $r\left(t\right)$ is obtained. At the receiver, the CP is removed from $r\left(t\right)$, and the Wigner transform is applied to obtain the TF received signal ${\mathit{Y}}_{tf}\in {\mathbb{C}}^{N\times M}$. The Wigner transform is given by (7) and (8):

(7)${\mathit{Y}}_{tf}(t,f)=\int r\left(t^{\prime }\right)g_{rx}^{*}(t^{\prime }-t)e^{-j2\pi f{t}^{\prime }}d{t}^{\prime }$

(8)${\mathit{Y}}_{tf}[n,m]={\mathit{Y}}_{tf}(t,f){|}_{t=nT,f=m\Delta f}$

In (7), $g_{rx}\left(t\right)$ is the rectangular received pulse waveform. After applying the symplectic finite Fourier transform (SFFT) to ${\mathit{Y}}_{tf}$, the DD-domain-received symbols ${\mathit{Y}}_{dd}\in {\mathbb{C}}^{N\times M}$ are obtained. The SFFT transformation is defined as follows:

(9)${\mathit{Y}}_{dd}[k,l]={\displaystyle \frac{1}{\sqrt{NM}}}\sum _{k=0}^{N-1}\sum _{l=0}^{M-1}{\mathit{Y}}_{tf}[n,m]e^{-j2\pi ({\displaystyle \frac{kn}{N}}-{\displaystyle \frac{lm}{M}})}$

After discretization and simplification, the input–output relationship of OTFS-ISAC is expressed as follows:

(10)${\mathit{Y}}_{dd}[k,l]=\sum _{p=1}^P{\mathit{X}}_{dd}[{[k-k_p]}_N,{[l-l_p]}_M]\times H_{k_p,l_p}[k,l]+\mathit{W}(k,l)$

where $\mathit{W}\in {\mathbb{C}}^{N\times M}$ is the AWGN matrix, $k_p=f_{d_p}N/\Delta f$, $l_p={\tau }_{p}M/T$, ${[•]}_N$ denotes the modulo operation, and ${\mathit{H}}_{k_p,l_p}\in {\mathbb{C}}^{N\times M}$ is a factor matrix related to $k_p$ and $l_p$. The ($k,l$)-th element of ${\mathit{H}}_{k_p,l_p}$ is calculated as:

(11)${\mathit{H}}_{k_p,l_p}[k,l]={\displaystyle \frac{1}{N{M}^2}}(H_1(k_p,l_p,k,l)+H_2(k_p,l_p,k,l))e^{j2\pi f_{d_p}T}$

where $H_1(k_p,l_p,k,l)$ and $H_2(k_p,l_p,k,l)$ are expressed as:

(12)$H_1(k_p,l_p,k,l)=N\sum _{m=0}^{M-1}\sum _{m^{\prime }=0}^{M-1}{e}^{-j2\pi {\displaystyle \frac{l(m^{\prime }-m)}{M}}}\{\sum _{i^{\prime }=l_p}^{M-1}{e}^{j2\pi {\displaystyle \frac{m^{\prime }-m+f_{d_p}T}{M}}}\}$

(13)$H_2(k_p,l_p,k,l)=N{e}^{-j2\pi {\displaystyle \frac{k-k_p}{N}}}\sum _{m=0}^{M-1}\sum _{m^{\prime }=0}^{M-1}{e}^{-j2\pi {\displaystyle \frac{l(m^{\prime }-m)}{M}}}\sum _{i^{\prime }=0}^{l_p}{e}^{j2\pi {\displaystyle \frac{m^{\prime }-m+f_{d_p}T}{M}}}$

Furthermore, we perform vectorization on ${\mathit{H}}_{k_p,l_p}$, ${\mathit{X}}_{dd}$, ${\mathit{Y}}_{dd}$, and $\mathit{W}$. Let ${\mathit{h}}_{k_p,l_p}=\mathrm{vec}\left({\mathit{H}}_{k_p,l_p}\right)$, ${\mathit{x}}_{dd}=\mathrm{vec}\left({\mathit{X}}_{dd}\right)$, ${\mathit{y}}_{dd}=\mathrm{vec}\left({\mathit{Y}}_{dd}\right)$, and $\mathit{w}=\mathrm{vec}\left(\mathit{W}\right)$, respectively, where $\mathrm{vec}(•)$ represents column-wise vectorization; (9) can be expressed as:

(14)${\mathit{y}}_{dd}=\sum _{p=1}^P{\alpha }_p\mathbf{\Xi }(k_p,l_p)\mathit{F}(k_p,l_p){\mathit{x}}_{dd}+\mathit{w}$

where $\mathbf{\Xi }=\mathrm{diag}\left({\mathit{h}}_{k_p,l_p}\right)$, and $\mathit{F}(k_p,l_p)\in {\mathbb{C}}^{N\times M}$ is the permutation matrix. For $i=kM+l$, the $(i,j)$-th element of $\mathit{F}(k_p,l_p)$ can be expressed as:

(15)$\mathit{F}(k_p,l_p)[i,j]=\left\{\begin{array}{c}1,j={[k-k_p]}_{N}M+{[l-l_p]}_M\hfill \\ 0,otherwise\hfill \end{array}\right.$

To simplify the notation, let $\mathit{Q}(k_p,l_p)=\mathbf{\Xi }(k_p,l_p)\mathit{F}(k_p,l_p)$. Thus, (14) can be rewritten as:

(16)${\mathit{y}}_{dd}=\sum _{p=1}^P{\alpha }_p\mathit{Q}(k_p,l_p){\mathit{x}}_{dd}+\mathit{w}$

Defining $\mathit{G}=[\mathit{Q}(k_1,l_1),...,\mathit{Q}(k_P,l_P)]$, $\mathit{X}={\mathit{I}}_P\otimes {\mathit{x}}_{dd}$, and $\mathit{\alpha }={[{\mathit{\alpha }}_1,...,{\mathit{\alpha }}_P]}^{\mathrm{T}}$, (16) can be expressed as:

(17)${\mathit{y}}_{dd}=\mathit{GX}\mathit{\alpha }+\mathit{w}$

2.3. The RET Target Detection Algorithm for OTFS-ISAC

Relative entropy can characterize the similarity between the distributions of random variables, and thus can be used to construct target detection algorithms. Since the waveform design in this paper is based on the DD domain, the subscripts “$dd$” for ${\mathit{x}}_{dd}$ and ${\mathit{y}}_{dd}$ will be omitted for simplicity in the following discussion.

The binary hypothesis testing model for target detection in OTFS-ISAC is as follows:

(18)$\left\{\begin{array}{c}{\mathrm{H}}_0:\mathit{y}=\mathit{w}\hfill \\ {\mathrm{H}}_1:\mathit{y}=\mathit{GX}\mathit{\alpha }+\mathit{w}\hfill \end{array}\right.$

In this work, it is assumed that $\mathit{\alpha }$ is a mutually independent zero-mean complex Gaussian random vector with covariance matrix ${\mathit{R}}_{\alpha }$, and the covariance matrix of $\mathit{w}$ is ${\mathit{R}}_{\mathit{w}}$. Clearly, ${\mathit{R}}_{{\mathrm{H}}_0}={\mathit{R}}_{\mathit{w}}$. Therefore, the probability density functions of ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$ are given by:

(19)$f_{{\mathrm{H}}_0}\left(\mathit{y}\right)={\displaystyle \frac{1}{{\pi }^{NM}det\left({\mathit{R}}_w\right)}}exp(-{\mathit{y}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{y})$

(20)$f_{{\mathrm{H}}_1}\left(\mathit{y}\right)={\displaystyle \frac{1}{{\pi }^{NM}det\left({\mathit{R}}_{{\mathrm{H}}_1}\right)}}exp(-{\mathit{y}}^{\mathrm{H}}{\mathit{R}}_{{\mathrm{H}}_1}^{-1}\mathit{y})$

where ${\mathit{R}}_{{\mathrm{H}}_1}=\mathit{GX}{\mathit{R}}_{\alpha }{\mathit{X}}^H{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_w$. In this work, it is assumed that the receiver has prior knowledge of ${\mathit{R}}_{\alpha }$ and ${\mathit{R}}_{\mathit{w}}$. This is reasonable because ${\mathit{R}}_{\alpha }$ and ${\mathit{R}}_{\mathit{w}}$ can be estimated through previously received signals.

Therefore, the relative entropy between ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$ is given by:

(21)$\begin{array}{cc}\hfill \mathrm{D}\left({\mathrm{H}}_0\right|\left|{\mathrm{H}}_1\right)& =\int f_{{\mathrm{H}}_0}\left(\mathit{y}\right)ln{\displaystyle \frac{f_{{\mathrm{H}}_0}\left(\mathit{y}\right)}{f_{{\mathrm{H}}_1}\left(\mathit{y}\right)}}d\mathit{y}\hfill \\ \hfill & =lndet\left({\mathit{R}}_{{\mathrm{H}}_1}\right)-lndet\left({\mathit{R}}_{\mathit{w}}\right)+\mathrm{tr}\left({\mathit{R}}_{{\mathrm{H}}_1}^{-1}{\mathit{R}}_{\mathit{w}}\right)+NM\hfill \end{array}$

In (21), the property ${\mathrm{E}}_{{\mathrm{H}}_0}\left({\mathit{y}}^{\mathrm{H}}{\mathit{R}}_{{\mathrm{H}}_1}^{-1}\mathit{y}\right)=\mathrm{tr}\left({\mathit{R}}_{{\mathrm{H}}_1}^{-1}{\mathit{R}}_{{\mathrm{H}}_0}\right)$ is used.

Therefore, by estimating the covariance matrix ${\widehat{\mathit{R}}}_{\mathit{y}}$, we can calculate $\mathrm{D}\left({\mathrm{H}}_{\mathit{y}}\right|\left|{\mathrm{H}}_0\right)$ and $\mathrm{D}\left({\mathrm{H}}_{\mathit{y}}\right|\left|{\mathrm{H}}_1\right)$ through (21). If $\mathrm{D}({\mathrm{H}}_{\mathit{y}}\left|\right|{\mathrm{H}}_1)>\mathrm{D}({\mathrm{H}}_{\mathit{y}}\left|\right|{\mathrm{H}}_0)$, it indicates that the distribution of the received signal is more similar to ${\mathrm{H}}_0$, meaning that there is no target in the received signal; otherwise, the target is present. The RET target detection algorithm for OTFS-ISAC is summarized in Algorithm 1.

Algorithm 1: The RET Target Detection Algorithm for OTFS-ISAC

Input: $\mathit{y}$, ${\mathit{R}}_{{\mathrm{H}}_1}$, ${\mathit{R}}_{\mathit{w}}$ Output: Indicator $\mathcal{L}$ for the presence of the targets Estimate ${\widehat{\mathit{R}}}_{\mathit{y}}$ based on $\mathit{y}$; Calculate $\mathrm{D}\left({\mathrm{H}}_{\mathit{y}}\right|\left|{\mathrm{H}}_0\right)$ and $\mathrm{D}\left({\mathrm{H}}_{\mathit{y}}\right|\left|{\mathrm{H}}_1\right)$ through (21); If $\mathrm{D}({\mathrm{H}}_{\mathit{y}}\left|\right|{\mathrm{H}}_1)<\mathrm{D}({\mathrm{H}}_{\mathit{y}}\left|\right|{\mathrm{H}}_0)$, $\mathcal{L}=\mathrm{True}$; Otherwise, $\mathcal{L}=\mathrm{False}$; Return $\mathcal{L}$.

2.4. The I-OTFS-WD Algorithm

To enhance the target detection capability, it is necessary to increase the distinguishability between the distributions of ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$. Therefore, the waveform design problem is modeled as an optimization problem that maximizes the relative entropy.

2.4.1. Design of the Objective Function

We take $\mathrm{D}\left({\mathrm{H}}_0\right|\left|{\mathrm{H}}_1\right)$ as the objective function for the waveform design problem. Based on the OTFS-ISAC input–output relationship, (21) can be rewritten as:

(22)$\begin{array}{cc}\hfill & \mathrm{D}({\mathrm{H}}_0\left|\right|{\mathrm{H}}_1)=\int f_{{\mathrm{H}}_0}\left(\mathit{y}\right)ln{\displaystyle \frac{f_{{\mathrm{H}}_0}\left(\mathit{y}\right)}{f_{{\mathrm{H}}_1}\left(\mathit{y}\right)}}d\mathit{y}\hfill \\ \hfill & =lndet(\mathit{GX}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})-lndet\left({\mathit{R}}_{\mathit{w}}\right)+\mathrm{tr}\left({(\mathit{GX}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})}^{-1}{\mathit{R}}_{\mathit{w}}\right)+NM\hfill \end{array}$

Ignoring the constant terms, the objective function can be re-expressed as:

(23)$\mathrm{obj}=lndet(\mathit{GX}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})-lndet\left({\mathit{R}}_{\mathit{w}}\right)+\mathrm{tr}\left({(\mathit{GX}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})}^{-1}{\mathit{R}}_{\mathit{w}}\right)$

2.4.2. Design of the Constraints

In this work, the constraints are the power constraint and communication QoS constraint. Specifically, assuming that the transmission power is $P_t$, the power constraint is:

(24)${\mathit{X}}^{\mathrm{H}}\mathit{X}\le P{P}_t$

For communication, let us assume that there are $P_C$ channels, with channel attenuation factors $\mathit{\beta }={[{\beta }_1,...,{\beta }_{P_C}]}^{\mathrm{T}}$. Then, the received signal for the communication user is:

(25)${\mathit{y}}_C={\mathit{G}}_C\mathit{X}\mathit{\beta }+{\mathit{w}}_C$

where ${\mathit{G}}_C=[\mathit{Q}({\widehat{k}}_1,{\widehat{l}}_1),...,\mathit{Q}({\widehat{k}}_{P_C},{\widehat{l}}_{P_C})]$, and ${\widehat{k}}_i$ and ${\widehat{l}}_i$ represent the Doppler tap and delay tap of the i-th channel, respectively. ${\mathit{w}}_C$ is the communication AWGN, with its covariance matrix denoted as ${\mathit{R}}_{{\mathit{w}}_C}={\sigma }_{{\mathit{w}}_C}^2{\mathit{I}}_{NM}$.

To ensure communication quality, the received signal-to-noise ratio (SNR) for the communication user must meet a minimum requirement. Therefore, the communication constraint is as follows:

(26)${\displaystyle \frac{{\mathit{\beta }}^{\mathrm{H}}{\mathit{X}}^{\mathrm{H}}{\left({\mathit{G}}_C\right)}^{\mathrm{H}}{\mathit{G}}_C\mathit{X}\mathit{\beta }}{NM{\sigma }_{w_C}^2}}\ge \eta$

where $\eta$ represents the minimum communication SNR requirement.

2.4.3. Solution to the Waveform Design Optimization Problem

Based on Section 2.4.1 and Section 2.4.2, the optimization problem for the OTFS-ISAC waveform design, which maximizes relative entropy, is expressed as follows:

(27)$P_0:\left\{\begin{array}{cc}\underset{\mathit{X}}{max}\hfill & \mathrm{D}\left({\mathrm{H}}_0\right|\left|{\mathrm{H}}_1\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {\mathit{X}}^{\mathrm{H}}\mathit{X}\le P{P}_t\hfill \\ \hfill & {\displaystyle \frac{{\beta }^{\mathrm{H}}{\mathit{X}}^{\mathrm{H}}{\left({\mathit{G}}_C\right)}^{\mathrm{H}}{\mathit{G}}_C\mathit{X}\beta }{NM{\sigma }_{w_C}^2}}\ge \eta \hfill \end{array}\right.$

$P_0$ is a non-convex optimization problem; in this work, the MM framework is used to address it. First, the $lndet(•)$ term in the objective function (23) is handled.

Lemma 1.

The following conclusion holds for the $lndet(•)$ term in (23):

(28) $\begin{array}{cc}\hfill \mathfrak{A}\left({\mathit{U}}_1\right)& =lndet({\mathit{GXR}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})-lndet\left({\mathit{R}}_{\mathit{w}}\right)\hfill \\ \hfill & =lndet\left({\mathit{V}}_1{\mathit{U}}_1^{-1}{\mathit{V}}_1^{\mathrm{H}}\right)\hfill \end{array}$

where ${\mathit{V}}_1=[{\mathit{I}}_P,{\mathit{0}}_{P\times NM}]\in {\mathbb{C}}^{P\times (P+NM)}$, ${\mathit{U}}_1\in {\mathbb{C}}^{(P+NM)\times (P+NM)}$ and can be expressed as:

(29) ${\mathit{U}}_1=\left[\begin{array}{cc}{\mathit{I}}_{P\times P}& {\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\\ \mathit{GX}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}& {\mathit{R}}_{\mathit{w}}+\mathit{GX}{\mathit{R}}_{\alpha }{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\end{array}\right]$

The proof of Lemma 1 is presented in Appendix A.

According to Lemma 1 in [20], $\mathfrak{A}\left({\mathit{U}}_1\right)$ is a convex function with respect to ${\mathit{U}}_1$. According to [23], $\mathfrak{A}\left({\mathit{U}}_1\right)$ can be lower-bounded by its supporting hyperplane, Thus, the following conclusion can be drawn:

(30)$\begin{array}{cc}\hfill \mathfrak{A}\left({\mathit{U}}_1\right)& \ge \mathfrak{A}\left({\mathit{U}}_1^{\left(t\right)}\right)+\mathrm{tr}\left(\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right)({\mathit{U}}_1-{\mathit{U}}_1^{\left(t\right)})\right)\hfill \\ \hfill & \ge \mathfrak{A}\left({\mathit{U}}_1^{\left(t\right)}\right)+\mathrm{tr}\left(\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right){\mathit{U}}_1\right)-\mathrm{tr}\left(\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right){\mathit{U}}_1^{\left(t\right)}\right)\hfill \end{array}$

where ${\mathit{U}}_1^{\left(t\right)}$ represents ${\mathit{U}}_1$ at the t-th iteration, and $\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right)$ is the conjugate gradient matrix of $\mathfrak{A}\left({\mathit{U}}_1\right)$ at the t-th iteration of the MM algorithm, expressed as:

(31)$\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right)=-{\left({\mathit{U}}_1^{\left(t\right)}\right)}^{-1}{\mathit{V}}_1^{\mathrm{H}}{\left({\mathit{V}}_1{\left({\mathit{U}}_1^{\left(t\right)}\right)}^{-1}{\mathit{V}}_1^{\mathrm{H}}\right)}^{-1}{\mathit{V}}_1{\left({\mathit{U}}_1^{\left(t\right)}\right)}^{-1}$

The matrix $\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right)$ is partitioned as follows:

(32)$\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right)=\left[\begin{array}{cc}{\mathit{D}}_{11}& {\mathit{D}}_{12}\\ {\mathit{D}}_{12}^{\mathrm{H}}& {\mathit{D}}_{22}\end{array}\right]$

Then, in (30), $\mathrm{tr}\left(\mathit{D}\left({\mathit{U}}_1^{\left(t\right)}\right){\mathit{U}}_1\right)$ can be simplified as:

(33)$\begin{array}{cc}\hfill \mathrm{tr}(\mathit{D}({\mathit{U}}_1^{(t)}){\mathit{U}}_1)& =\mathrm{tr}(2\mathrm{Re}({\mathit{D}}_{12}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})+{\mathit{D}}_{22}({\mathit{R}}_{\mathit{w}}+\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})+{\mathit{D}}_{11})\hfill \\ \hfill & =2\mathrm{Re}(\mathrm{tr}({\mathit{D}}_{12}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})+{\mathit{D}}_{22}({\mathit{R}}_{\mathit{w}}+\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})+{\mathit{D}}_{11})\hfill \end{array}$

Combining (33), (30) can be rewritten as:

(34)$\mathfrak{A}\left({\mathit{U}}_1\right)\ge c_1+2\mathrm{Re}(\mathrm{tr}({\mathit{D}}_{12}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}))+\mathrm{tr}\left({\mathit{D}}_{22}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\right)$

where $c_1$ is expressed as:

(35)$c_1=lndet\left({\mathit{V}}_1{\left({\mathit{U}}_1^{\left(t\right)}\right)}^{-1}{\mathit{V}}_1^{\mathrm{H}}\right)+\mathrm{tr}({\mathit{D}}_{11}+{\mathit{D}}_{22}{\mathit{R}}_{\mathit{w}}-\mathit{D}{\mathit{U}}_1^{\left(t\right)})$

Using properties $\mathrm{tr}\left({\mathit{A}}^{\mathrm{T}}\mathit{B}\right)={\mathrm{vec}}^{\mathrm{T}}\left(\mathit{A}\right)\mathrm{vec}\left({\mathit{B}}^{\mathrm{T}}\right)$, $\mathrm{tr}\left(\mathit{ABCD}\right)={\mathrm{vec}}^{\mathrm{T}}\left(\mathit{D}\right)(\mathit{A}\otimes \mathit{C})\mathrm{vec}\left(\mathit{B}\right)$, and ${({\mathit{R}}_{\alpha }^{{\displaystyle \frac{1}{2}}})}^{\mathrm{H}}={\mathit{R}}_{\alpha }^{{\displaystyle \frac{1}{2}}}$, the second term in (34) can be reformulated as:

(36)$\begin{array}{cc}\hfill 2\mathrm{Re}(\mathrm{tr}({\mathit{D}}_{12}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}))& =2\mathrm{Re}(\mathrm{tr}{({\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{D}}_{12}\mathit{G}\mathit{X})}^{\mathrm{H}})\hfill \\ \hfill & =2\mathrm{Re}(\mathrm{tr}({\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{D}}_{12}^{\mathrm{H}}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}))\hfill \\ \hfill & =2\mathrm{Re}\left({\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathit{s}}\right)\hfill \end{array}$

where $\widehat{\mathit{x}}=\mathrm{vec}\left(\mathit{X}\right)$, and $\widehat{\mathit{s}}=\mathrm{vec}({({\mathit{G}}^{\mathrm{H}}{\mathit{D}}_{12}^{\mathrm{H}}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{\mathrm{T}})$

Similarly, the third term in (34) is handled as follows:

(37)$\begin{array}{cc}\hfill \mathrm{tr}\left({\mathit{D}}_{22}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\right)& =\mathrm{tr}\left({\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{D}}_{22}\mathit{G}\mathit{X}\right)\hfill \\ \hfill & =\mathrm{tr}\left({\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\widehat{\mathit{D}}}_{22}\mathit{X}\right)\hfill \\ \hfill & ={\widehat{\mathit{x}}}^{\mathrm{H}}({\mathit{R}}_{\mathit{\alpha }}^{*}\otimes {\widehat{\mathit{D}}}_{22})\widehat{\mathit{x}}\hfill \end{array}$

where ${\widehat{\mathit{D}}}_{22}={\mathit{G}}^{\mathrm{H}}{\mathit{D}}_{22}\mathit{G}$. Based on (36) and (37), (34) can be rewritten as:

(38)$\mathfrak{A}\left({\mathit{U}}_1\right)\ge c_1+2\mathrm{Re}\left({\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathit{s}}\right)+{\widehat{\mathit{x}}}^{\mathrm{H}}({\mathit{R}}_{\mathit{\alpha }}^{*}\otimes {\widehat{\mathit{D}}}_{22})\widehat{\mathit{x}}$

Next, we handle the $\mathrm{tr}(•)$ term in (23).

Lemma 2.

The following conclusion holds for the $\mathrm{tr}(•)$ term in (23):

(39) $\mathrm{tr}\left({(\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})}^{-1}{\mathit{R}}_{\mathit{w}}\right)=NM-\mathrm{tr}\left({\mathit{R}}_{\mathit{w}}^{-1}{\left({\mathit{V}}_2{\mathit{U}}_2^{-1}{\mathit{V}}_2^{\mathrm{H}}\right)}^{-1}\right)$

where ${\mathit{V}}_2=[{\mathit{I}}_{NM},{\mathit{0}}_{NM\times NM}]\in {\mathbb{C}}^{NM\times 2NM}$, ${\mathit{U}}_2$ is expressed as:

(40) ${\mathit{U}}_2=\left[\begin{array}{cc}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}& \mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\\ \mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}& {\mathit{R}}_{\mathit{w}}+\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\end{array}\right]$

The proof of Lemma 2 can be found in Appendix B.

According to Lemma 2 in [20], (38) is a convex function with respect to ${\mathit{U}}_2$. Therefore, ignoring the constant term in (38), we have:

(41)$-\mathrm{tr}\left({\mathit{R}}_{\mathit{w}}^{-1}{\left({\mathit{V}}_2{\mathit{U}}_2^{-1}{\mathit{V}}_2^{\mathrm{H}}\right)}^{-1}\right)\ge -\mathrm{tr}\left({\mathit{R}}_{\mathit{w}}^{-1}{\left({\mathit{V}}_2{\left({\mathit{U}}_2^{\left(t\right)}\right)}^{-1}{\mathit{V}}_2^{\mathrm{H}}\right)}^{-1}\right)+\mathrm{t}r\left(\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right)({\mathit{U}}_2-{\mathit{U}}_2^{\left(t\right)})\right)$

where $\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right)$ is the matrix of the t-th iteration of the MM algorithm, and its expression is:

(42)$\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right)=-{\left({\mathit{U}}_2^{\left(t\right)}\right)}^{-1}{\mathit{V}}_2^{\mathrm{H}}{\left({\mathit{V}}_2{\left({\mathit{U}}_2^{\left(t\right)}\right)}^{-1}{\mathit{V}}_2^{\mathrm{H}}\right)}^{-1}{\mathit{R}}_{\mathit{w}}^{-1}{\left({\mathit{V}}_2{\left({\mathit{U}}_2^{\left(t\right)}\right)}^{-1}{\mathit{V}}_2^{\mathrm{H}}\right)}^{-1}{\mathit{V}}_2{\left({\mathit{U}}_2^{\left(t\right)}\right)}^{-1}$

The matrix $\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right)$ is partitioned as follows:

(43)$\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right)=\left[\begin{array}{cc}{\mathit{F}}_{11}& {\mathit{F}}_{12}\\ {\mathit{F}}_{21}& {\mathit{F}}_{22}\end{array}\right]$

Then, $\mathrm{tr}\left(\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right){\mathit{U}}_2\right)$ can be further simplified as:

(44)$\mathrm{tr}\left(\mathit{F}\left({\mathit{U}}_2^{\left(t\right)}\right){\mathit{U}}_2\right)=\mathrm{tr}\left({\mathit{F}}_{22}{\mathit{R}}_{\mathit{w}}\right)+{\widehat{\mathit{x}}}^{\mathrm{H}}({\mathit{R}}_{\alpha }^{*}\otimes {\widehat{\mathit{F}}}_{\mathit{1}}+{\mathit{R}}_{\alpha }^{*}\otimes {\widehat{\mathit{F}}}_{\mathit{2}})\widehat{\mathit{x}}$

where ${\widehat{\mathit{F}}}_1={\mathit{G}}^{\mathrm{H}}{\mathit{F}}_{11}\mathit{G}$ and ${\widehat{\mathit{F}}}_2={\mathit{G}}^{\mathrm{H}}({\mathit{F}}_{12}+{\mathit{F}}_{21})\mathit{G}$.

Similarly, the communication constraint can be vectorized as follows:

(45)${\displaystyle \frac{{\beta }^{\mathrm{H}}{\mathit{X}}^{\mathrm{H}}{\left({\mathit{G}}_C\right)}^{\mathrm{H}}{\mathit{G}}_C\mathit{X}\beta }{NM{\sigma }_{{\mathit{w}}_C}^2}}={\displaystyle \frac{{\widehat{\mathit{x}}}^{\mathrm{H}}({\mathit{R}}_{\beta }^{*}\otimes {\widehat{\mathit{G}}}_C)\widehat{\mathit{x}}}{NM{\sigma }_{{\mathit{w}}_C}^2}}$

where ${\mathit{R}}_{\beta }$ is the covariance matrix of the channel, and ${\widehat{\mathit{G}}}_C={{\mathit{G}}_C}^{\mathrm{H}}{\mathit{G}}_C$.

Based on (38), (44) and (45), the optimization problem at the t-th iteration of Equation (27) can be expressed as:

(46)$P_1^{\left(t\right)}:\left\{\begin{array}{cc}\underset{\widehat{\mathit{x}}}{max}\hfill & {\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathbf{\Psi }}\widehat{\mathit{x}}+2\mathrm{Re}\left({\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathit{s}}\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathit{x}}\le P{P}_t\hfill \\ \hfill & {\widehat{\mathit{x}}}^{\mathrm{H}}\widehat{\mathbf{\Theta }}\widehat{\mathit{x}}\ge \eta \hfill \end{array}\right.$

where $\widehat{\mathbf{\Psi }}$ is expressed as:

(47)$\widehat{\mathbf{\Psi }}={\mathit{R}}_{\mathit{\alpha }}^{*}\otimes {\widehat{\mathit{D}}}_{22}+{\mathit{R}}_{\mathit{\alpha }}^{*}\otimes {\widehat{\mathit{F}}}_1+{\mathit{R}}_{\mathit{\alpha }}^{*}\otimes {\widehat{\mathit{F}}}_2$

$\widehat{\mathbf{\Theta }}$ is expressed as:

(48)$\widehat{\mathbf{\Theta }}={\displaystyle \frac{{\mathit{R}}_{\beta }^{*}\otimes {\widehat{\mathit{G}}}_C}{NM{\sigma }_{{\mathit{w}}_C}^2}}$

Furthermore, let $\mathit{A}={[{\mathit{S}}_1^T,{\mathit{S}}_2^T,...,{\mathit{S}}_P^T]}^T\in {\mathbb{C}}^{NM{P}^2\times NM}$, where for ${\mathit{S}}_i$, its $(i-1)NMiNM$-th rows constitute the identity matrix ${\mathit{I}}_{NM}$, while all other elements are zero. Using $\mathit{A}$, we have $\widehat{\mathit{x}}=\mathit{A}\mathit{x}$. Then, (46) can be rewritten as:

(49)$P_2^{\left(t\right)}:\left\{\begin{array}{cc}\underset{\mathit{x}}{max}\hfill & {\mathit{x}}^{\mathrm{H}}\mathbf{\Psi }\mathit{x}+2\mathrm{Re}\left({\mathit{x}}^{\mathrm{H}}\mathit{s}\right)\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & {\mathit{x}}^{\mathrm{H}}\mathit{x}\le P_t\hfill \\ \hfill & {\mathit{x}}^{\mathrm{H}}\mathbf{\Theta }\mathit{x}\ge \eta \hfill \end{array}\right.$

where $\mathbf{\Psi }={\mathit{A}}^{\mathrm{H}}\widehat{\mathbf{\Psi }}\mathit{A}$, $\mathit{s}={\mathit{A}}^{\mathrm{H}}\widehat{\mathit{s}}$, and $\mathbf{\Theta }={\mathit{A}}^{\mathrm{H}}\widehat{\mathbf{\Theta }}\mathit{A}$.

Since the communication constraint in (49) is non-convex, the optimization problem remains non-convex. Noting that (49) is a generalized QCQP problem, the SDR technique from [22] can be used to solve it. According to [22], (49) can be transformed into the following convex optimization problem:

(50)$P_3^{\left(t\right)}:\left\{\begin{array}{cc}\underset{\aleph }{max}\hfill & \mathrm{tr}(\Delta \aleph )\hfill \\ \mathrm{s}.\mathrm{t}.\hfill & \mathrm{tr}(\aleph )\le P_t+1\hfill \\ \hfill & \mathrm{tr}(\Lambda \aleph )\ge \eta \hfill \end{array}\right.$

where $\aleph =\breve{\mathit{x}}{\breve{\mathit{x}}}^{\mathrm{H}}$, $\breve{\mathit{x}}={[{\mathit{x}}^{\mathrm{T}},g]}^{\mathrm{T}}$, $\left|g\right|=1$, and the expression of $\Delta$ is:

(51)$\Delta =\left[\begin{array}{cc}\mathbf{\Psi }& \mathit{s}\\ {\mathit{s}}^{\mathrm{H}}& 0\end{array}\right]$

The expression of $\Lambda$ is:

(52)$\Lambda =\left[\begin{array}{cc}\mathbf{\Theta }& 0\\ 0& 0\end{array}\right]$

According to [22], if the optimal solution of optimization problem (50) is ${\aleph }^{★}$, then the optimal ${\mathit{x}}^{★}$ is the eigenvector corresponding to the largest eigenvalue of ${\aleph }^{★}$.

3. Results and Discussion

The simulation parameters of this work are shown in Table 1. Without loss of generality, we set the DD domain-transmitted symbols ${\mathit{X}}_{dd}$ as a random symbol matrix with an amplitude of 1, and ${\mathit{R}}_{\mathit{\alpha }}={\mathit{I}}_P$. These simulation parameters align with those of typical practical millimeter-wave ISAC systems. Based on the simulation parameters, the maximum detection range of the OTFS-ISAC system is 240 m. Considering that high Doppler resolution is not required for target detection tasks and to reduce processing complexity, $N=16$ is selected.

Firstly, we simulate the RET target detection algorithm for OTFS-ISAC. In the simulation, we select the LRT target detection algorithm with a false alarm probability of ${10}^{-6}$ as a baseline for comparison.

Figure 2 illustrates the variation in the detection probability of the RET target detection algorithm with respect to SNR. In the simulation, RET requires multiple echo samples to estimate ${\widehat{\mathit{R}}}_{\mathit{y}}$. Therefore, we set different numbers of echo samples to observe the impact of sample size on the RET algorithm’s performance. Let the sample size be L. To ensure that the ${\widehat{\mathit{R}}}_{\mathit{y}}$ is full-rank, it must satisfy $L\ge NM=256$. However, when $L=256$, the ${\widehat{\mathit{R}}}_{\mathit{y}}$ tends to be unstable and may even be singular. Therefore, we chose 300, slightly larger than 256, as the minimum simulation sample size. In traditional sensing systems based on LRT, such as radar, the detectable SNR typically ranges from 10 dB to 13 dB. However, for targets at very long distances, the two-way propagation loss is inversely proportional to the fourth power of the target distance, resulting in a significantly lower received SNR than the detectable SNR. To validate the superior target detection capability of the proposed OTFS-ISAC system, we set the minimum SNR to −20 dB.

The simulation results show that the performance of the RET algorithm depends on the number of samples used to estimate ${\widehat{\mathit{R}}}_{\mathit{y}}$. When the sample size is 300, the performance of RET is inferior to LRT at any SNR level. However, as the sample size increases, the performance of RET improves significantly. With a sample size of 1000, RET achieves a detection probability of 0.956 even at an SNR of −20 dB, which is 9.12 times higher than LRT. This outcome arises because the RET algorithm heavily relies on the accuracy of ${\widehat{\mathit{R}}}_{\mathit{y}}$ estimation. A larger sample size leads to a more accurate estimation of ${\widehat{\mathit{R}}}_{\mathit{y}}$. Moreover, under accurate ${\widehat{\mathit{R}}}_{\mathit{y}}$ estimation, the RET algorithm is less sensitive to noise. This is because the essence of RET is to “highlight” the target by evaluating the similarity between the echo signal distribution and the distributions of ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$, rather than relying directly on SNR.

Next, we simulate the proposed I-OTFS-WD method. Figure 3 illustrates the MM iteration process of the I-OTFS-WD method under different normalized $P_t$ values. In this simulation, we validate the convergence speed of the I-OTFS-WD algorithm through the variation of relative entropy during the iterative process. The reason for setting different $P_t$ values is that practical OTFS-ISAC systems adjust the transmit power based on varying scenarios. This simulation also validates the relationship between relative entropy and $P_t$. It can be observed that as $P_t$ increases, the relative entropy achieved by the optimized waveform also increases. Additionally, Figure 3 demonstrates that the algorithm converges rapidly, requiring only a single iteration to obtain the optimal waveform.

Figure 4 illustrates the variation of relative entropy during the MM iteration process of the I-OTFS-WD method. In general, the sensing and communication performance of ISAC systems are mutually constrained. This simulation evaluates the impact of different communication QoS constraints on the waveforms optimized by the I-OTFS-WD algorithm. In the simulation, different communication QoS levels are set by varying $\eta$. The results in Figure 4 reveal an interesting observation: under the condition of feasible solutions, communication QoS does not affect the optimal waveform. This is because the OTFS-ISAC system proposed in this paper uses a single waveform to simultaneously accomplish both communication and target detection tasks. As long as the communication QoS is satisfied, the algorithm tends to fully utilize the transmit power, as shown in Figure 3. Under feasible conditions, the waveform that maximizes relative entropy always satisfies the communication QoS.

Figure 5 demonstrates the impact of the proposed I-OTFS-WD method on target detection performance. In this simulation, target detection is performed using the RET algorithm with both the initial waveform and the optimized waveform. This is to validate the impact of the waveforms designed by the I-OTFS-WD algorithm on detection performance and the sample size required by the RET algorithm. The simulation results show that the optimized waveform significantly enhances the system’s target detection capability for the same number of samples. For instance, at a low SNR of −18 dB, the detection probability with 600 samples using the optimized waveform reaches 0.976, which is close to the detection probability of 1.0 with 1000 samples. In other words, under comparable detection performance, the optimized waveform reduces the sample size required by the RET algorithm by 40%. The fundamental reason why the I-OTFS-WD algorithm enhances targets detection performance is as follows: The objective function of the I-OTFS-WD algorithm is to maximize the relative entropy between ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$. By optimizing the transmitted signal vector $\mathit{x}$, the I-OTFS-WD ensures that the similarity between ${\mathrm{H}}_0$ and ${\mathrm{H}}_1$ is minimized in terms of both energy and direction, thereby improving the performance of the RET algorithm.

4. Conclusions

To enhance the detection capability of ISAC systems under low-SNR conditions and improve robustness in high-speed scenarios, this work proposes an information-theoretic OTFS-ISAC target detection framework. The framework employs OTFS waveforms to construct the RET target detection algorithm. To further improve the target detection performance of OTFS-ISAC, the I-OTFS-WD method based on maximizing relative entropy is proposed.

The simulation results indicate that the OTFS-ISAC RET target detection algorithm achieves a 9.12-fold improvement in detection probability under low-SNR conditions compared to the traditional LRT detection algorithm when sufficient samples are available. The results also show that the proposed I-OTFS-WD method converges within a single iteration and reduces the required sample size for the RET algorithm under comparable detection performance. Specifically, at low SNR, the required sample size is reduced by 40% compared to that of non-optimized waveforms. Additionally, under feasible conditions, the greater the power constraint, the better the performance of the optimal waveform, which remains unaffected by communication QoS.

The proposed OTFS-ISAC system demonstrates significant application potential in scenarios such as pedestrian monitoring in intelligent transportation systems and vehicle tracking in the Internet of Things. These applications demand the multifunctional integration of sensing and communication to reduce system costs while ensuring performance. However, limitations remain in the current system. Specifically, the RET target detection algorithm proposed in this work requires more than 400 samples to achieve satisfactory performance under low-SNR conditions, which poses practical challenges.

In future work, we will focus on improving the target detection performance of the OTFS-ISAC system under small sample conditions. Additionally, we will explore waveform design tailored to target parameter estimation in the OTFS-ISAC system.

Author Contributions

Conceptualization, Q.M.; methodology, Q.M. and X.S.; formal analysis, Q.M.; investigation, simulation, and analysis, Q.M.; writing—original draft preparation, Q.M. and C.X.; writing—review and editing, X.S., Y.G. and L.C. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The original contributions presented in the study are included in the article material; further inquiries can be directed to the corresponding author.

Acknowledgments

The authors sincerely thank the Radar and Positioning Team of the University of Electronic Science and Technology of China for their invaluable support and constructive feedback during this work.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ISACS	Integrated sensing and communication
OFDM	Orthogonal frequency division multiplexing
OTFS	Orthogonal time frequency space
DD	Doppler-delay
TF	Time–frequency
MIMO	Multiple-input multiple-output
RET	Relative entropy test
LRT	Likelihood ratio test
SNR	Signal-to-noise ratio
I-OTFS-WD	Iterative OTFS-ISAC waveform design
QoS	Quality of service
MM	Minorization-maximization
QCQP	Quadratically constrained quadratic program
SDR	Semidefinite relaxation
CP	Cyclic prefix
SFFT	Symplectic finite Fourier transform
ISFFT	Inverse SFFT
AWGN	Additive white Gaussian noise

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Table

Figure 1. The modulation and demodulation process of the OTFS signal.

Figure 2. Performance curves of the RET target detection algorithm for OTFS-ISAC.

Figure 3. Curves of relative entropy variation during the MM iterative process of the I-OTFS-WD.

Figure 4. Curves of relative entropy variation during the MM iterative process of the I-OTFS-WD under different communication QoS.

Figure 5. RET performance curves of the optimized waveform for OTFS-ISAC.

Table 1

The simulation parameters.

Symbol	Parameter	Value
$f_c$	Carrier frequency	77 GHz
N	Number of Doppler samples	16
M	Number of delay samples	16
B	Bandwidth of the signal	10 MHz
$\Delta f$	Subcarrier spacing	625 kHz
P	Number of targets	5

Appendix A

Using $det(\mathit{I}+\mathit{A}\mathit{B})=det(\mathit{I}+\mathit{B}\mathit{A})$, $lndet(•)$ of (23) can be written as:(A1)$$\begin{array}{cc}\hfill \mathfrak{A}\left({\mathit{U}}_1\right)& =lndet(\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})-lndet\left({\mathit{R}}_{\mathit{w}}\right)\hfill \\ \hfill & =lndet(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})\hfill \\ \hfill & =-lndet{(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{-1}\hfill \end{array}$$

Using the matrix inversion lemma and the block matrix inversion theorem, the result of Lemma 1 can be obtained.

Appendix B

Using the matrix inversion lemma, $\mathrm{tr}(•)$ of (23) can be written as:(A2)$$\begin{array}{cc}\hfill & \mathrm{tr}\left({(\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+{\mathit{R}}_{\mathit{w}})}^{-1}{\mathit{R}}_{\mathit{w}}\right)\hfill \\ \hfill & =\mathrm{tr}(({\mathit{R}}_{\mathit{w}}^{-1}-{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{G}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{-1}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}){\mathit{R}}_{\mathit{w}})\hfill \\ \hfill & =\mathrm{tr}(\mathit{I}-{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{-1}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})\hfill \\ \hfill & =NM-\mathrm{tr}({\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{-1}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})\hfill \end{array}$$

By applying the matrix inversion lemma again, the second term in the last equality of (A2) can be written as:(A3)$$\begin{array}{cc}\hfill & -\mathrm{tr}({\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{(\mathit{I}+{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{\mathit{R}}_{\mathit{w}}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}})}^{-1}{\mathit{R}}_{\mathit{\alpha }}^{{\displaystyle \frac{1}{2}}}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})\hfill \\ \hfill & =-\mathrm{tr}\left({\mathit{R}}_{\mathit{w}}^{-1}(\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}-\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}{({\mathit{R}}_{\mathit{w}}+\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})}^{-1}\mathit{G}\mathit{X}{\mathit{R}}_{\mathit{\alpha }}{\mathit{X}}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}})\right)\hfill \end{array}$$

By applying the block matrix inversion theorem to (A3), the conclusion of Lemma 2 can be obtained.

References

1. Liu, F.; Masouros, C.; Petropulu, A.P.; Griffiths, H.; Hanzo, L. Joint Radar and Communication Design: Applications, State-of-the-Art, and the Road Ahead. IEEE Trans. Commun.; 2020; 68, pp. 3834-3862. [DOI: https://dx.doi.org/10.1109/TCOMM.2020.2973976]

2. Mishra, K.V.; Bhavani Shankar, M.; Koivunen, V.; Ottersten, B.; Vorobyov, S.A. Toward Millimeter-Wave Joint Radar Communications: A Signal Processing Perspective. IEEE Signal Process. Mag.; 2019; 36, pp. 100-114. [DOI: https://dx.doi.org/10.1109/MSP.2019.2913173]

3. Ma, D.; Shlezinger, N.; Huang, T.; Liu, Y.; Eldar, Y.C. Joint Radar-Communication Strategies for Autonomous Vehicles: Combining Two Key Automotive Technologies. IEEE Signal Process. Mag.; 2020; 37, pp. 85-97. [DOI: https://dx.doi.org/10.1109/MSP.2020.2983832]

4. Lu, S.; Liu, F.; Li, Y.; Zhang, K.; Huang, H.; Zou, J.; Li, X.; Dong, Y.; Dong, F.; Zhu, J. et al. Integrated Sensing and Communications: Recent Advances and Ten Open Challenges. IEEE Internet Things J.; 2024; 11, pp. 19094-19120. [DOI: https://dx.doi.org/10.1109/JIOT.2024.3361173]

5. Liu, F.; Cui, Y.; Masouros, C.; Xu, J.; Han, T.X.; Eldar, Y.C.; Buzzi, S. Integrated Sensing and Communications: Toward Dual-Functional Wireless Networks for 6G and Beyond. IEEE J. Sel. Areas Commun.; 2022; 40, pp. 1728-1767. [DOI: https://dx.doi.org/10.1109/JSAC.2022.3156632]

6. Liu, F.; Zheng, L.; Cui, Y.; Masouros, C.; Petropulu, A.P.; Griffiths, H.; Eldar, Y.C. Seventy Years of Radar and Communications: The road from separation to integration. IEEE Signal Process. Mag.; 2023; 40, pp. 106-121. [DOI: https://dx.doi.org/10.1109/MSP.2023.3272881]

7. Raviteja, P.; Phan, K.T.; Hong, Y.; Viterbo, E. Orthogonal Time Frequency Space (OTFS) Modulation Based Radar System. Proceedings of the 2019 IEEE Radar Conference (RadarConf); Boston, MA, USA, 22–26 April 2019; pp. 1-6. [DOI: https://dx.doi.org/10.1109/RADAR.2019.8835764]

8. Wei, Z.; Yuan, W.; Li, S.; Yuan, J.; Bharatula, G.; Hadani, R.; Hanzo, L. Orthogonal Time-Frequency Space Modulation: A Promising Next-Generation Waveform. IEEE Wirel. Commun.; 2021; 28, pp. 136-144. [DOI: https://dx.doi.org/10.1109/MWC.001.2000408]

9. Hadani, R.; Rakib, S.; Tsatsanis, M.; Monk, A.; Goldsmith, A.J.; Molisch, A.F.; Calderbank, R. Orthogonal Time Frequency Space Modulation. Proceedings of the 2017 IEEE Wireless Communications and Networking Conference (WCNC); San Francisco, CA, USA, 19–22 March 2017; pp. 1-6. [DOI: https://dx.doi.org/10.1109/WCNC.2017.7925924]

10. Raviteja, P.; Phan, K.T.; Hong, Y.; Viterbo, E. Interference Cancellation and Iterative Detection for Orthogonal Time Frequency Space Modulation. IEEE Trans. Wirel. Commun.; 2018; 17, pp. 6501-6515. [DOI: https://dx.doi.org/10.1109/TWC.2018.2860011]

11. Gaudio, L.; Kobayashi, M.; Caire, G.; Colavolpe, G. On the Effectiveness of OTFS for Joint Radar Parameter Estimation and Communication. IEEE Trans. Wirel. Commun.; 2020; 19, pp. 5951-5965. [DOI: https://dx.doi.org/10.1109/TWC.2020.2998583]

12. Gaudio, L.; Kobayashi, M.; Bissinger, B.; Caire, G. Performance Analysis of Joint Radar and Communication using OFDM and OTFS. Proceedings of the 2019 IEEE International Conference on Communications Workshops (ICC Workshops); Shanghai, China, 20–24 May 2019; pp. 1-6. [DOI: https://dx.doi.org/10.1109/ICCW.2019.8757044]

13. Miao, Q.; Liang, J.; Zhang, G.; Han, J. A Target Parameters Estimation Method for CP-OTFS Based Terrestrial Radar System. IEEE Wirel. Commun. Lett.; 2025; 14, pp. 233-237. [DOI: https://dx.doi.org/10.1109/LWC.2024.3497959]

14. Zieliński, T.P.; Karpovich, P.; Abratkiewicz, K.; Maksymiuk, R.; Samczyński, P.; Duda, K.; Wypich, M. Wireless OTFS-Based Integrated Sensing and Communication for Moving Vehicle Detection. IEEE Sens. J.; 2024; 24, pp. 6573-6583. [DOI: https://dx.doi.org/10.1109/JSEN.2024.3350238]

15. Stoica, P.; He, H.; Li, J. New Algorithms for Designing Unimodular Sequences With Good Correlation Properties. IEEE Trans. Signal Process.; 2009; 57, pp. 1415-1425. [DOI: https://dx.doi.org/10.1109/TSP.2009.2012562]

16. Stoica, P.; He, H.; Li, J. On Designing Sequences With Impulse-Like Periodic Correlation. IEEE Signal Process. Lett.; 2009; 16, pp. 703-706. [DOI: https://dx.doi.org/10.1109/LSP.2009.2021378]

17. Cui, G.; Fu, Y.; Yu, X.; Li, J. Local Ambiguity Function Shaping via Unimodular Sequence Design. IEEE Signal Process. Lett.; 2017; 24, pp. 977-981. [DOI: https://dx.doi.org/10.1109/LSP.2017.2700396]

18. Bell, M. Information theory and radar waveform design. IEEE Trans. Inf. Theory; 1993; 39, pp. 1578-1597. [DOI: https://dx.doi.org/10.1109/18.259642]

19. Yang, Y.; Blum, R.S. MIMO radar waveform design based on mutual information and minimum mean-square error estimation. IEEE Trans. Aerosp. Electron. Syst.; 2007; 43, pp. 330-343. [DOI: https://dx.doi.org/10.1109/TAES.2007.357137]

20. Tang, B.; Zhang, Y.; Tang, J. An Efficient Minorization Maximization Approach for MIMO Radar Waveform Optimization via Relative Entropy. IEEE Trans. Signal Process.; 2018; 66, pp. 400-411. [DOI: https://dx.doi.org/10.1109/TSP.2017.2771726]

21. Sun, Y.; Babu, P.; Palomar, D.P. Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning. IEEE Trans. Signal Process.; 2017; 65, pp. 794-816. [DOI: https://dx.doi.org/10.1109/TSP.2016.2601299]

22. Luo, Z.q.; Ma, W.k.; So, A.M.c.; Ye, Y.; Zhang, S. Semidefinite Relaxation of Quadratic Optimization Problems. IEEE Signal Process. Mag.; 2010; 27, pp. 20-34. [DOI: https://dx.doi.org/10.1109/MSP.2010.936019]

23. Boyd, S.; Vandenberghe, L. Convex Optimization; Cambridge University Press: Cambridge, UK, 2004.