Discrete-Time Indefinite Stochastic Linear Quadratic Optimal Control with Second Moment Constraints

Abstract

This paper studies the discrete-time stochastic linear quadratic (LQ) problem with a second moment constraint on the terminal state, where the weighting matrices in the cost functional are allowed to be indefinite. By means of the matrix Lagrange theorem, a new class of generalized difference Riccati equations (GDREs) is introduced. It is shown that the well-posedness, and the attainability of the LQ problem and the solvability of the GDREs are equivalent to each other.

(ProQuest: ... denotes non-US-ASCII text omitted.)

Weihai Zhang 1 and Guiling Li 2

Academic Editor:Xuejun Xie

1, College of Electrical Engineering and Automation, Shandong University of Science and Technology, Qingdao 266590, China
2, College of Mathematics and Systems Science, Shandong University of Science and Technology, Qingdao 266590, China

Received 1 April 2014; Accepted 29 April 2014; 18 May 2014

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

LQ control, initiated by Kalman [1] and extended to stochastic systems by Wonham [2], is one of the most important classes of optimal control issues from both theory and application point of view; we refer the reader to [2-8] for representative work in this area. Different from the classical LQ in modern control theory, it was found in [9, 10] that a stochastic LQ problem with indefinite control weighting matrices can still be well-posed, which evoked a series of subsequent researches; see, for example, [11, 12].

It is well known that in practical engineering, the system state and control input are always subject to various constraints, so how to solve the constrained stochastic LQ issue is a more attractive topic; we refer the reader to [13-19]. Reference [14] presented a tractable approach for LQ controller design of the system with additive noise. Reference [16] was about the constrained LQ of deterministic systems with state equality constraints. Reference [13] studied the parametrization of the solutions of finite-horizon constrained LQ control. Reference [15] was devoted to a stochastic LQ optimal control and an application to portfolio selection, where the control variable is confined to a cone, and all the coefficients of the state equation are random processes. Reference [19] studied the indefinite stochastic LQ control problem of continuous-time Itô systems with a linear equality constraint M x ( T ) = ξ on the terminal state and gave a necessary condition for the existence of an optimal controller. Reference [20] generalized the results of [19] to discrete-time stochastic systems.

In this paper, different from [19, 20] on the constraint conditions, we would like to deal with stochastic LQ control of discrete-time multiplicative noise systems with a second moment constraint E [ x ^{( T ) T} x ( T ) ] = c and such constraints are often encountered in _{H ∞} filtering design; see [21, 22]. By means of Lagrange theorem, we present a necessary condition for the existence of an optimal linear state feedback control with the second moment constraint on the terminal states. It is proved that the solvability of GDRE is necessary and sufficient for the existence of an optimal control under either of the state feedback case or of the open-loop forms. Moreover, we show that the well-posedness and the attainability of the constrained LQ problem, the feasibility of the LMI, and the solvability of the GDRE are equivalent to each other. The novel contribution of this paper is to consider a constrained discrete-time LQ optimal stochastic control, which includes some results of [23] as special cases. A new class of generalized difference Riccati equations (GDREs) is first introduced.

The remainder of the paper is organized as follows. Section 2 gives some definitions and preliminaries. In Section 3, the optimal state feedback control is studied using the matrix Lagrange theorem. We give a necessary and sufficient condition for the well-posedness of the constrained LQ control in Section 4. Section 5 shows the equivalence among the well-posedness and the attainability of the LQ problem, the feasibility of the LMI, and the solvability of the GDRE. The set of all optimal controls is determined. We conclude the paper in Section 6.

Throughout the paper, the following notations are adopted: ^{M T} denotes the transpose of M . M > 0 ( M ...5; 0 ) : M is a positive definite (positive semidefinite) symmetric matrix. tr ... ( M ) : the trace of a square matrix M . ^{R m × n} : the space of all m × n matrices. ^{S n} : the space of all n × n symmetric matrices.

2. Problem Setting

Consider the following constrained discrete-time stochastic LQ control problem.

Problem 1.

Consider [figure omitted; refer to PDF] where the state x ( t ) ∈ ^{R n} , the control input u ( t ) ∈ ^{R m} , and the noise w ( t ) ∈ ^{R 1} , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , [figure omitted; refer to PDF] The process { w ( _{t 0} ) , w ( _{t 0} + 1 ) , ... , w ( T - 1 ) } is a sequence of second-order stationary random variables defined on a complete probability space ( Ω , F , P ) . Without loss of generality, we assume that [figure omitted; refer to PDF] where _{δ s t} is the Kronecker delta, s , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } . c ...5; 0 is a constant, A ( t ) , B ( t ) , C ( t ) , D ( t ) , Q ( t ) , and R ( t ) are matrices having appropriate dimensions determined from context, and Q ( t ) and R ( t ) are real symmetric indefinite matrices. _{x 0} is a given deterministic vector.

Definition 2.

Problem 1 is called well-posed, if ∀ _{x 0} ∈ ^{R n} , [figure omitted; refer to PDF]

Definition 3.

Problem 1 is called attainable, if ∀ _{x 0} ∈ ^{R n} , there exists a sequence { ^{u *} ( _{t 0} ) , ^{u *} ( _{t 0} + 1 ) , ... , ^{u *} ( T - 1 ) } , such that V ( _{x 0} ) = J ( _{t 0} , _{x 0} ; ^{u *} ( _{t 0} ) , ^{u *} ( _{t 0} + 1 ) , ... , ^{u *} ( T - 1 ) ) . In this case, { ^{u 0 *} , ^{u 1 *} , ... , ^{u N - 1 *} } is called an optimal control sequence.

Now, let us consider a mathematical programming (MP) problem in a matrix space: [figure omitted; refer to PDF]

Definition 4.

Let ^{X *} be a point satisfying [figure omitted; refer to PDF] and then ^{X *} is said to be a constraint regular point if the gradient vectors ∇ _{h j} ( ^{X *} ) , j = 1 , ... , p , are linearly independent.

Lemma 5 (Lagrange theorem [24]).

Assume that the functions f , _{h 1} , ... , _{h p} , are twice continuously differentiable. If a regular point ^{X *} is also a relative minimum point for the original MP, then there exists a vector λ ∈ ^{R p} such that [figure omitted; refer to PDF] where the Lagrangian function L ( X , λ ) : = f ( X ) + ^{λ T} h ( X ) .

3. A Necessary Condition for State Feedback Control

In this section, by the matrix Lagrange theorem, we present a necessary condition for Problem 1 based on a new type of GDREs.

Let X ( t ) = E [ x ( t ) x ^{( t ) T} ] . Through a simple calculation, the following deterministic optimal control Problem 6 is equivalent to the original Problem 1 under the state feedback u ( t ) = K ( t ) x ( t ) for t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } .

Problem 6.

Consider [figure omitted; refer to PDF] with [figure omitted; refer to PDF]

Remark 7.

If Problem 1 has a linear feedback optimal control solution ^{u *} ( t ) = ^{K *} ( t ) x ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , then ^{K *} ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , are the optimal solution of Problem 6.

Theorem 8.

If Problem 1 is attainable by u ( t ) = ^{K *} ( t ) x ( t ) , and the regular point ( ^{K *} ( t ) , ^{X *} ( t ) ) is a locally optimal solution of Problem 6, then there exist symmetric matrices P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , and λ ∈ ^{R 1} solving the following GDRE: [figure omitted; refer to PDF] where ^{G [dagger]} is the Moore-Penrose generalized inverse of G . Moreover, [figure omitted; refer to PDF] with Y ( t ) ∈ ^{R m × n} , t = _{t 0} , _{t 0} + 1 , ... , T - 1 , being any given real matrices: [figure omitted; refer to PDF]

To prove Theorem 8, we mainly use Lemma 5 to Problem 6 together with the following lemma to obtain GDRE (10) and then apply the technique of completing squares to show (12).

Lemma 9 (see [12]).

Let A , B , C be given matrices with appropriate sizes; then the matrix equation [figure omitted; refer to PDF] has a solution X if and only if [figure omitted; refer to PDF] Moreover, any solution to A X B = C can be represented by [figure omitted; refer to PDF] where Y is any matrix with appropriate size.

Proof.

According to Remark 7, ^{K *} ( t ) is also the optimal solution of Problem 6. Problem 6 is a typical MP problem about X ( t ) and K ( t ) as follows: [figure omitted; refer to PDF] where [figure omitted; refer to PDF] Let matrices P ( t + 1 ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , be the Lagrangian multipliers of [figure omitted; refer to PDF] and let λ ∈ ^{R 1} be the Lagrangian multiplier of h [ X ( T ) ] = 0 ; then the Lagrangian function [figure omitted; refer to PDF] According to the the matrix Lagrange theorem, we obtain [figure omitted; refer to PDF] Based on the partial rule of gradient matrices, (20) can be transformed into [figure omitted; refer to PDF] Let [figure omitted; refer to PDF] Then we obtain [figure omitted; refer to PDF] Applying Lemma 9, we have [figure omitted; refer to PDF] Equation (21) yields [figure omitted; refer to PDF] Substituting ^{K *} ( t ) = - G ^{( t ) [dagger]} H ( t ) + Y ( t ) - G ^{( t ) [dagger]} G ( t ) Y ( t ) into (26), it follows [figure omitted; refer to PDF] Without loss of generality, we can assume that P is symmetric. Otherwise, we can take P ¯ = ( ^{P T} + P ) / 2 . The objective functional [figure omitted; refer to PDF] A completion of square implies [figure omitted; refer to PDF] We assert that P ( t + 1 ) must satisfy [figure omitted; refer to PDF] If it is not so, there is G ( l ) for l ∈ { _{t 0} , ... , T - 1 } with a negative eigenvalue λ < 0 . Denote the unitary eigenvector with respect to λ by _{v λ} . Let δ ...0; 0 be an arbitrary scalar; we construct a control sequence as follows: [figure omitted; refer to PDF] The associated cost functional becomes [figure omitted; refer to PDF] Let δ [arrow right] ∞ ; then J ( _{t 0} , _{x 0} ; u ~ ( _{t 0} ) , u ~ ( _{t 0} + 1 ) , ... , u ~ ( T - 1 ) ) [arrow right] - ∞ , which contradicts the attainability of Problem 1. So (30) holds.

In view of (29) and (30), (11) and (12) are easily derived. The proof is completed.

Remark 10.

In Theorem 8, in order to apply matrix Lagrange theorem, we assume the optimal solution ^{( _{K *} ( t ) , _{X *} ( t ) ) T} is a regular point. Generally speaking, for a given LQ control, it is easy to examine the regular condition.

Below, we present a numerical example to illustrate the effectiveness of Theorem 8.

Example 11.

In Problem 1, we set [figure omitted; refer to PDF] The state and control weighting matrices are as [figure omitted; refer to PDF] By the relationship between Problems 1 and 6, we know [figure omitted; refer to PDF] Applying Theorem 8, we obtain [figure omitted; refer to PDF] Stage 2. Consider [figure omitted; refer to PDF] Stage 1. Consider [figure omitted; refer to PDF] The optimal cost value of Problem 1 is [figure omitted; refer to PDF] We are able to test the regular condition of ^{( K * ( t ) , X * ( t ) ) T} as follows. In Problem 6, [figure omitted; refer to PDF] which is linear about X ( t ) and quadratic about K ( t ) , while h ( X ( T ) ) = tr ... [ X ( T ) ] - c is linear about X ( T ) . By simple calculations, ∇ _{h 1} ( ^{K *} ( 0 ) , ^{X *} ( 0 ) ) , ∇ _{h 2} ( ^{K *} ( 1 ) , ^{X *} ( 1 ) ) , and ∇ h ( ^{X *} ( T ) ) are all nonzero vectors and hence are linearly independent.

4. Well-Posedness

In this section, we first establish the link between the well-posedness of Problem 1 and the feasibility of some LMIs and then prove that the solvability of GDRE (10) is not only necessary but also sufficient to the well-posedness of Problem 1. Moreover, the well-posedness and the attainability of Problem 1, the feasibility of some LMIs, and the solvability of GDRE (10) are equivalent to each other.

Theorem 12.

Problem 1 is well-posed if there exist symmetric matrices P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , and λ ∈ ^{R 1} solving the following LMIs: [figure omitted; refer to PDF] where [figure omitted; refer to PDF]

Proof.

Note that [figure omitted; refer to PDF] By (41), it is easy to deduce that the cost functional is bounded from below by [figure omitted; refer to PDF] Hence, Problem 1 is well-posed.

Remark 13.

Theorem 12 tells us that any symmetric matrices P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , and λ ∈ ^{R 1} satisfying LMIs (41)-(42) provide a lower bound [figure omitted; refer to PDF] for the cost function. In what follows, we will show that this lower bound is an exact optimal cost value if P ( t ) and λ ∈ ^{R 1} solve GDRE (10).

We have shown that if the LMIs (41)-(42) are satisfied, then the constrained LQ Problem 1 is well-posed. Below, we further show some other equivalent conditions.

Lemma 14 (extended Schur's lemma [25]).

Let the matrices M = ^{M T} , H , G = ^{G T} be given with appropriate sizes. Then, the following three conditions are equivalent: [figure omitted; refer to PDF]

Theorem 15.

Problem 1 is well-posed if and only if there exist symmetric matrices P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , and λ ∈ ^{R 1} satisfying GDRE (10). Furthermore, the optimal cost is [figure omitted; refer to PDF]

A key to prove Theorem 15 is the necessity part, where the stochastic optimization principle is used.

Proof.

Necessity. For _{t 0} ...4; l ...4; T - 1 , define [figure omitted; refer to PDF] By the stochastic optimization principle, when ^{V _{l 1}} [ x ( _{l 1} ) ] is finite, then so is ^{V _{l 2}} [ x ( _{l 2} ) ] for any _{l 1} ...4; _{l 2} . Since Problem 1 is assumed to be well-posed at _{t 0} , ^{V l} [ x ( l ) ] is finite at any stage 0 ...4; l ...4; T - 1 . Now let us start with l = T - 1 , and let P ( T ) = Q ( T ) - λ I , and we have [figure omitted; refer to PDF] Since ^{V T - 1} [ x ( T - 1 ) ] is finite, using Lemma 4.3 of [23], there exists a symmetric matrix P ( T - 1 ) such that [figure omitted; refer to PDF] The obtained solution sequence of symmetric matrices P ( t ) , t = l , l + 1 , ... , T - 1 , and λ ∈ ^{R 1} to GDRE (10) satisfy [figure omitted; refer to PDF] Then by the stochastic optimality principle, the following holds: [figure omitted; refer to PDF] Lemma 4.3 of [23] provides necessary and sufficient conditions for the finiteness of ^{V l - 1} [ x ( l - 1 ) ] : [figure omitted; refer to PDF] Moreover, [figure omitted; refer to PDF] The above proves the necessity part via mathematical induction.

Sufficiency. From the proof of Theorem 8, if GDRE (10) admits a solution P ( t ) and λ , Problem 1 is not only well-posed, but also attainable. The proof of this theorem is complete.

5. Other Equivalent Conditions

In this section, we present some other equivalent conditions for Problem 1.

Theorem 16.

For the constrained LQ Problem 1, the following are equivalent:

(i) Problem 1 is well-posed.

(ii) Problem 1 is attainable.

(iii): The LMIs (41)-(42) are feasible.

(iv) The GDRE (10) is solvable.

Furthermore, when any one of the above conditions is satisfied, Problem 1 is attainable by [figure omitted; refer to PDF] where P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , are solutions to the GDRE (10).

Proof.

Applying Theorems 12-15, ( i ) ... ( iii ) ... ( iv ) . ( ii ) [implies] ( iv ) is shown by Theorem 8. The rest is to prove ( iv ) [implies] ( ii ) and (56). Let P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , solve the GDRE (10). In view of [figure omitted; refer to PDF] a completion of squares yields [figure omitted; refer to PDF] which shows [figure omitted; refer to PDF]

Finally, we present a general expression for the optimal control set based on the solution to GDRE (10).

Theorem 17.

Assume that the GDRE (10) admits a solution. Then the set of all optimal controls is determined by [figure omitted; refer to PDF] where Y ( t ) ∈ ^{R m × n} and Z ( t ) ∈ ^{R m} are arbitrary random variables defined on the probability space ( Ω , F , P ) . Moreover, the optimal cost value is uniquely given by [figure omitted; refer to PDF] where P ( t ) , t ∈ { _{t 0} , _{t 0} + 1 , ... , T - 1 } , and λ are the solution to the GDRE (10).

Proof.

This theorem can be proved by repeating the same procedure as in Theorem 5.1 of [23].

6. Conclusion

In this paper, we have investigated a class of indefinite stochastic LQ control problems with second moment constraints on the terminal state. By the matrix Lagrange theorem, we have established a new GDRE (10) associated with the constrained optimization Problem 1. In addition, by introducing LMIs (41)-(42), we show that the well-posedness and the attainability of Problem 1, the feasibility of the LMIs (41)-(42), and the solvability of GDRE (10) are equivalent to each other.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (no. 61174078), the Research Fund for the Taishan Scholar Project of Shandong Province of China and SDUST Research Fund (no. 2011KYTD105), and State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (Grant no. LAPS13018).

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

References

[1] R. E. Kalman, "Contributions to the theory of optimal control," Boletín de la Sociedad Matemática Mexicana , vol. 5, no. 2, pp. 102-119, 1960.

[2] W. M. Wonham, "On a matrix Riccati equation of stochastic control," SIAM Journal on Control and Optimization , vol. 6, no. 4, pp. 681-697, 1968.

[3] C. E. de Souza, M. D. Fragoso, "On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control," Systems and Control Letters , vol. 14, no. 3, pp. 233-239, 1990.

[4] B. S. Chen, W. Zhang, "Stochastic _{H 2} / _{H ∞} control with state-dependent noise," IEEE Transactions on Automatic Control , vol. 49, no. 1, pp. 45-57, 2004.

[5] W. Zhang, B. S. Chen, "On stabilizability and exact observability of stochastic systems with their applications," Automatica , vol. 40, no. 1, pp. 87-94, 2004.

[6] B. D. O. Anderson, J. B. Moore Optimal Control Linear Quadratic Mathods , Prentice-Hall, New York, NY, USA, 1989.

[7] F. L. Lewis Optimal Control , John Wiley and Sons, New York, NY, USA, 1986.

[8] Y. Huang, W. Zhang, H. Zhang, "Infinite horizon linear quadratic optimal control for discrete-time stochastic systems," Asian Journal of Control , vol. 10, no. 5, pp. 608-615, 2008.

[9] S. Chen, X. J. Li, X. Y. Zhou, "Stochastic linear quadratic regulators with indefinite control weight costs," SIAM Journal on Control and Optimization , vol. 36, no. 5, pp. 1685-1702, 1998.

[10] M. A. Rami, X. Y. Zhou, "Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls," IEEE Transactions on Automatic Control , vol. 45, no. 6, pp. 1131-1143, 2000.

[11] D. D. Yao, S. Zhang, X. Y. Zhou, "Stochastic linear-quadratic control via semidefinite programming," SIAM Journal on Control and Optimization , vol. 40, no. 3, pp. 801-823, 2001.

[12] M. Ait Rami, J. B. Moore, X. Y. Zhou, "Indefinite stochastic linear quadratic control and generalized differential Riccati equation," SIAM Journal on Control and Optimization , vol. 40, no. 4, pp. 1296-1311, 2001.

[13] A. Ferrante, G. Marro, L. Ntogramatzidis, "A parametrization of the solutions of the finite-horizon LQ problem with general cost and boundary conditions," Automatica , vol. 41, no. 8, pp. 1359-1366, 2005.

[14] D. Bertsimas, D. B. Brown, "Constrained stochastic LQC: a tractable approach," IEEE Transactions on Automatic Control , vol. 52, no. 10, pp. 1826-1841, 2007.

[15] Y. Hu, X. Y. Zhou, "Constrained stochastic LQ control with random coefficients, and application to portfolio selection," SIAM Journal on Control and Optimization , vol. 44, no. 2, pp. 444-466, 2005.

[16] S. Ko, R. R. Bitmead, "Optimal control for linear systems with state equality constraints," Automatica , vol. 43, no. 9, pp. 1573-1582, 2007.

[17] A. Kojima, M. Morari, "LQ control for constrained continuous-time systems," Automatica , vol. 40, no. 7, pp. 1143-1155, 2004.

[18] Z. Zhou, C. Randy, "An algorithm for state constrained stochastic linear-quadratic control," in Proceedings of the American Control Conference, pp. 1476-1481, San Francisco, Calif, USA, June 2011.

[19] Y. L. Huang, W. H. Zhang, "Study on stochastic linear quadratic optimal control with constraint," Acta Automatica Sinica , vol. 32, no. 2, pp. 246-254, 2006.

[20] G. Li, W. Zhang, "Discrete-time indefinite stochastic linear quadratic optimal control with equality constraints," in Proceeding of the Chinese Control and Decision Conference, pp. 4999-5004, Guiyang, China, May 2013.

[21] H. Dong, Z. Wang, D. W. C. Ho, H. Gao, "Variance-constrained _{H ∞} filtering for a class of nonlinear time-varying systems with multiple missing measurements: the finite-horizon case," IEEE Transactions on Signal Processing , vol. 58, no. 5, pp. 2534-2543, 2010.

[22] L. Ma, Y. Bo, Y. Zhou, Z. Guo, "Error variance-constrained _{H ∞} filtering for a class of nonlinear stochastic systems with degraded measurements: the finite horizon case," International Journal of Systems Science , vol. 43, no. 12, pp. 2361-2372, 2012.

[23] M. Ait Rami, X. Chen, X. Y. Zhou, "Discrete-time indefinite LQ control with state and control dependent noises," Journal of Global Optimization , vol. 23, no. 3-4, pp. 245-265, 2002.

[24] D. P. Bertsekas Constrained Optimization and Lagrange Multiplier Methods , Athena Scientific, Belmont, Mass, USA, 1996.

[25] A. Albert, "Conditions for positive and nonnegative definiteness in terms of pseudoinverses," SIAM Journal on Applied Mathematics , vol. 17, no. 2, pp. 434-440, 1969.