Nested Maximum Entropy Designs for Computer Experiments

Abstract

Presently, computer experiments with multiple levels of accuracy are widely applied in science and engineering. This paper introduces a class of nested maximum entropy designs for such computer experiments. A multi-layer DETMAX algorithm is proposed to construct nested maximum entropy designs. Based on nested maximum entropy designs, we also propose an integer-programming procedure to specify the sample sizes in multi-fidelity computer experiments. Simulated annealing techniques are used to tackle complex optimization problems in the proposed methods. Illustrative examples show that the proposed nested entropy designs can yield better prediction results than nested Latin hypercube designs in the literature and that the proposed sample-size determination method is effective.

1. Introduction

With the rapid development of computer simulation technology, computer experiments have been widely used in the manufacturing industry, system engineering, natural science, and other fields [1,2]. Statistical designs for computer experiments have received considerable attention [3,4,5,6]. In many real applications, multi-fidelity computer experiments with different levels of accuracy are often encountered. More accurate experiments need longer computational time, while faster experiments have relatively low accuracy. It seems inefficient to study them separately. Therefore, many authors have studied statistical analysis for integrating multi-fidelity computer experiments with different levels of accuracy [7,8,9,10]. The experimental design issue for such computer experiments has been investigated by [11,12,13,14,15], and many others.

Shannon entropy is a basic concept of information theory [16]. Ref. [17] introduced the use of Shannon entropy as a measure of experimental information for spatial design. He argued that the experiment that minimizes the expected entropy, which is the entropy of the posterior distribution, can provide the largest amount of information for prediction. Ref. [18] proved that minimizing the posterior entropy is equivalent to maximizing the entropy of the prior distribution. The maximum entropy criterion was subsequently adopted as one of the major approaches for computer experiments [1]. Ref. [19] proposed the DETMAX algorithm [20] to efficiently construct the maximum entropy design. Ref. [21] proposed a sequential framework for conducting computer experiments with the maximum entropy criterion. However, to the best of our knowledge, there is no research on the maximum entropy design for multi-fidelity computer experiments.

In this paper, we introduce a class of nested maximum entropy designs with multi-layer structures for multi-fidelity computer experiments. Unlike [14]’s nested Latin hypercube designs, a nested maximum entropy design allows for flexibility in sample sizes as the sample size in each larger design does not need to be a multiple of that in a smaller one. Since computer experiments with higher accuracy are more important, we first consider the optimization of lower layers in the nested maximum entropy designs. Based on a layer-by-layer optimization strategy [11,22], a multi-layer DETMAX algorithm is proposed to construct such nested maximum entropy designs. The algorithm begins to generate a maximum entropy design for the lowest layer. Subsequently, we fix the design points optimized in lower layers, and optimize the current layer according to the maximum entropy criterion step by step, until the whole design is completely optimized. Based on nested maximum entropy designs, we also propose an integer-programming procedure to specify the sample sizes in multi-accuracy computer experiments under the budget constraint. Simulated annealing techniques [23] are adopted to tackle complex optimization problems in the proposed approaches. Illustrative examples are presented to show the effectiveness of our methods.

The contributions of this paper are summarized as follows. First, we introduce a new type of model-based design for multi-fidelity computer experiments based on information entropy. Second, our methods are flexible in sample sizes of multi-fidelity computer experiments. Third, this paper first provides an entropy-based strategy to determine sample sizes of multi-fidelity computer experiments.

The rest of this paper is organized as follows. Section 2 reviews the concept of maximum entropy designs for a single level of accuracy. In Section 3, the DETMAX algorithm is extended to construct nested maximum entropy designs. Section 4 deals with the sample-size determination of multi-accuracy computer experiments. Section 5 provides numerical examples. We end this paper with some concluding remarks in Section 6.

2. Review of Maximum Entropy Designs

In this section, we give a review of maximum entropy designs. Consider the following Kriging model,

(1)$y\left(\mathbf{x}\right)=\mathbf{f}{\left(\mathbf{x}\right)}^{{}^{\prime }}\mathit{\beta }+Z\left(\mathbf{x}\right),$

where $\mathbf{x}\in {\mathbb{R}}^p$, $\mathbf{f}\left(\mathbf{x}\right)={(f_1\left(\mathbf{x}\right),\dots ,f_m\left(\mathbf{x}\right))}^{{}^{\prime }}$ is a prespecified vector of regression functions, $\mathit{\beta }={({\beta }_1,\dots ,{\beta }_m)}^{{}^{\prime }}$ presents a vector of unknown regression coefficients, $Z\left(\mathbf{x}\right)$ is a stationary Gaussian process with zero mean, variance ${\sigma }^2$, and the correlation function

$R({\mathbf{x}}_1-{\mathbf{x}}_2|\mathit{\varphi })=exp\left[-\sum _{i=1}^p{\varphi }_i{(x_{1i}-x_{2i})}^2\right]$

for ${\mathbf{x}}_1={(x_{11},\dots ,x_{1p})}^{{}^{\prime }}$ and ${\mathbf{x}}_2={(x_{21},\dots ,x_{2p})}^{{}^{\prime }}$ and a vector of correlation parameters $\mathit{\varphi }={({\varphi }_1,\dots ,{\varphi }_p)}^{{}^{\prime }}$ with ${\varphi }_i>0$ for $i=1,\dots ,p$, denoted by $Z\sim \mathrm{GP}(0,{\sigma }^2,\mathit{\varphi })$.

Let $D=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}$ and $\mathbf{y}={(y\left({\mathbf{x}}_1\right),\dots ,y\left({\mathbf{x}}_n\right))}^{{}^{\prime }}$ represent a design with n runs and the corresponding vector of response values, respectively. Ref. [18] used the expected change in information to evaluate the design D. Since entropy is the negative of information, maximizing the expected change in information is equivalent to maximizing the entropy of the responses at the points in the design, denoted by $H\left({\mathbf{Y}}_D\right)$. In the context of Gaussian process models, the design relevant part of $H\left({\mathbf{Y}}_D\right)$ is $\log \left(\det \left({\sigma }^2\mathbf{R}\right)\right)/2$, where $\mathbf{R}$ is the $n\times n$ correlation matrix whose $(i,j)$th entry is $R({\mathbf{x}}_i-{\mathbf{x}}_j|\mathit{\varphi })$. Therefore, a maximum entropy design D maximizes the determinant of the covariance matrix of the set of responses $\mathbf{y}$ at the points in the design [2],

(2)$\underset{D}{max}\det \left({\sigma }^2\mathbf{R}\right).$

Because of the independence between ${\sigma }^2$ and the design D, (2) is equivalent to

(3)$\underset{D}{max}\det \left(\mathbf{R}\right).$

Here the vector of correlation parameters, $\mathit{\varphi }$, in $\mathbf{R}$ are assumed to be known.

3. Construction of Nested Maximum Entropy Designs

In this section, we extend the maximum entropy design to the case of multiple layers and propose the corresponding construction algorithms.

3.1. Nested Maximum Entropy Designs

Nested designs with multiple layers are usually used for multi-fidelity computer experiments [11,12,22,24,25]. Assume that we have a computer experiment with K levels, and the accuracy declines gradually from level 1 to level K. For each $k=1,\dots ,K$, the Kriging model for computer experiments at the kth level of accuracy is

$y_k\left({\mathbf{x}}_i\right)=\mathbf{f}{\left({\mathbf{x}}_i\right)}^{\prime }{\mathit{\beta }}_k+Z_k\left({\mathbf{x}}_i\right),i=1,\dots ,n_k,$

where $\mathbf{f}\left({\mathbf{x}}_i\right)$ and ${\mathit{\beta }}_k$ are straightforward extensions to those in (1), and $Z_k\sim \mathrm{GP}(0,{\sigma }_k^2,{\mathit{\varphi }}_k)$. ${\mathbf{R}}_k$ is the $n_k\times n_k$ matrix whose $(i,j)$th entry is $R({\mathbf{x}}_i-{\mathbf{x}}_j|{\mathit{\varphi }}_k)$ with the known vector ${\mathit{\varphi }}_k$.

Let a nested design $D^{\left(K\right)}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_K}\}$ with K layers $D^{\left(1\right)}\subset \cdots \subset D^{\left(K\right)}$, where $D^{\left(k\right)}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_k}\}$ denotes the kth layer of the nested design for each $k=1,\dots ,K$ and $n_1<\cdots <n_K$. The vector $\mathbf{s}=(n_1,\dots ,n_K)$ represents the structure of $D^{\left(K\right)}$. Please note that $D^{\left(k\right)}$ with smaller k is used for computer experiments with higher accuracy, which are more important. Similar to some definitions of optimal nested designs [11,22], we call $D^{\left(K\right)}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_K}\}$ a nested maximum entropy design if the following conditions hold: the first layer $D^{\left(1\right)}$ is a maximum entropy design that maximizes $\det \left({\mathbf{R}}_1\right)$; for each $k=2,\dots ,K$, $D^{\left(k\right)}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_k}\}$ maximizes $\det \left({\mathbf{R}}_k\right)$ with fixed optimized $D^{(k-1)}$.

By the above definition, a nested maximum entropy design can be constructed by a sequential algorithm; see Algorithm 1.

Algorithm 1 Construction of a nested maximum entropy design with K layers

Initialization: $k=1$, randomly construct the first layer $D^{\left(1\right)}$. Optimize $D^{\left(1\right)}$ using the maximum criteria to obtain $D_{\mathrm{best}}^{\left(1\right)}$. Recursive step: for $k=2,\dots ,K$ do    Enlarge $D_{\mathrm{best}}^{(k-1)}$ to $D^{\left(k\right)}$, where $D^{\left(k\right)}$ is the kth layer of the design with the structure $\mathbf{s}$. Maximize the entropy $\det \left({\mathbf{R}}_k\right)$ corresponding to $D^{\left(k\right)}$ by optimizing $D^{\left(k\right)}\setminus D_{\mathrm{best}}^{(k-1)}$. Output $D_{\mathrm{best}}^{\left(k\right)}$. end for

3.2. Multi-Layer DETMAX Algorithm

This subsection presents optimization algorithms for constructing each layer of a nested maximum entropy design.

The maximum entropy design can be obtained by a DETMAX-based algorithm [19]. It is optimized through a series of “excursions” to improve the det($\mathbf{R}$) corresponding to the current design by adding or removing appropriate points from the current design until the det($\mathbf{R}$) for the resulting n-point design cannot be increased. Except for the initial and final designs which have exactly n points, the number of chosen points of any designs constructed on this excursion can be greater or less than n. We extend this algorithm to the multi-layer case.

The flow chart in Figure 1 describes the procedure of the multi-layer DETMAX algorithm. The layer-by-layer optimization strategy [11,22] is adopted here. First, the initial design of the first layer, denoted as $D_0^{\left(1\right)}$, is randomly generated and then optimized. If $D_{\mathrm{best}}^{\left(1\right)}$ obtained through a series of excursions meets the stopping condition, then the first layer has been completed. Subsequently, optimize the second layer with the first layer fixed. Each layer is optimized in turn until the last layer is optimized, and then output $D_{\mathrm{best}}^{\left(K\right)}$.

We give the details for optimizing each layer in the above algorithm. Suppose we now optimize the kth layer of the design $D^{\left(k\right)}$. One excursion starts with the $n_k$-point design and ends when the number of points in $D^{\left(k\right)}$ reaches exactly $n_k$ again. The procedure for making excursions is described as follows. Let $ϵ$ denote a prespecified threshold.

Step 1. Add a point at which the variance function ${\sigma }_{0|D^{\left(k\right)}}^2$ is largest, or subtract a point corresponding to the maximum element of the diagonal of ${\mathbf{R}}_k^{-1}$.

Step 2. The current design $D^{\left(k\right)}$ has $n_k^{{}^{\prime }}$ points.

If $n_k^{{}^{\prime }}>n_k$, remove a point if $D^{\left(k\right)}$ is not in $F_k$ and add a point otherwise.

If $n_k^{{}^{\prime }}<n_k$, add a point if $D^{\left(k\right)}$ is not in $F_k$ and remove a point otherwise.

The new design updated by this step has $n_k^{\mathrm{new}}$ points and correlation matrix ${\mathbf{R}}_k^{\mathrm{new}}$.

Step 3. If $n_k^{\mathrm{new}}\ne n_k$, back to Step 2. Otherwise, if $\mid \det \left({\mathbf{R}}_k^{\mathrm{new}}\right)-\det \left({\mathbf{R}}_k\right)\mid <ϵ$, stop the procedure and output $D_{\mathrm{best}}^{\left(k\right)}$; if $\mid \det \left({\mathbf{R}}_k^{\mathrm{new}}\right)-\det \left({\mathbf{R}}_k\right)\mid \ge ϵ$, do the following:

If det(${\mathbf{R}}_k^{\mathrm{new}}$) < det(${\mathbf{R}}_k$), let ${\mathbf{R}}_k={\mathbf{R}}_k^{\mathrm{new}}$, place all the designs generated on this excursion into $F_k$. Go to Step 1.

If $\det \left({\mathbf{R}}_k^{\mathrm{new}}\right)\ge \det \left({\mathbf{R}}_k\right)$, let ${\mathbf{R}}_k$ = ${\mathbf{R}}_k^{new}$, $F_k=⌀$, $D_{\mathrm{best}}^{\left(k\right)}=D^{\left(k\right)}$ and ${\mathbf{R}}_k^{\mathrm{best}}={\mathbf{R}}_k$. Go to Step 1.

In Step 1, the selection of whether to add or subtract a point is made randomly. The best point ${\mathbf{x}}_0$ is obtained by maximizing ${\sigma }_{0|D^{\left(k\right)}}^2$, which is given by

${\sigma }_{0|D^{\left(k\right)}}^2\left({\mathbf{x}}_0\right)={\sigma }_k^2(1-{\mathbf{r}}_k^{{}^{\prime }}{\mathbf{R}}_k^{-1}{\mathbf{r}}_k),$

where ${\mathbf{r}}_k={(R({\mathbf{x}}_0-{\mathbf{x}}_1|{\mathit{\varphi }}_k),\dots ,R({\mathbf{x}}_0-{\mathbf{x}}_{n_k}|{\mathit{\varphi }}_k))}^{{}^{\prime }}$. To determine the best site ${\mathbf{x}}_0$ to add to the current design, we adopt a grid search procedure [26] for $p=2$, and the simulated annealing algorithm [23] for $p\ge 3$ (see Algorithm 2). The point we subtract in Step 1 is always in $D^{\left(k\right)}\setminus D_{\mathrm{best}}^{(k-1)}$.

Algorithm 2 Simulated annealing in excursions

Step 0: Input the starting temperature $T=T_0$$>0$, the ending temperature $T_{\mathrm{end}}$$>0$, the length of Markov chain L, search step size $\lambda$, Boltzmann’s constant $k_0=1$, reduction factor $\alpha$ ($0<\alpha <1$) and an initial solution $\mathbf{x}={\mathbf{x}}^{\left(0\right)}$, which is randomly generated in ${[0,1]}^p$. while $T>T_{\mathrm{end}}$ do    for $k=1,\dots ,L$, do      Step 1: ${\mathbf{x}}_{\mathrm{new}}=\mathbf{x}+\lambda \mathbf{u}$, where $\mathbf{u}$ is generated by sampling random values from $N_p(0,1)$.      Step 2:      if ${\sigma }_{0|D^{\left(k\right)}}^2\left({\mathbf{x}}_{\mathrm{new}}\right)>{\sigma }_{0|D^{\left(k\right)}}^2\left(\mathbf{x}\right)$, then         $\mathbf{x}={\mathbf{x}}_{\mathrm{new}}$, $\lambda =0.99\lambda$.      else         Randomly generate a real number r in $[0,1]$.         if $r<exp\left(-\frac{{\sigma }_{0|D^{\left(k\right)}}^2\left({\mathbf{x}}_{\mathrm{new}}\right)-{\sigma }_{0|D^{\left(k\right)}}^2\left(\mathbf{x}\right)}{k_{0}T}\right)$ then           $\mathbf{x}={\mathbf{x}}_{\mathrm{new}}$, $\lambda =0.99\lambda$.         else           Go back to Step 1.         end if      end if    end for    Step 3: T = $\alpha$T. end while Step 4: Output the best solution $\mathbf{x}$.

Since the design points are bounded, the determinant of the corresponding covariance matrix is bounded. According to the monotone convergence theorem, our algorithm can converge to a limit after considerable iteration times. However, because of the nonconvex feature of the problem like other design construction problems [27], the limit may not be the global solution. To better approximate the global solution, the above algorithm can be conducted repeatedly with several random initial designs, and the final output is the best one among the corresponding results. Several two-dimensional nested maximum entropy designs constructed by the proposed algorithm can be seen in Figure 2.

4. Sample-Size Determination of Multi-Accuracy Computer Experiments

A related issue to experimental design is sample-size determination. Both sample-size determination and experimental design should be implemented before we obtain data. Sample-size determination should be considered earlier than experimental design, since the latter is usually conducted with given sample sizes. The problem of sample-size determination in computer experiments has attracted much attention in the literature; see [28,29,30], among others. However, these studies focused on computer experiments with one level of accuracy, and there is little work for the case of more than one level. In this section, we propose a method to determine sample sizes of multi-accuracy computer experiments based on the entropy criterion.

There is no data available when we implement sample-size determination. For multi-fidelity computer experiments, we consider the maximum entropy of possible nested designs with different sample sizes. We first introduce a concept of integrative entropy, which is an extension to (3). For a K-layer nested maximum entropy design D with structure $\mathbf{s}=(n_1,\dots ,n_K)$, the integrative entropy of D is defined by

(4)$En=\sum _{k=1}^K{w}_k\det \left({\mathbf{R}}_k\right),$

where $w_k$ is a non-negative weight and ${\mathbf{R}}_k$ is the correlation matrix of layer k for $k=1,\dots ,K$. The choice of weights, $w_k=\sqrt[p]{n_k-1}$, in (4) can be found in [15,27]. Please note that the integrative entropy in (4) is a function that depends on $n_1,\dots ,n_K$, i.e., $En=En\left(D\right)=En(n_1,\dots ,n_K)$.

For a computer experiment with K levels of accuracy, let $b_k$ denote the cost at the kth level, $k=1,\dots ,K$. Assume that the total budget is B. We specify the sample sizes $n_1,\dots ,n_K$ through maximizing the integrative entropy under the budget constraint, i.e., solving the optimization problem,

(5)$\begin{array}{ccc}& & maxEn(n_1,\dots ,n_K),\hfill \\ & & \mathrm{s}.\mathrm{t}.\sum _{k=1}^K{n}_k{b}_k\le B,n_1,\dots ,n_K\in \mathbb{N},\hfill \end{array}$

where $\mathbb{N}$ denotes the set of non-negative integers.

The above optimization problem is a nonlinear knapsack problem [31]. There are many techniques for this problem such as the branch-and-bound algorithm, dynamic programming, and the decomposition method. Please note that the objective function in (5) is very complicated. We adopt the simulated annealing algorithm to solve this integer-programming problem since the algorithm possesses the features of avoiding local optimum, high flexibility, and good convergence properties. Due to the complexity of this problem, we select multiple initial points to run Algorithm 3, and output the solution corresponding to the greatest objective function value. Let $\mathrm{randint}(1,m)$ denote an integer randomly chosen from $\{1,\dots ,m\}$, $m\in \mathbb{N}$.

Algorithm 3 Simulated annealing in sample-size determination

Step 0: Initialize $T=T_0=100$, $T_{\mathrm{end}}=1$, $L=100$, $k_0=1$, $\alpha =0.7$. Randomly generate an initial solution $\mathbf{n}={\mathbf{n}}^{\left(0\right)}={(n_1,\dots ,n_K)}^{{}^{\prime }}$. Step 1: Repeat Step 0 until ${\displaystyle \sum _{k=1}^K{n}_k{b}_k\le B}$ is satisfied. while $T>T_{\mathrm{end}}$ do    for $k=1,\dots ,L$, do      Step 2: $n_s^{{}^{\prime }}=n_s+{(-1)}^t$, where $s=\mathrm{randint}(1,K)$ and $t=\mathrm{randint}(1,2)$. ${\mathbf{n}}_{\mathrm{new}}={(n_1,\dots ,n_s^{{}^{\prime }},\dots ,n_K)}^{{}^{\prime }}$.      Step 3:      if $En\left({\mathbf{n}}_{\mathrm{new}}\right)>En\left(\mathbf{n}\right)$, then         $\mathbf{n}={\mathbf{n}}_{\mathrm{new}}$.      else         Randomly generate a real number r in $[0,1]$.         if $r<exp\left(-\frac{En\left({\mathbf{n}}_{\mathrm{new}}\right)-En\left(\mathbf{n}\right)}{k_{0}T}\right)$ then           $\mathbf{n}={\mathbf{n}}_{\mathrm{new}}$.         else           Go back to Step 2.         end if      end if    end for    Step 4: T=$\alpha$T. end while Step 5: Output the best solution $\mathbf{n}$.

5. Illustrations

In this section, we provide several examples to illustrate the effectiveness of the proposed methods. Examples 1, 2, and 3 demonstrate the prediction performance of the proposed nested maximum entropy designs, which are constructed by applying the algorithm in Section 3. Example 4 presents an application of the proposed sample-size determination method in Section 4.

Here we consider the case of $K=2$. In Examples 1–3, let $D_h=\{{\mathbf{x}}_1^h,\dots ,{\mathbf{x}}_{n_1}^h\}$ and $D_l=\{{\mathbf{x}}_1^l,\dots ,{\mathbf{x}}_{n_2}^l\}$ represent the design sets of the high-accuracy experiment (HE) with $n_1$ runs and the design sets of the low-accuracy experiment (LE) with $n_2$ runs, respectively. We denote the HE response associated with $D_h$ as $y_h$, and the LE response associated with $D_l$ as $y_l$. The prediction model in [32] is used, and the corresponding predictor of $y_h$ is denoted by ${\widehat{y}}_h$. The FNLHD [11] is compared with the proposed entropy design. Prediction performance is evaluated with the empirical mean squared prediction error (MSPE),

$\mathrm{MSPE}=\frac{1}{10000}\sum _{i=1}^{10000}{[y_h\left({\mathbf{x}}_i\right)-{\widehat{y}}_h\left({\mathbf{x}}_i\right)]}^2,$

where $\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{10000}\}$ is generated by Latin hypercube sampling ([33]).

5.1. Example 1

In this example, we use the following function from [19],

$y_h\left(\mathbf{x}\right)=[1-exp(-1/\left(2x_2\right))](2300x_1^3+1900x_1^2+2092x_1+60)/(100x_1^3+500x_1^2+4x_1+20),\mathbf{x}\in {[0,1]}^2,$

as the HE function. In addition, the LE function is:

$\begin{array}{ccc}& & y_l=[1+y_h(x_1+1/20,x_2+1/20)+y_h(x_1+1/20,max(0,x_2-1/20))\hfill \\ & & +y_h(x_1-1/20,x_2-1/20)+y_h(x_1-1/20,max(0,x_2-1/20))]/4\hfill \end{array}$

We implement the two design methods with $\mathbf{s}=(16,25)$ and $\mathbf{s}=(25,32)$, respectively. In our method, ${\mathit{\varphi }}_1={(200,200)}^{{}^{\prime }}$, ${\mathit{\varphi }}_2={(50,50)}^{{}^{\prime }}$, $ϵ={10}^{-15}$, and ${\sigma }_1^2={\sigma }_2^2=1$. MSPEs over 100 repetitions are shown in Figure 3. It can be seen that the proposed method outperforms FNLHD.

5.2. Example 2

In this example, the following function from [34] is exploited,

$y_h\left(\mathbf{x}\right)=2/3exp(x_1+x_2)-x_{4}sin\left(x_3\right)+x_3,\mathbf{x}\in {[0,1]}^4,$

as the HE function. In addition, the LE function is:

$\begin{array}{ccc}& & y_l=1.2y_h-1.\hfill \end{array}$

In Example 2, we use Algorithm 2 to optimize the nested maximum entropy designs with $\mathbf{s}=(10,15)$ and $\mathbf{s}=(20,24)$ for $p=4$. Take ${\mathit{\varphi }}_1={(200,200,200,200)}^{{}^{\prime }}$, ${\mathit{\varphi }}_2={(10,10,10,10)}^{{}^{\prime }}$, ${\sigma }_1^2={\sigma }_2^2=1$, $ϵ={10}^{-15}$, $L=100$, $\lambda =0.3$, $k_0=1$, $T_0=100$, and $T_{\mathrm{end}}=1$ in the proposed method. Table 1 displays the MSPEs over 100 replicates for the proposed method with four values of reduction factors $\alpha$, 0.99, 0.9, 0.8, 0.7 in Algorithm 2. The results from FNLHD are also shown. We can see that our designs can yield better prediction results compared with FNLHDs. Moreover, the proposed simulated annealing algorithm is insensitive to the selection of parameter $\alpha$.

5.3. Example 3

In this example, let the following function from [34],

$y_h\left(\mathbf{x}\right)=\frac{x_1}{2}[\sqrt{1+(x_2+x_3^2)x_4/x_1^2}-1]+(x_1+3x_4)\times exp[1+sin\left(x_3\right)],\mathbf{x}\in {[0,1]}^4,$

act as the HE function. In addition, the LE function is:

$\begin{array}{ccc}& & y_l=[1+sin\left(x_1\right)/10]y_h(x_1,x_2,x_3,x_4)-2x_1+x_2^2+x_3^2+0.5.\hfill \end{array}$

In Example 3, $\mathbf{s}=(10,25)$ and $\mathbf{s}=(15,22)$, respectively. Algorithm 2 is used again in the proposed method for $p=4$. We set ${\mathit{\varphi }}_1={(200,200,200,200)}^{{}^{\prime }}$, ${\mathit{\varphi }}_2={(10,10,10,10)}^{{}^{\prime }}$, ${\sigma }_1^2={\sigma }_2^2=1$, $ϵ={10}^{-15}$, $L=100$, $\lambda =0.3$, $k_0=1$, $T_0=100$, $T_{\mathrm{end}}=1$, and four values of $\alpha$, 0.99, 0.9, 0.8, 0.7, as in Section 5.2. Table 2 presents the MSPEs over 100 replicates. Similar to the conclusions in Section 5.2, the proposed approach is better than FNLHD, and our algorithm is insensitive to the selection of parameter $\alpha$.

5.4. Example 4

Consider the experiments with two levels of accuracy and four input variables. Suppose that the costs of running an HE and LE are 40 thousand RMB and 25 thousand RMB, respectively. Let the total budget be one million RMB. According to the method in Section 4, we solve the optimization problem,

$\begin{array}{l}maxEn(n_1,n_2),\\ \mathrm{s}.\mathrm{t}.40n_1+25n_2\le 1000,n_1,n_2\in \mathbb{N},\end{array}$

where $En(n_1,n_2)$ is defined by (4). We run Algorithm 3 with ${\mathit{\varphi }}_1={(200,200,200,200)}^{{}^{\prime }}$, ${\mathit{\varphi }}_2={(10,10,10,10)}^{{}^{\prime }}$, ${\sigma }_1^2={\sigma }_2^2=1$, $L=100$, $\lambda =0.3$, $k_0=1$, $T_0=100$, $T_{\mathrm{end}}=1$, and five initial points. The optimal sample sizes are given by $(n_1,n_2)=(10,24)$, corresponding to the integrative entropy value 3.67. The corresponding nested design is shown in Table 3, where the first ten rows display the HE design.

6. Concluding Remarks

In this paper, we have introduced a new class of nested designs, nested maximum entropy designs, for multi-fidelity computer experiments. Such designs possess flexible run numbers in each layer and can provide a considerable amount of information for prediction. A multi-layer DETMAX algorithm has been proposed to construct nested maximum entropy designs. The related maximum entropy criterion has been used to determine the sample sizes for each level of accuracy in multi-fidelity computer experiments.

There are some limitations of our work. Due to the complexity of the optimization problem with the entropy criterion, the proposed algorithms can only handle relatively simple cases, such as relatively low dimensions and relatively small run sizes. In addition, extensions of the proposed approaches can be made in several directions. First, the corresponding designs for finite design regions [35] can be studied in the future. Second, our methods can be modified to accommodate both qualitative and quantitative factors [9,36,37]. Third, sequential frameworks [21,32,38,39] for multi-fidelity computer experiments can be developed by the proposed entropy criterion.

Author Contributions

Conceptualization, W.M. and S.X.; methodology, W.M.; software, C.L.; validation, W.M.; formal analysis, W.M. and C.L.; investigation, W.M.; resources S.X.; data curation, C.L.; writing—original draft preparation, W.M. and C.L.; writing—review and editing, W.M., C.L. and S.X.; visualization, C.L.; supervision, W.M.; project administration, S.X.; funding acquisition, S.X. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

Not applicable.

Acknowledgments

The authors gratefully acknowledge the editors and reviewers for their professional comments.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The procedure of multi-layer DETMAX algorithm.

Figure 2. Some nested maximum entropy designs with [Forumla omitted. See PDF.]. The first four rows display nested maximum entropy designs with [Forumla omitted. See PDF.] and the last row shows five nested maximum entropy designs with [Forumla omitted. See PDF.]. The blue dots denote the points in the first layer. Adding the points depicted by red stars, we obtain the second layer. In the last row, the whole design comprises all the points including green plus signs. The correlation parameters are set as: for two-layer designs, (I) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (II) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (III) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (IV) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], and (V) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.]; for three-layer designs, [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], [Forumla omitted. See PDF.].

Figure 2. Some nested maximum entropy designs with [Forumla omitted. See PDF.]. The first four rows display nested maximum entropy designs with [Forumla omitted. See PDF.] and the last row shows five nested maximum entropy designs with [Forumla omitted. See PDF.]. The blue dots denote the points in the first layer. Adding the points depicted by red stars, we obtain the second layer. In the last row, the whole design comprises all the points including green plus signs. The correlation parameters are set as: for two-layer designs, (I) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (II) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (III) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], (IV) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], and (V) [Forumla omitted. See PDF.], [Forumla omitted. See PDF.]; for three-layer designs, [Forumla omitted. See PDF.], [Forumla omitted. See PDF.], [Forumla omitted. See PDF.].

Figure 3. The boxplots of MSPEs in Example 1. Class 1 and 2 correspond to nested maximum entropy designs and FNLHDs, respectively. (a) [Forumla omitted. See PDF.]; (b) [Forumla omitted. See PDF.].

Table 1

MSPEs in Example 2 (standard deviations in parentheses).

	$\mathit{s}=(10,15)$	$\mathit{s}=(20,24)$
	MSPE	MSPE
Proposed ($\alpha =0.99$)	0.3122 (0.0535)	0.3321 (0.0345)
Proposed ($\alpha =0.9$)	0.3100 (0.0580)	0.3330 (0.0302)
Proposed ($\alpha =0.8$)	0.3195 (0.0556)	0.3295 (0.0354)
Proposed ($\alpha =0.7$)	0.3149 (0.0596)	0.3264 (0.0338)
FNLHD	0.3932 (0.0620)	0.3745 (0.0453)

Table 2

MSPEs in Example 3 (standard deviations in parentheses).

	$\mathit{s}=(10,25)$	$\mathit{s}=(15,22)$
	MSPE	MSPE
Proposed ($\alpha =0.99$)	1.2816 (0.1726)	1.3878 (0.2737)
Proposed ($\alpha =0.9$)	1.3397 (0.2759)	1.3736 (0.2785)
Proposed ($\alpha =0.8$)	1.3026 (0.2108)	1.4316 (0.2973)
Proposed ($\alpha =0.7$)	1.3352 (0.2526)	1.4249 (0.3299)
FNLHD	1.4756 (0.3505)	1.5116 (0.3773)

Table 3

The best design with structure $(10,24)$ in Section 5.4.

Run	${\mathit{x}}_1$	${\mathit{x}}_2$	${\mathit{x}}_3$	${\mathit{x}}_4$
1	0.528	0.960	0.113	0.525
2	0.389	0.404	0.227	0.895
3	0.251	0.866	0.110	0.738
4	0.256	0.343	0.684	0.205
5	0.315	0.572	0.404	0.783
6	0.777	0.433	0.434	0.893
7	0.563	0.214	0.566	0.196
8	0.160	0.873	0.787	0.401
9	0.807	0.775	0.712	0.271
10	0.381	0.688	0.702	0.763
11	0.999	0.965	0.125	0.904
12	0	0	0	1
13	1	0.581	0	0
14	1	1	1	1
15	1	1	0.497	0
16	1	0.587	1	0
17	0	0.381	1	1
18	0	0.367	0	0.396
19	0	1	0	0.860
20	0.301	1	1	1
21	0.758	1	1	0.397
22	1	0.159	1	1
23	1	0	0	0.397
24	1	0.582	0.527	0.615

References

1. Sacks, J.; Welch, W.J.; Mitchell, T.J.; Wynn, H.P. Design and analysis of computer experiments. Stat. Sci.; 1989; 4, pp. 409-423.

2. Santner, T.J.; Williams, B.J.; Notz, W.I. The Design and Analysis of Computer Experiments; Springer: New York, NY, USA, 2018; Volume 2.

3. Gryder, R.W.; Wilson, S.R.; Swieringa, K.A.; Edwards, D.J. Space-filling designs for multi-layer nested factors. Qual. Eng.; 2019; 31, pp. 269-278.

4. Mu, W.; Wei, Q.; Cui, D.; Xiong, S. Best Linear Unbiased Prediction for Multifidelity Computer Experiments. Math. Probl. Eng.; 2018; 2018, pp. 1-7.

5. Shang, B.; Apley, D.W. Fully-sequential space-filling design algorithms for computer experiments. J. Qual. Technol.; 2021; 53, pp. 173-196.

6. Wang, Y.; Sun, F.; Xu, H. On design orthogonality, maximin distance, and projection uniformity for computer experiments. J. Am. Stat. Assoc.; 2022; 117, pp. 375-385.

7. Kennedy, M.C.; O’Hagan, A. Predicting the output from a complex compute code when fast approximations are available. Biometrika; 2000; 87, pp. 1-13.

8. Mu, W.; Xiong, S. A class of space-filling designs and their projection properties. Stat. Probab. Lett.; 2018; 141, pp. 129-134.

9. Qian, P.Z.G.; Wu, H.; Wu, C.F.J. Gaussian Process Models for Computer Experiments with Qualitative and Quantitative Factors. Technometrics; 2008; 50, pp. 383-396.

10. Wei, Y.; Xiong, S. Bayesian integrative analysis for multi-fidelity computer experiments. J. Appl. Stat.; 2019; 46, pp. 1973-1987.

11. Chen, D.; Xiong, S. Flexible nested Latin hypercube designs for computer experiments. J. Qual. Technol.; 2017; 49, pp. 337-353.

12. Dash, S.; Mandal, B.N.; Parsad, R. On the construction of nested orthogonal Latin hypercube designs. Metrika; 2020; 83, pp. 347-353.

13. Guo, B.; Chen, X.P.; Liu, M.Q. Construction of Latin hypercube designs with nested and sliced structures. Stat. Pap.; 2020; 61, pp. 727-740.

14. Qian, P.Z. Nested Latin hypercube designs. Biometrika; 2009; 96, pp. 957-970.

15. Rennen, G.; Husslage, B.; Van Dam, E.R.; Den Hertog, D. Nested maximin Latin hypercube designs. Struct. Multidiscip. Optim.; 2010; 41, pp. 371-395.

16. Cover, T.M.; Thomas, J.A. Elements of Information Theory; 2nd ed. John-Wiley & Sons: Hoboken, NJ, USA, 2006.

17. Lindley, D.V. On a measure of the information provided by an experiment. Ann. Math. Stat.; 1956; 27, pp. 986-1005. [DOI: https://dx.doi.org/10.1214/aoms/1177728069]

18. Shewry, M.C.; Wynn, H.P. Maximum entropy sampling. J. Appl. Stat.; 1987; 14, pp. 165-170.

19. Currin, C.; Mitchell, T.; Morris, M.; Ylvisaker, D. Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J. Am. Stat. Assoc.; 1991; 86, pp. 953-963.

20. Mitchell, T.J. An Algorithm for the Construction of “D. Optimal” Experimental Designs. Technometrics; 1974; 16, pp. 203-210.

21. Jiang, Z.; Zhang, W.; Zhang, L. Sequential Maximum Entropy Approach to Design of Virtual Experiment. J. Syst. Simul.; 2007; 19, pp. 3876-3879.

22. Chen, D.; Xiong, S. Optimization of Nested Latin Hypercube Designs. J. Sys. Sci. Math. Scis; 2017; 37, 53.

23. Kirkpatrick, S.; Gelatt, C.D., Jr.; Vecchi, M.P. Optimization by simulated annealing. Readings Comput. Vis. Issues Probl. Princ. Paradig.; 1987; 220, pp. 606-615.

24. Chen, H.; Zhang, Y.; Yang, X. niform projection nested Latin hypercube designs. Stat. Pap.; 2021; 62, pp. 2031-2045.

25. Xu, J.; Duan, X.; Wang, Z.; Yan, L. A general construction for nested Latin hypercube designs. Stat. Probab. Lett.; 2018; 134, pp. 134-140.

26. Hebble, T.L.; Mitchell, T.J. “Repairing” response surface designs. Technometrics; 1972; 14, pp. 767-779.

27. Mu, W.; Xiong, S. On algorithmic construction of maximin distance designs. Commun. Stat.-Simul. Comput.; 2017; 46, pp. 7972-7985.

28. Harari, O.; Bingham, D.; Dean, A.; Higdon, D. Computer experiments: Prediction accuracy, sample size and model complexity revisited. Stat. Sin.; 2009; pp. 899-919.

29. Loeppky, J.L.; Sacks, J.; Welch, W.J. Choosing the sample size of a computer experiment: A practical guide. Technometrics; 2009; 51, pp. 366-376.

30. Sahama, T.R.; Diamond, N.T. Sample size considerations and augmentation of computer experiments. J. Stat. Comput. Simul.; 2001; 68, pp. 307-319.

31. Ma, L.; Zhang, H.; Liu, Y.; Ning, A. Advanced Operations Research Tutorial; Shanghai People’s Publishing House: Shanghai, China, 2015.

32. Xiong, S.; Qian, P.Z.; Wu, C.J. Sequential design and analysis of high-accuracy and low-accuracy computer codes. Technometrics; 2013; 55, pp. 37-46.

33. McKay, M.D.; Beckman, R.J.; Conover, W.J. A Comparison of Three Methods for Selecting Values of Input Variables in The Analysis of Output from a Computer Code. Technometrics; 1979; 21, pp. 239-245.

34. Cox, D.D.; Park, J.S.; Singer, C.E. A Statistical Method for Tuning a Computer Code to a Data Base. Comput. Stat. Data Anal.; 2001; 37, pp. 77-92.

35. Tan, M.H. Minimax designs for finite design regions. Technometrics; 2013; 55, pp. 346-358.

36. Han, G.; Santner, T.J.; Notz, W.I.; Bartel, D.L. Prediction for Computer Experiments Having Quantitative and Qualitative Input Variables. Technometrics; 2009; 51, pp. 278-288.

37. Zhou, Q.; Qian, P.Z.G.; Zhou, S. A Simple Approach to Emulation for Computer Models With Qualitative and Quantitative Factors. Technometrics; 2011; 53, pp. 266-273. [DOI: https://dx.doi.org/10.1198/TECH.2011.10025]

38. Le Gratiet, L.; Cannamela, C. Cokriging-based sequential design strategies using fast cross-validation techniques for multi-fidelity computer codes. Technometrics; 2020; 57, pp. 418-427.

39. Li, X.; Wang, X.; Xiong, S. A sequential design strategy for integrating low-accuracy and high-accuracy computer experiments. Commun. Stat.-Simul. Comput.; 2020; 52, pp. 817-824.