DLT-GAN: Dual-Layer Transfer Generative Adversarial Network-Based Time Series Data Augmentation Method

Abstract

In actual production processes, analysis and prediction tasks commonly rely on large amounts of time-series data. However, real-world scenarios often face issues such as insufficient or imbalanced data, severely impacting the accuracy of analysis and predictions. To address this challenge, this paper proposes a dual-layer transfer model based on Generative Adversarial Networks (GANs) aiming to enhance the training speed and generation quality of time-series data augmentation under small-sample conditions while reducing the reliance on large training datasets. This method introduces a module transfer strategy based on the traditional GAN framework which balances the training between the discriminator and the generator, thereby improving the model’s performance and convergence speed. By employing a dual-layer network structure to transfer the features of time-series signals, the model effectively reduces the generation of noise and other irrelevant features, improving the similarity of the generated signals’ characteristics. This paper uses speech signals as a case study, addressing scenarios where speech data are difficult to collect and the limited number of speech samples available for effective feature extraction and analysis. Simulated speech timbre generation is conducted, and the experimental results on the CMU-ARCTIC database show that, compared to traditional methods, this approach achieves significant improvements in enhancing the consistency of generated signal features and the model’s convergence speed.

1. Introduction

In many scenarios, analyzing and predicting system operations based on historical data is a crucial foundation for ensuring safety and production efficiency [1,2,3]. However, in actual production processes, the collected historical data often suffer from issues such as insufficient quantity and imbalance [2], and labeling a large amount of unlabeled data is equally challenging [3]. Time-series data occupy a significant portion of such data, for example, tasks involving speech data augmentation with limited samples using deep learning to enhance timbre consistency [4,5] or utilizing data-driven methods to analyze and predict the operational status of power systems [6]. These cases impose high requirements on the quantity and quality of time-series datasets. Therefore, it is necessary to develop new time-series data generation methods to compensate for the deficiencies in existing datasets.

To obtain sufficient samples and expand datasets, traditional methods primarily follow two paths for data augmentation: first, data augmentation through data processing techniques [7], such as introducing noise, interpolation smoothing, resampling, and time-stretching to extend existing data in cases of speech data scarcity, and second, building various simulation models and software based on mechanism analysis, such as ATP and PSCAD, which are used for simulating distribution network faults and collecting status signal data. However, these two methods exhibit certain limitations in practical applications. The data generated by simulations often differ from the real-world collected signal data, resulting in suboptimal performance of models trained on these data when applied in real-world scenarios [8]. On the other hand, while data augmentation methods increase the volume of data, they do not significantly improve data diversity or the consistency of key features [9]. As a result, models struggle to learn effective features beyond those in the existing data, ultimately affecting the model’s performance in real-world applications. Therefore, a new method for enhancing time-series data with small sample sizes is essential and urgent.

Since the proposal of Generative Adversarial Networks (GANs) in 2014 [10], research in this field has seen rapid growth in recent years. The applications of GANs have expanded to various fields, such as Natural Language Processing (NLP), and have gradually been applied to the generation and prediction of time-series and sequential data [11]. For example, SeqGAN [12] introduced a sequential data generation framework that employs a generator composed of RNNs with LSTM cells and a discriminator built using Convolutional Neural Networks (CNNs) to address the challenge of generating discrete time-series data. Quant GAN [13], a data-driven model, utilizes Temporal Convolutional Networks (TCN) with skip connections for both the generator and discriminator, aiming to capture long-term dependencies in financial time-series data. C-RNN-GAN [14] was developed as a GAN architecture capable of learning the features of continuous sequential data to generate music. Similarly, RCGAN [15] was introduced to enhance the generation of synthetic continuous time-series datasets with corresponding labels. TimeGAN [16], which combines unsupervised GAN networks with supervised autoregressive models, has been used to generate time-series data while maintaining temporal dynamics. SynSigGAN [17], applied in the medical field, is used to generate various types of continuous physiological and biomedical signal data. Finally, TCGAN [18] integrates the advantages of Convolutional Neural Networks and the generative capabilities of GANs, demonstrating strong performance in time-series data classification and clustering.

However, the aforementioned methods rely on sufficient training data for modeling. When faced with small sample sizes or limited labeled data, the models often encounter issues such as difficulty in convergence or suboptimal generation results [19], making it challenging to meet the needs of small sample data augmentation in real-world production environments. In recent studies, data generation methods based on GANs have gradually been applied to small sample problems. For instance, a study [20] proposed an improved small-sample time-series prediction method based on TimeGAN with an Attention mechanism which better covers the distribution of the original data and effectively reduces prediction errors under small sample conditions. Another study [21] introduced StyleFormerGAN-VC, a GAN model that combines Attention mechanisms and AdaIN to achieve better cross-lingual voice conversion. Additionally, a new data augmentation framework was presented [22], along with an improved GAN training method that incorporates feature selection and model-based transfer learning to generate target data for power system operation, supporting data-driven methods in power systems.

However, due to the semi-supervised learning architecture of GANs, which consists of two independent models—a generator and a discriminator—it is prone to issues such as gradient vanishing caused by the mismatch in training progress between the generator and the discriminator during the training process [23]. Therefore, within this architecture, a key challenge is ensuring that the forward propagation of generated data and the backpropagation of weight gradients are smoothly and reasonably transmitted between these two independent models, allowing for the convergence of adversarial training. At the same time, it is essential to maintain the diversity of generated data rather than being limited to repetitive patterns within a constrained feature space. Since the proposal of GANs, optimizing and improving this process has been a central research focus. Two primary optimization approaches have emerged in this regard: one focuses on constructing more reasonable objective functions to measure the difference between the distribution of generated data and real data, with a typical example being WGAN [24]; the other approach aims to optimize the weight update process [25], such as adjusting alternating training strategies (e.g., reducing the number of discriminator training steps while increasing those of the generator).

To address the aforementioned issues and enhance the efficiency and quality of time-series data synthesis under small sample conditions, this paper proposes a novel time-series data generation method suitable for small sample data named Dual-Layer Transfer GAN (DLT-GAN). To mitigate the overfitting and gradient vanishing problems that commonly arise in GANs when training with small samples, this paper introduces a module transfer strategy aimed at balancing the training intensity between the discriminator and the generator, ensuring stable model convergence. By constructing a dual-layer transfer network, the method significantly optimizes the generation of small-sample data and enhances the similarity between the generated data and the target data in the feature space. Experiments demonstrate that this method outperforms existing approaches in the task of voice timbre conversion with small speech datasets, with the generated data better matching the feature distribution of the target data, thereby exhibiting superior conversion performance.

2. Materials and Methods

2.1. Dual-Layer Transfer Framework

In recent years, machine learning and deep learning have achieved success in many fields by training models on large datasets. However, in real-world applications, many scenarios are faced with the challenge of having only a small number of samples or very few labeled samples, and labeling large amounts of unlabeled data requires significant human effort. Consequently, the question of how to enable model learning with a limited number of samples has become a critical issue. Few-shot learning has emerged as an approach aimed at developing models capable of solving problems using only a small number of samples [26,27,28].

With the development of few-shot learning, research on the synthesis and prediction of time-series data has gradually gained momentum. As a representative type of time-series data, speech data have attracted significant attention from researchers, particularly in the task of timbre conversion under small-sample conditions. This paper uses this task as the experimental focus to validate the effectiveness of the proposed method. The challenge of this task lies in the difficulty of effectively extracting key features of the target timbre when the amount of collected target speech data is limited, resulting in a significant discrepancy between the generated speech data and the target timbre. Currently, there are two main approaches to solving this problem. The first is a data augmentation-based method [29], which enhances the features of the target dataset samples or expands the dataset by leveraging auxiliary datasets or additional information, enabling the model to better extract relevant features. The second approach builds on existing models or knowledge and adapts them to new domains through various techniques. This can be categorized into two solutions: model fine-tuning [30] and transfer learning [31]. However, each of the aforementioned approaches has its respective limitations. For example, data augmentation methods are prone to introducing excessive noise and irrelevant features, which can compromise the effectiveness of subsequent feature conversion [32]. The other two approaches face challenges such as overfitting and high computational complexity, making it difficult to achieve fast and accurate feature conversion when the target data are scarce and exhibit limited characteristics.

To address the aforementioned issues, this chapter proposes a dual-layer transfer network model that combines the two core approaches discussed earlier. By constructing a two-tiered neural network consisting of coarse transfer and fine transfer layers, the model progressively achieves feature conversion of time-series data under limited target data conditions, thus overcoming the limitations of traditional time-series data augmentation methods in small-sample settings, such as difficulty in feature extraction and suboptimal augmentation effects [33,34,35]. The first layer, the coarse transfer network, performs weight pretraining, extracting low-level features (e.g., trends, periodicity) from the source data, allowing the model to capture stable foundational characteristics within the original data. This enables the generative model to learn fundamental time-series patterns, granting the model more generalized feature representations and effectively constraining the generation of irrelevant features during subsequent augmentation. Building on this initial transfer, the second layer, the fine transfer network, further extracts higher-level features, emphasizing complex dependencies and detailed patterns within the sequences, ultimately completing the training of the generative model. By separating the modeling of foundational and complex features, this dual-layer transfer architecture enhances the quality of feature migration, achieving time-series feature conversion under small sample conditions.

2.2. Coarse Transfer Network

To reduce the loss of target features during the first feature transfer and to enhance the transfer performance of the network, this paper adopts a convolutional neural network (CNN) [36,37], which is more effective in extracting features from two dimensions. The convolution operation can be expressed as

(1)$y_{i,j}=\sum _{m-1}^M\sum _{n-1}^N{W}_{m,n}·x_{i+m,j+n}+b$

In this context, $y_{i,j}$ represents the result of the convolution, $W_{m,n}$ denotes the weights of the convolutional kernel, $x_{i+m,j+n}$ refers to the input data and b indicates the bias term. At the same time, one of the important characteristics of time-series data is hierarchical structure and inherent order. An effective means of representing this structure is through the use of recurrent neural networks (RNNs); however, RNNs have high computational demands due to their difficulty in achieving parallelization [38]. Therefore, in configuring the coarse transfer network, we employ gated CNNs [39,40]. The gated convolutional network, through the introduction of a gating mechanism, controls the output of convolution operations, allowing for selective extraction among different features and enhancing the model’s flexibility and expressive capacity. This mechanism also enables dynamic adjustment of activation values based on the characteristics of the input data, facilitating the capture of complex temporal dependencies. Its nonlinear properties further enable the network to learn more intricate patterns and allow for the parallel processing of sequential data, demonstrating commendable performance in the field of speech modeling.

In gated CNNs, Gated Linear Units (GLUs) function as activation functions. The GLU is a data-driven activation function, where the output of the (l + 1)th layer is computed from the output $H_{l+1}$ of the lth layer and the model hyperparameters $W_l{V}_l{b}_l$ and $c_1$:

(2)$H_{l+1}=\sigma \left(H_l·V_l+c_l\right)\otimes \left(H_l·W_l+b_l\right)$

$\sigma$ represents the sigmoid function and ⊗ denotes the element-wise multiplication operation. This type of gating mechanism allows information to be selectively propagated based on the state of the previous layer.

In configuring the coarse transfer network, residual modules are employed [41]. The output of the residual block can be further expressed as a decomposition of the convolution operation:

(3)$y=x+Conv\left(x\right)=x+F\left(x,W_i\right)$

In this context, $Conv\left(x\right)$ represents the convolution operation, $F(x,W_i)$ is the output of the layer with weights, $x,{\omega }_i$ is the input, and y is the output. Through residual connections, this approach effectively mitigates the issues of vanishing or exploding gradients during training. Practical evidence has shown that residual modules can indeed enhance training efficiency.

Finally, in the upsampling stage, the Pixel Shuffler method [42], commonly used in super-resolution research, is employed. The operation of the Pixel Shuffler can be expressed as a transformation of the feature map’s shape:

(4)$X_{out}=PixelShuffle\left(X_{in},r\right)$

In this context, $X_{in}$ represents the input feature map and r denotes the desired upsampling factor. The Pixel Shuffler performs the following specific shape transformation on the feature map:

(5)$\left[H,W,C\right]\stackrel{\left(l-1\right)thLayerConvolution}{\to }\left[H,W,C·r^2\right]\stackrel{ShuffleTransformation}{\to }\left[H·r,W·r,C\right]$

Here, $H,W,C$ denote the dimensions of the original features.

The specific structure of the coarse transfer network is shown in Figure 1. Compared to the single-feature extraction and transfer processes utilized in other methods, the coarse transfer network introduced in this paper not only serves as a weight pretraining mechanism, thereby enhancing the efficiency of subsequent feature transfer, but also effectively alleviates the issues of slow training speed and convergence challenges encountered in other approaches. Additionally, the coarse transfer network imposes effective constraints on the feature space of the source data, focusing on extracting lower-level foundational features in time-series data. This approach helps the model avoid learning overly complex or irrelevant features and filters out information that does not align with the core features. Consequently, it reduces noise and unrelated feature interference during the subsequent generation process, significantly enhancing the consistency of the generated data with the target data in terms of key features.

2.3. Fine Transfer Network

Since its introduction by Goodfellow in 2014, Generative Adversarial Networks have gradually attracted the attention of researchers in areas such as image data augmentation and speech data processing. The application of GANs in the field of time-series data generation has also become increasingly widespread. The fine transfer network for this task adopts a generative adversarial network architecture, with the loss functions for the generator and discriminator described as follows:

(6)$L_G=-E_{z\sim p_z\left(z\right)}\left[log\left(D\left(G\left(z\right)\right)\right)\right]$

(7)$L_D=-E_{z\sim p_{data}\left(x\right)}\left[log\left(D\left(x\right)\right)\right]-E_{z\sim p_z\left(z\right)}\left[log\left(1-D\left(G\left(z\right)\right)\right)\right]$

The network incorporates gated CNNs and residual modules, ensuring its adaptability to the processing of time-series data. Additionally, to address the widely prevalent issue of vanishing gradients in GANs,

(8)${\nabla }_{{\theta }_G}{E}_{z\sim p_z\left(z\right)}\left[log\left(1-D\left(G\left(z\right)\right)\right)\right],$

and improve the issue of the generator being unable to learn from backpropagated gradients due to overtraining of the discriminator, a module transfer approach is adopted. In this method, certain weights of the feature extraction network within the discriminator are set to update solely by the training of the generator:

(9)${\theta }_D^k⟵{\theta }_D^k-\eta {\nabla }_{{\theta }_D^k}{L}_G$

${\theta }_D^k$ represents the parameters in the discriminator, $\eta$ denotes the learning rate, and ${\nabla }_{{\theta }_D^k}{L}_G$ represents the gradient of the generator’s loss function for parameter ${\theta }_D^k$. This ensures alignment in the training levels of both networks, thereby enhancing the model’s convergence capability and the effectiveness of feature transfer.

The fine transfer network, leveraging the zero-sum game mechanism of GANs, extracts and transfers key features of the target data through the establishment of a well-defined objective function. Building on the foundation of the coarse transfer, the fine transfer network enables the generated data to align more closely with the target data in terms of essential features, swiftly capturing characteristic patterns, complex dependencies, and nonlinear traits within the sequences. This process ultimately enhances both the efficiency and quality of time-series data generation, bringing the generated data into closer alignment with target features. The structure of the Fine Transfer Network is depicted in Figure 2.

2.4. Module Transfer

Since the introduction of Generative Adversarial Networks by Goodfellow in 2014, this technology has garnered widespread attention. The basic GAN architecture consists of a generator and a discriminator, with its core mechanism involving a zero-sum game where the generator and discriminator are trained alternately. This process ultimately achieves Nash equilibrium, enabling the generator to independently generate and fit the target data. However, in GANs, the generator and discriminator are actually two independent models, with information being transmitted between them solely through the forward generation of data and the backward propagation of weight gradients. To ensure the smoothness and rationality of gradient transmission, thereby promoting the effective convergence of adversarial training while maintaining the diversity of generated data, existing mainstream methods can be classified into two main categories. The first approach involves optimizing the objective function to measure the difference between the generated data distribution and the real data distribution, such as in WGAN, which introduces Wasserstein distance in place of JS divergence, significantly enhancing generation quality and model stability. The second approach focuses on improving the weight update process, such as adjusting the alternating training strategy (combining single discriminator training with multiple generator training) and employing spectral normalization (Lipschitz constraint) to align the training levels of the generator and discriminator. This paper proposes a module transfer method that combines the two aforementioned approaches to optimize the Generative Adversarial Network, enhancing the smoothness of gradient transmission and the convergence capability of model training.

As a zero-sum game architecture model designed to enable the generator and discriminator to jointly reach Nash equilibrium, the objective function for training GANs can be described as follows:

(10)$V\left(D,G\right)=E_{x\sim p_{data}}\left[logD\left(x\right)\right]+E_{z\sim p_z}\left[log\left(1-D\left(G\left(z\right)\right)\right)\right]$

where $p_{data}$ represents the distribution of real samples, $p_z$ denotes the distribution of generated samples, x is the real sample, z is the generated sample, and E represents the expectation. The overall process of GANs can be described as follows:

(11)$G^{*}=arg\underset{G}{min}\underset{D}{max}V\left(D,G\right)$

However, during the training process, it is difficult to achieve a balance in the training levels of the discriminator and the generator. The generator often faces issues where it cannot learn from the backpropagated gradients due to excessive training of the discriminator. This problem arises because the model objective of the discriminator can be simplified to

(12)$D_J=-E_{x\sim p_{data}}\left[logD\left(x\right)\right]-E_{x\sim p_z}\left[log\left(1-D\left(G\left(x\right)\right)\right)\right]$

The objective of the discriminator is to classify real samples as positive and the samples generated by the generator as negative. Therefore, when its training level is excessively high or significantly surpasses that of the generator, the measurements of real and generated samples become too precise, resulting in overly large gradients backpropagated to the generator. This situation hinders the generator’s ability to train and update weights smoothly, ultimately leading to model failure. To address this issue, the module transfer strategy proposed in this section involves constructing a specific feature extraction network structure that is common to both the generator and discriminator. This part of the structure is trained alongside the generator, and the learned weights are directly transferred to the corresponding structure in the discriminator, without training it alongside the discriminator. Since the training level of this network structure and its weights are updated only alongside the generator, it restricts the overtraining of the discriminator when incorporated as a foundational structure within it. This approach ensures that the generator and discriminator maintain a relatively balanced training level, addressing the convergence issues arising from the difficulty in managing their training levels.

Module transfer is a neural network structure designed according to the type of generated sample data. In this section, the input data consist of temporal speech signals, and thus the module transfer network is structured as a multi-layer convolutional network containing gated CNNs. This design aims to extract features from the speech signals and facilitate the transfer of these extracted features to other network layers, which can be described as follows:

(13)$T\left(x\right)=\Phi \left(Con{v}_d\left(W,C\left(x\right)+b\right)\right)$

In the equation, d represents the number of input channels, $Conv$ denotes the convolution function, and $C\left(x\right)$ signifies the multi-layer convolutional structure. After incorporating the module transfer structure, the model of the discriminator can be described as follows:

(14)$D_J=-E_{x\sim p_{data}}\left[logD\left(T\left(x\right)\right)\right]-E_{x\sim p_z}\left[log\left(1-D\left(G\left(T\left(x\right)\right)\right)\right)\right]$

It can be seen that the introduction of a module transfer structure in the discriminator, where the weights are updated solely with the generator’s training, allows both the discriminator and generator to be based on the same underlying features for modeling. This alignment of their training levels restricts the discriminator from overtraining independently and transmitting excessively large gradients back to the generator, thereby ensuring smoother training and enhancing the model’s convergence success rate. Additionally, since the module transfer strategy allows for selective adjustment of module types and quantities as needed, transferring only the essential modules instead of the entire network without altering the overall structure enables more flexible optimization of the generator and discriminator. This flexibility avoids the instability that can result from rigid intensity settings and reduces the risk of global overfitting associated with directly adjusting the loss function. Moreover, this approach lowers computational demands, making the network better suited for training under small sample conditions. The rough principle of the Module Transfer strategy is shown in Figure 3 below.

3. Time Series Data Augmentation Process

To address the challenge of time-series data generation under small-sample conditions, this paper establishes a dual-layer feature transfer model. Through the training of both coarse transfer and fine transfer generative models, progressive feature extraction of the target data is achieved, allowing the source domain data to gradually migrate and converge in the feature space closer to the target data, thereby reducing the presence of irrelevant features and enhancing augmentation effectiveness under small-sample conditions. Furthermore, a module transfer strategy is integrated into the fine transfer network, effectively mitigating common issues in GAN training such as gradient vanishing and mode collapse. This improvement accelerates model convergence and significantly enhances generation quality, enabling traditional GAN models to perform more effectively when faced with small-sample training scenarios.

The specific process for generating time-series data using this method is as follows:

The first step involves constructing the source domain dataset by either simulating the target scenario to collect a substantial amount of source domain data B or utilizing data augmentation techniques. During model training, this dataset is compared with the target data and serves as the medium for extracting target features.

The second step involves the coarse transfer of target features. The small sample data A containing target features and the source domain data B are input into the coarse transfer network for supervised learning. The initial feature extraction and transfer are achieved through gated convolutional layers and residual modules, resulting in the generation of intermediate data A₁.

The third step involves the fine transfer of target features based on the coarse transfer. The weights from the feature extraction layers of the coarse transfer network are used as the initial weights for the corresponding layers in the GAN network. The target data A and intermediate data A₁ serve as input data for the discriminator and generator of the fine transfer network, respectively, enabling further feature transfer and refined extraction of target data features using the GAN framework.

Finally, to simulate the target data generation, once the model converges and training is complete, the source domain data C is used as input. Feature transfer is performed through the generators in both the coarse and fine transfer networks, resulting in the generation of simulated signals that exhibit the target features. The specific steps of the time series data augmentation process are illustrated in Figure 4 below.

4. Simulation Results

4.1. Experimental Settings

The database used in this experiment is the CMU-ARCTIC database [43], which contains recordings from seven different speakers: slt (American female), clb (American female), bdl (American male), rms (American male), jmk (Canadian male), awb (Scottish male), and ksp (Indian male). Each speaker recorded 1132 segments of speech, with each segment lasting approximately 4 to 5 s. The content of the speech samples consists of complete sentences, and the audio files are stored in an uncompressed PCM format with a sampling rate of 22.05 kHz, 8-bit depth, mono, and Windows format. From this, 200 segments of source domain speech and target speech were selected to form a complete sample parallel training set (whole sample). Additionally, 20 pairs of speech were randomly selected from these 200 pairs to create a small sample training set (part sample). Furthermore, 20 pairs of speech were chosen from the database as a test set.

The simulation experiments were conducted on a local machine equipped with a 12-core, 16-thread 12th Gen Intel(R) Core(TM) i5-12500H CPU, 16 GB of RAM, and a GeForce RTX 3060 GPU. The main tools used for network construction were Pytorch 2.1.1 and Librosa 0.7.1. In this experiment, both the coarse transfer network and the fine transfer network utilized the Adam optimizer with a batch size of one. The learning rate for the coarse transfer network was set to 0.0002, and the loss function employed was the mean absolute error (MAE). For the fine transfer network, the learning rate for the generator was 0.0002, while the discriminator’s learning rate was set to 0.0001.

Three sets of experiments were designed to validate the proposed method, focusing on the following key objectives: (1) the effectiveness and reliability of the generated data under the context of small sample sizes; (2) the performance of the module transfer strategy; and (3) the effectiveness and superiority of the Dual-Layer Transfer Network model.

In the experiments, both the training and testing sets downsampled the speech data to 16 kHz, followed by the extraction of speech features using the WORLD analysis system [44]. During feature extraction, 24 Mel-cepstral coefficients (MCEPs) [45], as well as the fundamental frequency [46] and Aperiodicities(APs) [47], were extracted every 5 ms for each segment of speech. In this experiment, MCEPs were selected as the features that reflect most of the semantic information and sound details in the speech data, with the training focusing on relevant mappings around MCEPs. In the evaluation of experimental results, the quality of MCEPs was also considered a crucial assessment metric. The fundamental frequency primarily reflects the overall pitch range of the speech, and in this experiment, it underwent Gaussian log normalization. As for the APs, research indicates that mathematical transformations on APs do not significantly impact speech quality; therefore, these features were used directly in this experiment.

The evaluation metrics for this experiment primarily include Mel-cepstral distortion (MCD) and Modulation-spectra distance (MSD) [48,49]. MCD measures the distance between the Mel-cepstral coefficients of the target speech and the generated speech, describing the speech quality of the generated audio and its similarity to the target speech based on the overall structural differences in their Mel-cepstral. A smaller MCD indicates that the generated speech is closer in timbre to the target speech, resulting in higher quality. MSD assesses the local structural differences between the modulation spectra of the two Mel-cepstra, further evaluating the generated and target speech.

4.2. Model Accuracy Experiment

To verify the reliability of the proposed method in generating simulated temporal signals under small sample conditions, a Sequence-to-Sequence (S2S) parallel voice conversion (VC) strategy is used as a comparative baseline. This allows for a comparison of the performance differences between the proposed method and traditional parallel speech conversion methods when faced with small sample data. Additionally, the evaluation in this experiment includes comparisons with CycleGAN-VC3 and MelGAN-VC strategies to observe the conversion effects and convergence speed of the models when the input data consist of small sample parallel data. In Table 1, we present the average values of MCD and MSD for DLT-GAN, S2S-VC, CycleGAN-VC3, and MelGAN-VC when trained on the complete parallel dataset and the small sample parallel dataset, respectively, during evaluation.

The data indicate that S2S-VC, CycleGAN-VC3, and MelGAN-VC exhibit a significant decline in the quality of generated speech and similarity to the target speech when trained on the small sample dataset compared to their performance on the complete sample dataset. From the average values of MCD, the declines were 45.7%, 51.1%, and 50.4% for S2S-VC, CycleGAN-VC3, and MelGAN-VC, respectively. In terms of the average values of MSD, the declines were 28.7%, 31.7%, and 33.5%. Conversely, under the conditions of training on the small parallel dataset, DLT-GAN exhibited a relatively minor performance decline, with reductions of 37.5% in MCD and 9.67% in MSD. In horizontal comparisons, under small sample training conditions, the performance of DLT-GAN improved by 8.06%, 9.48%, and 4.98% compared to S2S-VC, CycleGAN-VC3, and MelGAN-VC, respectively. This demonstrates that the method proposed in this paper, DLT-GAN, achieves superior speech data enhancement effects under small sample conditions, exhibiting a higher similarity between generated speech features and target speech, as well as a stronger adaptability to small sample scenarios.

In addition to the MCD and MSD metrics, Figure 5 also presents the scattered plots of the MCEPs features of the generated speech from DLT-GAN, S2S-VC, CycleGAN-VC3, and MelGAN-VC under the small sample training set, facilitating a visual comparison with the target speech samples. As shown in the figure, the scattered plot of MCEPs for the speech samples generated by DLT-GAN demonstrates a higher degree of clustering and overlap within the target speech sample region. This indicates that the speech samples generated by DLT-GAN are closer to the target speech in terms of MCEP features.

In addition to the objective evaluation metrics mentioned above, this experiment also included a listening test to assess the quality of the converted speech. A total of 25 volunteers participated in a mean opinion score (MOS) test [50], where they rated the naturalness and similarity to the target speech of the audio samples generated by DLT-GAN, S2S-VC, CycleGAN-VC3, and MelGAN-VC. Among the 25 volunteers, there were 13 men and 12 women, all aged between 20 and 35 years. Their educational background was undergraduate level or higher, and all participants had good English reading and writing skills with no hearing impairments. Each participant listened to 10 converted speech samples from these four methods. In this experiment, each converted speech sample was played randomly to the listeners, who evaluated them using a five-point scale: one for “poor”, two for “fair”, three for “average”, four for “good”, and five for “excellent”. This allowed for a comparison of the differences in naturalness and similarity among the various speech samples. A higher MOS score indicates a superior subjective performance of the model. In Table 2 and Table 3, this paper provides the MOS values with a 95% confidence interval and conducts a variance analysis at a significance level of 0.05 on the data results of the two experimental groups. In the naturalness experiment, the F-statistic among the four groups of data is 4.83, while in the similarity experiment, the F-statistic is 4.67, both exceeding the critical value of three, indicating a significant difference in the means among the four groups in both experiments. By combining the specific data from the tables, we can conclude that DLT-GAN demonstrated a significant advantage over the other two models in terms of both the naturalness and similarity of the generated speech.

4.3. Effectiveness Experiment of Module Transfer

To test the feasibility of the module transfer strategy in the GAN network, we first conducted a comparative experiment on handwritten digit generation using the MNIST dataset [51]. In the experiment, we constructed two sets of GAN network models, with both the generator and discriminator following the classic DCGAN architecture. To achieve better generation results for comparison, we made adjustments to the network by replacing convolutional layers with a stride greater than one in the discriminator with pooling layers and by changing the Tanh activation function in the generator to a Sigmoid function. Network Two introduced the module migration strategy by setting the first layer convolutional network in both the generator and discriminator as the migration module. Some results generated by the two models are shown in Figure 6, where it is visually evident that Network Two produces superior generation results.

The loss decrease curves of the two models indicate that Network Two converges after approximately 2000 to 3000 iterations, while Network One requires at least 3000 to 4000 iterations to achieve a similar outcome. This suggests that the module migration strategy effectively accelerates the model’s convergence speed to some extent.

Subsequently, as part of the ablation study, we compared the performance differences between the DLT-GAN model with the module transfer strategy and the DLT-GAN model without it, both trained on the same small sample dataset. We evaluated the two models based on the MCD and MSD metrics, as shown in Table 4, and the results indicate that the module transfer strategy significantly enhances the model performance. In addition, we compared the average number of iterations required for both models to achieve optimal performance, which were approximately 200 and 250 iterations, respectively. This further validates the effectiveness of the module transfer strategy in enhancing the model’s convergence capability.

4.4. Ablation Study of the Dual-Layer Transfer Network

Finally, to validate the necessity of the Dual-Layer Transfer Network structure in the feature migration process of time-series data, we plotted the MCEPs feature scatter plots of the generated data from the source domain speech data after undergoing only the coarse and fine transfer networks and compared them with the generated data after both layers of migration. From Figure 7, it can be seen that a single-layer network migration can achieve a certain degree of timbre transformation, but with poor performance. In contrast, using a dual-layer network significantly improves the timbre conversion performance. In Table 5, we also observe that when measured by the MCD metric, the dual-layer migration improves timbre similarity by 49.09% and 32.28% compared to the coarse and fine migrations, respectively. From the perspective of the MSD metric, improvements of 66.13% and 48.92% are observed, respectively.

5. Conclusions

In summary, this paper addresses the performance bottleneck in simulating and generating time-series signals under small sample conditions by proposing a dual-layer transfer network structure based on module migration. The collaborative function of the coarse transfer network and the fine transfer network effectively constrains the source domain features, reduces the occurrence of irrelevant noise, and enhances the feature similarity of the generated data, significantly improving the effectiveness of time-series data augmentation in small-sample contexts. Additionally, to tackle the common training imbalance issue present in GAN models, the module migration strategy was introduced, which improves the model’s convergence rate and training efficiency while effectively mitigating issues of gradient vanishing and mode collapse during network training. Experimental results represented by voice signals demonstrated that the proposed method significantly outperforms existing approaches under small-sample conditions, achieving superior and stable outcomes in time-series data generation. This method shows promising prospects in various applications, including voice data augmentation and speaker identity verification. Future research could focus on exploring the application potential of this method in other types of time-series data augmentation tasks, further optimizing the data augmentation process, or applying the module migration strategy in a wider variety of generative adversarial networks.

Author Contributions

Conceptualization, Z.C., Y.P. and S.J.; methodology, Z.C. and Y.P.; validation, Z.C., Y.P. and S.J.; formal analysis, J.Q. and S.L.; investigation, H.Y.; resources, H.Y.; data curation, J.Q. and S.L.; writing—original draft preparation, Z.C.; project administration, Z.C. and Y.P.; funding acquisition, Y.P. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Schematic diagram of the coarse transfer network structure.

Figure 2. Schematic diagram of the fine transfer network structure.

Figure 3. Schematic diagram of module transfer principle.

Figure 4. Illustration of DLT-GAN time-series data augmentation process under few-shot conditions.

Figure 5. Scatter plots of MCEP features generated by the four methods.

Figure 6. Comparison of Ablation Experiment Results for the Module Migration Strategy in Handwritten Digit Experiment.

Figure 7. The MCEP feature scatter plot of single-layer transfer network generation results.

Table 1

The specific evaluation results of the four methods.

Dataset	Method	MCD	MSD
Whole	DLT-GAN	5.13	1.68
	S2S-VC	4.36	1.31
	CycleGAN-VC3	4.92	1.44
	MelGAN-VC	4.19	1.27
Part	DLT-GAN	8.21	1.86
	S2S-VC	8.93	1.92
	CycleGAN-VC3	9.07	2.02
	MelGAN-VC	8.64	1.91

Table 2

MOS for naturalness with 95% confidence intervals.

Dataset	Method	MOS
Part	DLT-GAN	3.9 ± 0.43
	S2S-VC	3.6 ± 0.31
	CycleGAN-VC3	3.2 ± 0.42
	MelGAN-VC3	3.5 ± 0.30

Table 3

MOS for similarity with 95% confidence intervals.

Dataset	Method	MOS
Part	DLT-GAN	3.7 ± 0.39
	S2S-VC	3.3 ± 0.31
	CycleGAN-VC3	3.1 ± 0.41
	MelGAN-VC3	3.4 ± 0.28

Table 4

Ablation study results of the module transfer strategy.

Dataset	Method	MCD	MSD
Part	DLT-GAN	8.21	1.86
Part	DLT-GAN/MT	9.24	2.39

Table 5

Evaluation results of single-layer transfer and dual-layer transfer networks.

Dataset	Method	MCD	MSD
Part	DLT-GAN	8.21	1.86
Part	Coarse Transfer	12.24	3.09
Part	Fine Transfer	10.86	2.77

References

1. Meddaoui, A.; Hain, M.; Hachmoud, A. The benefits of predictive maintenance in manufacturing excellence: A case study to establish reliable methods for predicting failures. Int. J. Adv. Manuf. Technol.; 2023; 128, pp. 3685-3690. [DOI: https://dx.doi.org/10.1007/s00170-023-12086-6]

2. Johnson, J.M.; Khoshgoftaar, T.M. Survey on deep learning with class imbalance. J. Big Data; 2019; 6, 27. [DOI: https://dx.doi.org/10.1186/s40537-019-0192-5]

3. Gao, F.; Hu, M.; Zhong, M.E.; Feng, S.; Tian, X.; Meng, X.; Huang, Z.; Lv, M.; Song, T.; Zhang, X. et al. Segmentation only uses sparse annotations: Unified weakly and semi-supervised learning in medical images. Med. Image Anal.; 2022; 80, 102515. [DOI: https://dx.doi.org/10.1016/j.media.2022.102515] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35780593]

4. Byambadorj, Z.; Nishimura, R.; Ayush, A.; Ohta, K.; Kitaoka, N. Text-to-speech system for low-resource language using cross-lingual transfer learning and data augmentation. EURASIP J. Audio, Speech Music. Process.; 2021; 2021, 42. [DOI: https://dx.doi.org/10.1186/s13636-021-00225-4]

5. Chen, J.; Teo, T.H.; Kok, C.L.; Koh, Y.Y. A Novel Single-Word Speech Recognition on Embedded Systems Using a Convolution Neuron Network with Improved Out-of-Distribution Detection. Electronics; 2024; 13, 530. [DOI: https://dx.doi.org/10.3390/electronics13030530]

6. Yuan, R.; Wang, B.; Sun, Y.; Song, X.; Watada, J. Conditional style-based generative adversarial networks for renewable scenario generation. IEEE Trans. Power Syst.; 2022; 38, pp. 1281-1296. [DOI: https://dx.doi.org/10.1109/TPWRS.2022.3170992]

7. Song, X.; Xiong, J.; Wang, M.; Mei, Q.; Lin, X. Combined Data Augmentation on EANN to Identify Indoor Anomalous Sound Event. Appl. Sci.; 2024; 14, 1327. [DOI: https://dx.doi.org/10.3390/app14041327]

8. Chen, S.; Kalanat, N.; Xie, Y.; Li, S.; Zwart, J.A.; Sadler, J.M.; Appling, A.P.; Oliver, S.K.; Read, J.S.; Jia, X. Physics-guided machine learning from simulated data with different physical parameters. Knowl. Inf. Syst.; 2023; 65, pp. 3223-3250. [DOI: https://dx.doi.org/10.1007/s10115-023-01864-z]

9. Yang, S.; Guo, S.; Zhao, J.; Shen, F. Investigating the effectiveness of data augmentation from similarity and diversity: An empirical study. Pattern Recognit.; 2024; 148, 110204. [DOI: https://dx.doi.org/10.1016/j.patcog.2023.110204]

10. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. Adv. Neural Inf. Process. Syst.; 2014; 27.

11. Gui, J.; Sun, Z.; Wen, Y.; Tao, D.; Ye, J. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Trans. Knowl. Data Eng.; 2021; 35, pp. 3313-3332. [DOI: https://dx.doi.org/10.1109/TKDE.2021.3130191]

12. Yu, L.; Zhang, W.; Wang, J.; Yu, Y. Seqgan: Sequence generative adversarial nets with policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence; San Francisco, CA, USA, 4–9 February 2017; Volume 31.

13. Wiese, M.; Knobloch, R.; Korn, R.; Kretschmer, P. Quant GANs: Deep generation of financial time series. Quant. Financ.; 2020; 20, pp. 1419-1440. [DOI: https://dx.doi.org/10.1080/14697688.2020.1730426]

14. Mogren, O. C-RNN-GAN: Continuous recurrent neural networks with adversarial training. arXiv; 2016; arXiv: 1611.09904

15. Esteban, C.; Hyland, S.L.; Rätsch, G. Real-valued (medical) time series generation with recurrent conditional gans. arXiv; 2017; arXiv: 1706.02633

16. Yoon, J.; Jarrett, D.; Van der Schaar, M. Time-series generative adversarial networks. Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2019; Volume 32.

17. Hazra, D.; Byun, Y.C. SynSigGAN: Generative adversarial networks for synthetic biomedical signal generation. Biology; 2020; 9, 441. [DOI: https://dx.doi.org/10.3390/biology9120441]

18. Huang, F.; Deng, Y. TCGAN: Convolutional Generative Adversarial Network for time series classification and clustering. Neural Netw.; 2023; 165, pp. 868-883. [DOI: https://dx.doi.org/10.1016/j.neunet.2023.06.033]

19. Li, X.; Deng, S.; Wang, S.; Lv, Z.; Wu, L. Review of small data learning methods. Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC); Tokyo, Japan, 23–27 July 2018; Volume 2, pp. 106-109.

20. Huang, K.; Luo, N. Improved TimeGAN Based on Attention for Time Series Prediction Method with Few Shot. J. East China Univ. Sci. Technol.; 2023; 49, pp. 890-899.

21. Ke, D.; Yao, W.; Hu, R.; Huang, L.; Luo, Q.; Shu, W. StyleFormerGAN-VC: Improving Effect of few shot Cross-Lingual Voice Conversion Using VAE-StarGAN and Attention-AdaIN. Proceedings of the 2022 IEEE/ACIS 23rd International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD); Taichung, Taiwan, 7–9 December 2022; pp. 235-239.

22. Lan, J.; Zhou, Y.; Guo, Q.; Sun, H. Data Augmentation for Data-driven Methods in Power System Operation: A Novel Framework using Improved GAN and Transfer Learning. IEEE Trans. Power Syst.; 2024; 39, pp. 6399-6411. [DOI: https://dx.doi.org/10.1109/TPWRS.2024.3364166]

23. Sajeeda, A.; Hossain, B.M. Exploring generative adversarial networks and adversarial training. Int. J. Cogn. Comput. Eng.; 2022; 3, pp. 78-89. [DOI: https://dx.doi.org/10.1016/j.ijcce.2022.03.002]

24. Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning; Sydney, Australia, 6–11 August 2017; pp. 214-223.

25. Arora, J.; Tushir, M.; Kherwa, P.; Rathee, S. Generative Adversarial Networks: A Comprehensive Review. Data Wrangling: Concepts, Applications and Tools; Wiley: Hoboken, NJ, USA, 2023; pp. 213-234.

26. Wang, Y.; Yao, Q.; Kwok, J.T.; Ni, L.M. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv. (CSUR); 2020; 53, pp. 1-34. [DOI: https://dx.doi.org/10.1145/3386252]

27. Conneau, A.; Ma, M.; Khanuja, S.; Zhang, Y.; Axelrod, V.; Dalmia, S.; Riesa, J.; Rivera, C.; Bapna, A. Fleurs: Few-shot learning evaluation of universal representations of speech. Proceedings of the 2022 IEEE Spoken Language Technology Workshop (SLT); Doha, Qatar, 9–12 January 2023; pp. 798-805.

28. Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P.H.; Hospedales, T.M. Learning to compare: Relation network for few-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Salt Lake City, UT, USA, 18–23 June 2018; pp. 1199-1208.

29. Kumar, P.; Toshniwal, D. Self-Supervison with data-augmentation improves few-shot learning. Appl. Intell.; 2024; 54, pp. 2976-2997. [DOI: https://dx.doi.org/10.1007/s10489-024-05340-1]

30. Liu, H.; Tam, D.; Muqeeth, M.; Mohta, J.; Huang, T.; Bansal, M.; Raffel, C.A. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2022; Volume 35, pp. 1950-1965.

31. Sun, Q.; Liu, Y.; Chua, T.S.; Schiele, B. Meta-transfer learning for few-shot learning. Proceedings of the IEEE/CVF Conference On Computer Vision and Pattern Recognition; Long Beach, CA, USA, 15–20 June 2019; pp. 403-412.

32. Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data; 2019; 6, 60. [DOI: https://dx.doi.org/10.1186/s40537-019-0197-0]

33. Wen, Q.; Sun, L.; Yang, F.; Song, X.; Gao, J.; Wang, X.; Xu, H. Time series data augmentation for deep learning: A survey. arXiv; 2020; arXiv: 2002.12478

34. Annaki, I.; Rahmoune, M.; Bourhaleb, M. Overview of Data Augmentation Techniques in Time Series Analysis. Int. J. Adv. Comput. Sci. Appl.; 2024; 15, pp. 1201-1211. [DOI: https://dx.doi.org/10.14569/IJACSA.2024.01501118]

35. Semenoglou, A.A.; Spiliotis, E.; Assimakopoulos, V. Data augmentation for univariate time series forecasting with neural networks. Pattern Recognit.; 2023; 134, 109132. [DOI: https://dx.doi.org/10.1016/j.patcog.2022.109132]

36. Li, Z.; Liu, F.; Yang, W.; Peng, S.; Zhou, J. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Trans. Neural Netw. Learn. Syst.; 2021; 33, pp. 6999-7019. [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3084827]

37. Rakhmatulin, I.; Dao, M.S.; Nassibi, A.; Mandic, D. Exploring convolutional neural network architectures for EEG feature extraction. Sensors; 2024; 24, 877. [DOI: https://dx.doi.org/10.3390/s24030877]

38. Mienye, I.D.; Swart, T.G.; Obaido, G. Recurrent neural networks: A comprehensive review of architectures, variants, and applications. Information; 2024; 15, 517. [DOI: https://dx.doi.org/10.3390/info15090517]

39. Dauphin, Y.N.; Fan, A.; Auli, M.; Grangier, D. Language modeling with gated convolutional networks. Proceedings of the International Conference on Machine Learning; Sydney, Australia, 6–11 August 2017; pp. 933-941.

40. Yang, C.; An, Z.; Zhu, H.; Hu, X.; Zhang, K.; Xu, K.; Li, C.; Xu, Y. Gated convolutional networks with hybrid connectivity for image classification. Proceedings of the AAAI Conference on Artificial Intelligence; New York, NY, USA, 7–12 February 2020; Volume 34, pp. 12581-12588.

41. He, K.; Zhang, X.; Ren, S.; Sun, J. Identity mappings in deep residual networks. Proceedings of the Computer Vision—ECCV 2016: 14th European Conference; Amsterdam, The Netherlands, 11–14 October 2016; Part IV 14 pp. 630-645.

42. Huang, C.K.; Nien, H.H. Multi chaotic systems based pixel shuffle for image encryption. Opt. Commun.; 2009; 282, pp. 2123-2127. [DOI: https://dx.doi.org/10.1016/j.optcom.2009.02.044]

43. Kominek, J.; Black, A.W. The CMU Arctic speech databases. Proceedings of the Fifth ISCA Workshop on Speech Synthesis; Pittsburgh, PA, USA, 14–16 June 2004.

44. Morise, M.; Yokomori, F.; Ozawa, K. World: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Trans. Inf. Syst.; 2016; 99, pp. 1877-1884. [DOI: https://dx.doi.org/10.1587/transinf.2015EDP7457]

45. Fukada, T.; Tokuda, K.; Kobayashi, T.; Imai, S. An adaptive algorithm for mel-cepstral analysis of speech. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-1992); San Francisco, CA, USA, 23–26 March 1992; Volume 92, pp. 137-140.

46. Brown, C.A.; Bacon, S.P. Fundamental frequency and speech intelligibility in background noise. Hear. Res.; 2010; 266, pp. 52-59. [DOI: https://dx.doi.org/10.1016/j.heares.2009.08.011]

47. Deshmukh, O.; Wilson, C.E. A measure of aperiodicity and periodicity in speech. Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’03); Hong Kong, China, 6–10 April 2003; Volume 1, I.

48. Kubichek, R. Mel-cepstral distance measure for objective speech quality assessment. Proceedings of the IEEE Pacific Rim Conference on Communications Computers and Signal Processing; Victoria, BC, Canada, 19–21 May 1993; Volume 1, pp. 125-128.

49. Zhang, Y.; Zhao, Y. Real and imaginary modulation spectral subtraction for speech enhancement. Speech Commun.; 2013; 55, pp. 509-522. [DOI: https://dx.doi.org/10.1016/j.specom.2012.09.005]

50. Streijl, R.C.; Winkler, S.; Hands, D.S. Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives. Multimed. Syst.; 2016; 22, pp. 213-227. [DOI: https://dx.doi.org/10.1007/s00530-014-0446-1]

51. Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag.; 2012; 29, pp. 141-142. [DOI: https://dx.doi.org/10.1109/MSP.2012.2211477]