Improving Text Classification in Agricultural Expert Systems with a Bidirectional Encoder Recurrent Convolutional Neural Network

Abstract

With the rapid development of internet and AI technologies, Agricultural Expert Systems (AESs) have become crucial for delivering technical support and decision-making in agricultural management. However, traditional natural language processing methods often struggle with specialized terminology and context, and they lack the adaptability to handle complex text classifications. The diversity and evolving nature of agricultural texts make deep semantic understanding and integration of contextual knowledge especially challenging. To tackle these challenges, this paper introduces a Bidirectional Encoder Recurrent Convolutional Neural Network (AES-BERCNN) tailored for short-text classification in agricultural expert systems. We designed an Agricultural Text Encoder (ATE) with a six-layer transformer architecture to capture both preceding and following word information. A recursive convolutional neural network based on Gated Recurrent Units (GRUs) was also developed to merge contextual information and learn complex semantic features, which are then combined with the ATE output and refined through max-pooling to form the final feature representation. The AES-BERCNN model was tested on a self-constructed agricultural dataset, achieving an accuracy of 99.63% in text classification. Its generalization ability was further verified on the Tsinghua News dataset. Compared to other models such as TextCNN, DPCNN, BiLSTM, and BERT-based models, the AES-BERCNN shows clear advantages in agricultural text classification. This work provides precise and timely technical support for intelligent agricultural expert systems.

1. Introduction

The rapid advancements in internet, big data, and artificial intelligence technologies have been transforming how agricultural knowledge is acquired through digital platforms. Despite the proliferation of web-based agricultural knowledge systems and the vast amount of data available, significant challenges remain. These include poor integration of agricultural resources, ineffective delivery of services, delayed access to experts, and mismatched resource allocation between science and practice. Consequently, farmers continue to face difficulties in obtaining timely and valuable agricultural knowledge through online platforms [1].

Integrating artificial intelligence into agriculture offers a promising solution to enhance the scope, quality, and efficiency of agricultural information services. Agricultural Expert Systems (AESs) are playing an increasingly vital role in modern agricultural production by combining data analysis, machine learning, and agricultural expertise to provide technical support for management decisions.

Agricultural text constitutes the core metadata of AESs. For AESs to effectively analyze and utilize this data, it needs to precisely interpret and understand the semantic content of the texts. This is accomplished through Natural Language Processing (NLP) technologies, which leverage domain-specific agricultural knowledge to support decision-making in production management. Accurate representation and classification of agricultural texts are essential for intelligent question-answering systems [2], knowledge retrieval [3], and decision-making [4] in AESs. However, traditional methods of handling agricultural text, such as manual feature extraction and shallow learning models, have proven insufficient due to the complexity, high dimensionality, and domain-specific nature of the data. Enhancing the precision of text classification and extracting deeper semantic elements remains crucial.

The representation and classification of agricultural texts pose key challenges for AESs. This study focuses on effectively representing and classifying short agricultural texts. Common approaches include traditional word vector models and pre-trained word vector models [5]. A traditional model such as Word2Vec [6] uses a sliding window to predict target words based on local context, but they often fail to capture long-range dependencies and the polysemy of words. Pre-trained models, such as BERT [7], have overcome these limitations by generating context-sensitive word vectors through self-supervised learning. BERT’s transformer architecture allows it to consider all words in a sentence simultaneously, capturing both semantic and logical relationships more effectively. This study adopts BERT to represent short agricultural texts and improve classification performance.

Agricultural text classification methods can be broadly categorized into traditional machine learning and deep learning approaches. While traditional methods, such as Support Vector Machines (SVM) [8], naive Bayes [9], and K-means [10], offer stable performance, they rely heavily on manual feature engineering, which is time-consuming and labor-intensive. In contrast, deep learning models, such as Convolutional Neural Networks (CNNs) [11], Recurrent Neural Networks (RNNs) [12], and their variants such as Long Short-Term Memory (LSTM) [13] and Gated Recurrent Units (GRUs) [14], automatically extract features from data, reducing the need for manual intervention. However, given the short, domain-specific nature of agricultural texts, relying on a single deep learning model can limit classification accuracy due to challenges such as sparse data and complex semantics.

To overcome these challenges, this paper introduces a fused neural network model that leverages the strengths of various neural architectures, aiming to more effectively capture semantic features and contextual information in agricultural texts. This approach seeks to improve classification accuracy and stability by resolving issues such as word ambiguity and data sparsity. The primary contributions of this study are as follows:

• (1). We propose a fusion model, AES-BERCNN, which integrates bidirectional encoding with recurrent convolutional neural networks to classify six types of agricultural text.

• (2). An agricultural text encoder is developed to effectively capture and represent complex semantic information, addressing the issue of word polysemy.

• (3). A comprehensive agricultural question dataset, covering six categories, is constructed to serve as a multifunctional sample set for both training and testing.

• (4). Extensive experiments show that the AES-BERCNN outperforms comparative models in terms of accuracy, precision, recall, and F1 scores on the agricultural question dataset.

The remainder of this paper is organized as follows. Section 2 analyzes the related work, Section 3 introduces the materials and methods, Section 4 illustrates the experimental results and analysis, Section 5 offers the discussion, and finally, Section 6 concludes the paper and provides future prospects.

2. Related Work

2.1. Agricultural Text Representation Methods

Text representation is fundamental to agricultural expert systems, as it directly impacts their overall effectiveness. Text vectorization, which transforms text into vectors that capture its semantics, is a key technique in this process. Common models used in agricultural text representation include TF-IDF, Word2Vec, and BERT.

Term Frequency-Inverse Document Frequency (TF-IDF) [15] is a widely used feature extraction algorithm. It evaluates the importance of a word within a document set or specific document in a corpus. In agricultural applications, Wei et al. [16] employed selected feature words and TF-IDF weighting to build a text representation model that highlights the importance of key terms. Du et al. [17] combined shallow syntactic analysis with TF-IDF and C-value for concept extraction, while Duan et al. [18] optimized TF-IDF for feature weighting. Zheng et al. [19] incorporated the Chi-square (CHI) method into TF-IDF to address the impact of feature words on class distributions. Xu et al. [20] applied TF-IDF and word clouds to extract news-related keywords, ultimately generating keyword vectors.

Word2Vec offers efficient implementations of the Continuous Bag-of-Words (CBOW) and Skip-Gram architectures to compute word vector representations [21,22]. Researchers have widely used Word2Vec to vectorize agricultural texts, leveraging their unique characteristics. For example, Xu et al. [23] used Word2Vec to convert words into vectors with syntactic and semantic information. Chen et al. [24] employed the skip-gram model in Word2Vec to obtain vector representations that capture contextual meanings. Wang [25] combined Word2Vec with an agricultural word segmentation dictionary and TF-IDF to address high-dimensional and sparse data challenges in rice-related texts. Xiao et al. [26] proposed a classification model integrating ERNIE + DPCNN and Word2Vec + TextCNN for classifying Chinese agricultural texts.

BERT [7] (Bidirectional Encoder Representations from Transformers) is a state-of-the-art pre-training model used to obtain rich text feature representations. BERT overcomes limitations of Word2Vec, such as its inability to handle multiple word meanings. In agricultural text processing, BERT has been applied effectively. For instance, Yang et al. [27] used BERT to fine-tune a question classification model, improving text classification performance. Yuan et al. [28] developed a rice phenomics relationship extraction model based on BERT, demonstrating its capability in handling complex agricultural data. Wang [29] employed a twelve-layer Chinese BERT model to vectorize agricultural texts, effectively addressing high dimensionality and sparsity. Cao et al. [30] proposed a sentiment analysis algorithm using an improved BERT model with a symmetrical structure to obtain sentence-level vectors. Chen et al. [31] introduced a joint entity recognition and relation extraction model named RoBERT-Agr, based on RoBERTa, WWM, and CRF algorithms.

In summary, TF-IDF highlights important words based on term frequency and document frequency, but it does not account for word context or semantics, treating terms independently. Word2Vec captures context through CBOW or skip-gram, generating low-dimensional, efficient word vectors. However, its static vectors do not adjust based on word context. BERT provides deep contextual embeddings by considering both left and right contexts of words, but its training and fine-tuning require significant computational resources. To address this, lightweight BERT models have emerged, reducing model complexity and pre-trained parameters while improving text vectorization efficiency. BERT effectively resolves issues of sparsity and word polysemy in agricultural texts and provides a strong foundation for text representation in agricultural expert systems. Lightweight BERT models meet real-time processing requirements, offering an efficient solution for text processing in these systems.

2.2. Agricultural Text Classification Approaches

Text classification involves two main steps: text feature extraction and the training of classification models. In traditional machine learning-based methods, these are two separate processes. Manual feature extraction is often complex and results in lower accuracy. However, through optimization and improvement, significant advancements have been made in agricultural text classification using traditional approaches.

Shi et al. [32] applied SVM to classify agricultural data, while Duan et al. [33] achieved a classification accuracy rate of 92.5% using SVM on automatically crawled agricultural web data. Cui et al. [34] integrated the Spark computing framework with a dis-tributed gradient boosting algorithm, enabling real-time and accurate forestry text classification. Du et al. [35] employed the naive Bayes algorithm to classify agricultural literature, achieving an average accuracy of 94%.

More recently, researchers have shifted from traditional machine learning to artificial neural networks for text classification. Various deep learning models have been applied, including CNN, RNN, and hybrid models that combine different neural architectures to improve classification performance. For instance, Zhao et al. [36] constructed a dataset of 48,000 dietary texts and developed a classification model using Word2Vec and LSTM networks, effectively categorizing dietary text. Zhao et al. [37] used Word2Vec and a bidirectional GRU model to classify tomato pests and disease-related queries. Zhang et al. [38] proposed a CNN-based model to analyze sentiment polarity in agricultural Q&A texts, while Liang et al. [39] developed a sentence similarity model using Word2Vec and LSTM to assist farmers with rice production challenges. Zhou et al. [40] utilized a multi-core CNN to extract multi-scale local information in agricultural texts.

Due to the limitations of using a single model for feature extraction, many researchers are exploring hybrid approaches that combine multiple models. Jin et al. [41] developed an agricultural question classification model by integrating BiGRU with a multi-scale CNN. Feng et al. [42] improved ResNet’s residual module and employed a Capsule Net-work (CapsNet) for rice knowledge classification. Wang et al. [43] introduced an Attention-ResLSTM-seq2seq network to classify rice-related questions, while Li et al. [44] combined the BERT pre-training model with a stacked LSTM network, achieving the best performance across nine categories of agricultural short texts. Wang et al. [45] utilized AL-BERT with match-LSTM to classify agricultural short texts, demonstrating that the model could accurately classify questions.

In conclusion, while traditional machine learning-based text classification techniques have achieved notable success, much of their performance still depends on manual feature engineering. These methods are limited in capturing deep semantic features and fail to account for logical relationships or contextual data. Deep learning methods, which can automatically extract local, contextual, and essential features, have proven more effective in text classification. However, relying on a single model has inherent drawbacks, particularly with datasets characterized by short text length and imbalanced sample distribution, such as in agricultural texts. A single model often struggles to uncover hidden semantic information or fully understand word associations.

However, several key challenges remain in agricultural text classification, including the absence of a comprehensive text corpus, which limits future research efforts. Additionally, agricultural texts tend to be short, feature diverse synonym expressions, and pose difficulties in feature extraction, all while suffering from imbalanced classification. These issues further complicate efforts to improve classification accuracy.

To overcome these challenges, this study constructs an agricultural question dataset by scraping question–answer pairs from multiple agricultural information service platforms, providing a solid foundation for future research. Furthermore, we propose a hybrid model, AES-BERCNN, which integrates an agricultural text encoder with a RCNN to enhance the classification accuracy of short agricultural texts.

3. Materials and Methods

3.1. Data Sources

3.1.1. Corpus Construction and Preprocessing

To address the scarcity of public agricultural corpora for text classification, this paper presents the construction of an agricultural Q&A corpus. The data sources for the corpus are the China Agricultural Technology Extension Information Platform (CAEIP) and the China Agricultural Information Network (CAIN). Using web crawling technology, a total of 257 MB of Q&A data was automatically collected from their agricultural Q&A community domains over 37 months, between March 2020 and March 2023. The corpus comprises six categories of agricultural Q&A pairs. The collected data underwent preprocessing, including data cleaning, tagging, and labeling. The flow of data construction and preprocessing is illustrated in Figure 1. As a result, an agricultural question dataset containing 100,284 question texts was constructed.

Agricultural Q&A text often contains noisy data, such as special webpage tags, carriage returns, and special characters, which can negatively impact model feature extraction. To ensure high-quality data for the corpus, these irrelevant characters must be removed before building the dataset. Data cleaning involves essential preprocessing tasks, including denoising, deduplication, and handling missing information. After cleaning, the data is labeled with “[CLS]” and “[SEP]” to mark the beginning and end of questions and answers. Finally, the texts are categorized into six distinct classes: breeding, planting, processing, fishery, edible fungi, and technology consulting. The distribution of data and key theme words across these categories are shown in Table 1.

The data were sampled as follows: for each category, 80% of the questions (80,482 questions) were allocated to the training set, 10% (9901 questions) to the test set, and the remaining 10% (9901 questions) to the validation set.

3.1.2. Dataset Analysis

The agricultural question dataset in this study exhibits significant differences in domain specificity and specialization compared to general multi-category datasets. One key characteristic is its extreme specialization. Since all questions pertain exclusively to the agricultural domain, with often ambiguous boundaries between categories, general domain models struggle to accurately categorize these questions. Another challenge is the uneven distribution of question categories. As shown in Figure 1, the “edible fungi” and “processing” categories account for less than 3% of the dataset each, whereas the “planting” category dominates with 65.3%. Additionally, the dataset features short text lengths, with most questions averaging around 10 characters and rarely exceeding 30. This brevity makes it difficult to extract meaningful semantic information, complicating the model training process.

3.2. Model of AES-BERCNN

3.2.1. The Architecture of AES-BERCNN

The AES-BERCNN is an advanced RCNN-based model that combines the robust semantic encoding capabilities of transformers with the context-capturing strengths of bidirectional gated recurrent unit (GRU) networks. This integration improves both the accuracy and generalization of agricultural text classification. The structure of the AES-BERCNN is depicted in Figure 2.

The model architecture consists of several layers: an input layer, a preprocessing and tokenization layer, an embedding layer, an agricultural text representation layer, an agricultural text classifier layer, a fully connected layer, and an output layer.

At the core of the AES-BERCNN are the agricultural text representation and classifier layers. The Agricultural Text Encoder (ATE) manages text representation, while the classifier is responsible for semantic feature extraction and classification. The AES-BERCNN network processes each word as a basic unit, feeding the output sequences from a 6-layer transformer encoder into the classifier layer. By combining the contextual understanding abilities of the transformer with the sequence modeling strengths of the RCNN, the model achieves multi-level semantic feature learning for textual information.

The model’s input is an agricultural question. After preprocessing, n initial expression vectors, $W_i\left(0\le i\le n\right)$, are generated as inputs. Following the ATE layer, these vectors, $W_i$, transform into various word representation vectors, $T_i$. Finally, the ATBSEC layer processes these vectors to yield the final classification result.

3.2.2. Layers of AES-BERCNN

• (1). Input Layer

The AES-BERCNN model begins by disambiguating the agricultural query, breaking the sentence into discrete word tokens. Special tokens [CLS] and [SEP] are added at the beginning and end to mark sentence boundaries. This segmentation is performed using BertTokenizer, which tokenizes the text while preserving the sentence structure. To meet the model’s input requirements, the sentence length is capped at 512 tokens.

• (2). Embedding Layer

Each word token is transformed into a 768-dimensional vector composed of token embeddings, segment embeddings, and position embeddings. Token embeddings capture the semantic characteristics of each word, segment embeddings distinguish sentence segments, and position embeddings indicate the word’s position in the sequence. These three embeddings are summed to generate a comprehensive initial feature representation of the text. This embedding process, performed by the BERT model, produces 768-dimensional vectors that combine both semantic and structural information for further processing.

• (3). Agricultural Text Representation Layer

In this layer, based on the transformer architecture, the embeddings are processed through six transformer encoding layers. Each layer consists of a self-attention mechanism and a feed-forward neural network. The self-attention mechanism computes attention weights using dot products of query, key, and value vectors, which are then normalized using the Softmax function. These weights are applied to the value vectors to create new word representations. The resulting outputs are processed via a feed-forward neural network with a ReLU activation, improving the contextual representation through successive encoding layers.

• (4). Agricultural Text Classifier Layer

The feature vectors from the agricultural text representation layer are passed to the classifier layer, which performs bidirectional semantic extraction, multi-feature concatenation, and max pooling. The bidirectional process captures relationships between words in both forward (left-to-right) and backward (right-to-left) directions, generating comprehensive feature vectors. These vectors are then concatenated with features from the previous layer, enriching the feature representation. Max pooling is applied to reduce dimensionality by selecting the maximum value for each feature across time steps, preserving the most important information.

• (5). Fully Connected Layer

The fully connected (FC) layer processes the feature vectors from the classifier layer, mapping high-dimensional features to the intended classifier dimensions through a linear transformation. This step further optimizes the feature vectors, preparing them for the final classification stage.

• (6). Output Layer

The output layer converts the processed feature vectors into a final classification result. Using the softmax function, it generates a probability distribution, allowing for the prediction to be applied directly in real-world scenarios.

3.2.3. Loss Function and Optimizer

During the training process, the cross-entropy loss function is employed to measure the difference between the predicted categories and the actual categories. This function calculates the negative log-likelihood of the model’s predictions, providing a guide for optimizing the model’s parameters. By evaluating the divergence between predicted probabilities and actual labels, it serves as a key metric for assessing model performance. The AES-BERCNN uses the cross-entropy loss function to quantify the gap between its outputs and the true labels, as described in (1).

(1)$CroessEntropy\left(y,\widehat{y}\right)=-{{\displaystyle \sum }}_{i=1}^N{y}_i\times \log \left({\widehat{y}}_i\right)$

where N is the number of categories in the dataset, and $y_i$ means the code of the true label category. ${\widehat{y}}_i$ is the distribution probability of the label category.

Parameters are refined using the Adam optimizer, which combines momentum and adaptive learning rate strategies to effectively handle large-scale datasets and sparse gradient challenges. The Adam optimizer dynamically adjusts the learning rate for each parameter, facilitating faster convergence while reducing the likelihood of getting trapped in local optima. The details of the Adam algorithm are described as follows:

(2)$m_t={\beta }_1·m_{t-1}+\left(1-{\beta }_1\right)·g_t$

(3)$n_t={\beta }_2·n_{t-1}+\left(1-{\beta }_2\right)·g_t^2$

(4)${\theta }_t={\theta }_{t-1}-{\displaystyle \frac{\alpha }{\sqrt{{\widehat{n}}_t+ϵ}}}{\widehat{m}}_t$

(5)${\widehat{m}}_t={\displaystyle \frac{m_t}{1-{\beta }_1^t}}, {\widehat{n}}_t={\displaystyle \frac{n_t}{1-{\beta }_2^t}}$

where $m_t$ represents the momentum estimate at step t (the first moment estimate), ${\beta }_1$ is the exponential decay rate for the momentum estimates, $m_{t-1}$ shows the momentum estimate from the previous step t, $g_t$ is the gradient at the current step $n_t$ is the second moment estimate at step t (the exponentially weighted moving average of the squared gradients), ${\beta }_2$ shows the exponential decay rate for the second moment estimates, $n_{t-1}$ shows the second moment estimate from the previous step t, $g_t^2$ represents the square of the gradient at the current step, ${\theta }_t$ and ${\theta }_{t-1}$ are the parameter value at step t and previous step t, $\alpha$ is the learning rate, ${\widehat{n}}_t$ shows the bias-corrected second moment estimate, ${\widehat{m}}_t$ is the bias-corrected momentum estimate, and $ϵ$ is a small constant added for numerical stability.

By utilizing the cross-entropy loss function in conjunction with the Adam optimizer, the model incrementally fine-tunes its parameters to boost classification accuracy and stability. These refinements take place during the training phase, ensuring that the model consistently improves its classification capabilities over time.

3.3. The Agricultral Texts Representation

To accurately represent the semantic information of word-level sequences in question text, this paper proposes an Agricultural Text Encoder (ATE) algorithm. The ATE generates hidden vectors for each individual word in agricultural texts. First, we define the input sentence of the model as $e=\left(e_1,e_2,e_3,\dots ,e_n\right)$, where $e_i$ represents the i-th character of the input sentence, and n denotes the length of the sentence.

In the embedding layer, the input sentence $e=\left(e_1,e_2,e_3,\dots ,e_n\right)$ is transformed into an input sequence, $=\left(E_1,E_2,E_3,\dots ,E_n\right)$, by summing the token embeddings, segment embeddings, and position embeddings. This input sequence, $E=\left(E_1,E_2,E_3,\dots ,E_n\right)$, is then fed into the encoder layer to extract features, resulting in a rich output sequence, $T=\left(t_1,t_2,t_3,\dots ,t_n\right)$, which serves as the sentence encoding for subsequent entity relationship joint extraction. ATE consists of multiple encoders, as illustrated in Figure 3.

The encoder is composed of two main components: multi-head attention and a feed-forward neural network. Each of these components is further enhanced with residual connections and normalization. The use of multi-head attention allows for the extraction of information from various perspectives, capturing multiple semantic meanings and thereby enhancing the comprehensiveness of information extraction.

In the self-attention mechanism, we initially compute three vectors: Query (Q), Key (K), and Value (V). The calculation for attention is defined in Equations (6)–(9).

(6)$Attention\left(Q,K,V\right)=Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$

(7)$Q=Linear\left(E\right)=E{W}_Q$

(8)$K=Linear\left(E\right)=E{W}_K$

(9)$V=Linear\left(E\right)=E{W}_V$

where $d_k$ represents the input dimension of the Q and K of each word, Softmax() is the normalized exponential function, and $W_Q$, $W_K$, and $W_V$ are the weight matrices.

To enhance the semantic representation of sentences and improve the model’s expressive capability, the multi-head attention mechanism processes multiple sets of self-attention on the original input sequence. The results from each set of self-attention are then concatenated and subjected to a linear transformation to produce the final output. The calculation formula is provided in Equations (10) and (11).

(10)$MultiHead\left(Q,K,V\right)=Concat\left(hea{d}_1,hea{d}_2,\dots ,hea{d}_6\right)W^0$

(11)$hea{d}_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$

where $W_i^Q$, $W_i^K$, and $W_i^V$ represent the weight matrix of $hea{d}_i$, $W^0$ is the additional weight matrix, and Concat() is the splice function.

Finally, a new word representation vector T is obtained through residual linking and normalization. The procedure for calculating the ATE is detailed in Algorithm 1.

Algorithm 1: Agricultural texts encoder algorithm
1:	input: $e=\left(e_1,e_2,e_3,\dots ,e_n\right)$
2:	for ($i=1;i\le n;i++$)                 // Embedding Layer
3:	$E_i$ $\leftarrow$ Token_$\mathrm{Embedding}(e_i$) + Segment_$\mathrm{Embedding}(e_i$) + Position_$\mathrm{Embedding}(e_i$)
4:	end for
5:	for ($j=1;j\le 6;j++$)             //Transformer encoder layers
	// Define Query (Q), Key (K), Value (V) for each head
6:	Q $\leftarrow$ $\mathrm{Linear}\left(\mathrm{E}\right)\leftarrow$ $\mathrm{E}*W_Q$
7:	K $\leftarrow$ $\mathrm{Linear}\left(\mathrm{E}\right)\leftarrow$ $\mathrm{E}*W_K$
8:	V $\leftarrow$ $\mathrm{Linear}\left(\mathrm{E}\right)\leftarrow$ $\mathrm{E}*W_V$
	// Calculate Scaled Dot-Product Attention
9:	$Attention\left(Q,K,V\right)$ $\leftarrow$ $Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$
	// Calculate Multi-Head Attention
10:	for ($i=1;i\le 6;i++$)
11:	$hea{d}_i \leftarrow Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$
12:	end for
13:	$MultiHead\left(Q,K,V\right) \leftarrow$ Concat $(hea{d}_1$$,hea{d}_2$$,\dots ,hea{d}_h$) * W0
14:	TH $\leftarrow$ Normalize (Multi-head_Output + E)     // Multi-head Output TH
	// Feed-Forward
15:	TF = linear (TH)
16:	TF = ReLU (TF)
17:	T $\leftarrow$ Normalize (TF + TH)           // Feed-Forward Output TF
18:	end for
19:	output: $T=\left(t_1,t_2,t_3,\dots ,t_n\right)$

3.4. Agricultural Text Classifier Layer

The structure of the agricultural text classifier is illustrated in Figure 4. The core component of the classifier is the bidirectional semantic extraction, which extracts hidden semantic information from agricultural texts and overcomes gradient vanishing and gradient explosion during hidden layer updates and iterations.

As seen from Figure 4, the core of bidirectional semantic extraction is the GRU, which incorporates a gating mechanism; one GRU unit includes the update gate and reset gate. The relevant formulas are shown in (12)–(15).

(12)$z_t=\sigma \left(W_z·\left[h_{t-1},x_t\right]\right)$

(13)$r_t=\sigma \left(W_r·\left[h_{t-1},x_t\right]\right)$

(14)$\widetilde{h_t}=tanh\left(W·\left[r_t\times h_{t-1},x_t\right]\right)$

(15)$h_t=\left(1-z_t\right)\times h_{t-1}+z_t\times \widetilde{h_t}$

where $z_t$ represents the update gate, which is used to control the amount of information saved from the previous time node to the current time node. A larger value indicates that the information from the hidden state of the previous node has a greater impact on the current node. In Equation (12), the update gate receives $h_{t-1}$ and $x_t$ for computation and is processed using an activation function. The larger the activation result, the more information is saved. $r_t$ represents the reset gate, which controls the amount of information from the previous node to be written into the current candidate set $\widetilde{{\mathit{h}}_{\mathit{t}}}$. If the value is zero, the information from the previous node state can be filtered out. In Equation (13), the reset gate operates on $h_{t-1}$ and $x_t$, similar to the update gate, and the larger the activation result, the more information is written.

Next, in Equation (14), the GRU combines the $r_t$ output from the previous reset gate with the input state $h_{t-1}$ from the previous moment and the input $x_t$ from the current moment, performing an operation to form the new memory content, $\widetilde{h_t}$, of the current node. In Equation (15), the GRU also combines the input state $h_{t-1}$ from the previous moment and the new memory content $\widetilde{h_t}$ to determine the current state, $h_t$, of the node. $\sigma$ is the Sigmoid function, $tanh$ is the hyperbolic tangent function, W is the weight, $x_t$ represents the input data at moment t, $h_{t-1}$ represents the input data from the previous moment, and $h_t$ is the output of the current moment.

For the unidirectional GRU neural network, its state is output from the beginning to the end. This structure has a disadvantage in processing sequential information because sometimes the sequential information that needs to be processed is influenced by both previous and subsequent input information. To address this issue, a bi-directional gated recurrent network is used. The bi-directional GRU neural network leverages the advantages of a bi-directional RNN and LSTM while making further modifications.

To fully utilize the contextual information of the text, we capture text information from the sequence forward and reverse directions. Two separate hidden layers are combined into a bi-directional layer, and the results of the sequences in both directions are processed via the activation function to merge and obtain the final results; the calculation is defined as (16)–(18).

(16)$\overrightarrow{h_t}=GRU\left(\overrightarrow{h_{t-1}},x_t\right)$

(17)$\overleftarrow{h_t}=GRU\left(\overleftarrow{h_{t+1}},x_t\right)$

(18)$H_t=\left[\overrightarrow{h_t};\overleftarrow{h_t}\right]$

where $\overrightarrow{h_t}$ and $\overleftarrow{h_t}$ denote the left-to-right and right-to-left hidden states, respectively. GRU denotes the GRU unit, and $x_t$ denotes the t-th element in the input sequence. “;” denotes the vector splicing operation. $H_t$ is the output text feature vector. To prevent overfitting of the model, the dropout technique is applied after the Bi-GRU layer. This technique randomly removes a certain percentage of Bi-GRU units from the neural network during model training to achieve regularization. The bidirectional semantic extraction algorithm is presented in Algorithm 2.

Algorithm 2: The bidirectional semantic extraction algorithm
1:	input: $\mathrm{X}=(x_1$$,x_2$$,x_3$$,\dots ,x_t$)
	// Define GRU Cell Operations
2:	function GRU$\text{\_}\mathrm{Cell}(h_{t-1}$, xt):
3:	$z_t\leftarrow \sigma \left(W_z·\left[h_{t-1},x_t\right]\right)$
4:	$r_t\leftarrow \sigma \left(W_r·\left[h_{t-1},x_t\right]\right)$
5:	$\widetilde{{\mathit{h}}_{\mathit{t}}}\leftarrow tanh\left(W·\left[r_t\times h_{t-1},x_t\right]\right)$
6:	${\mathit{h}}_{\mathit{t}}\leftarrow \left(1-z_t\right)\times {\mathit{h}}_{\mathit{t}-\mathbf{1}}+z_t\times \widetilde{{\mathit{h}}_{\mathit{t}}}$
7:	return ${\mathit{h}}_{\mathit{t}}$
8:	end function
	// Implement bidirectional semantic extraction
	// Initialize hidden
9:	h_forward [0] $\leftarrow$ 0
10:	$\mathrm{h}\text{\_}\mathrm{backward}[\mathrm{T}+1]\leftarrow$ 0
11:	for ($t=1;i\le T;t++$)               // Forward semantic extraction
12:	$\mathrm{h}\text{\_}\mathrm{forward}\left[\mathrm{t}\right]\leftarrow$ GRU_Cell(h_forward[t − 1], xt)
13:	end for
14:	for ($t=T;i\ge 0;t--$)               // Backward semantic extraction
15:	$\mathrm{h}\text{\_}\mathrm{backward}\left[\mathrm{t}\right]\leftarrow$ GRU_Cell(h_backward[t + 1] xt)
16	end for
17:	for ($t=1;i\le T;t++$)          // Combine forward and backward results
18:	${\mathrm{H}}_{\mathrm{t}}\leftarrow$ [h_forward[t], h_backward[t]]
19:	end for
20:	output:$\mathrm{H}\leftarrow$ (H1, H2, …, HT)

4. Results

We used Python 3.8 and CUDA 11.7 within PyCharm 2021 to develop our model, running on a 64-bit Windows 10 OS. The hardware configuration included an Intel Xeon(R) W-2255 CPU and an NVIDIA RTX A4000 GDDR6 16 GB GPU. The experimental datasets consisted of 100,284 agricultural questions collected for this study, in addition to the publicly available Tsinghua News dataset. Both datasets were split into training, validation, and testing sets in an 8:1:1 ratio.

4.1. Parameter Settings

In this study, based on the pre-trained BERT-based Chinese model from Hugging Face, which employs a 12-layer transformer architecture with a hidden layer dimension of 768, we utilized a 6-layer transformer encoder to extract the semantic information of agricultural short text. To enhance the classification accuracy of the AES-BERCNN, we evaluated various learning rates (LR) and dropout rates to identify the optimal parameters for the model. The main parameter settings for the AES-BERCNN are given in Table 2.

4.2. Evaluation Metrics

The model’s evaluation metrics include accuracy, precision, recall, and F1 score. The percentage of accurately predicted positive samples among all predicted positives is known as precision. The percentage of accurately anticipated positive samples among all actual positives in the dataset is measured by recall. The F1 score provides a fair evaluation of the model’s performance by combining precision and recall into a single statistic. The percentage of accurately predicted samples relative to the total number of samples is known as accuracy. The formulas for calculations are as follows (19)–(22):

(19)$Precision={\displaystyle \frac{TP}{TP+FP}}$

(20)$Recall={\displaystyle \frac{TP}{TP+FN}}$

(21)$F1={\displaystyle \frac{2\times precision\times recall}{precision+recall}}$

(22)$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FN+FP}}$

For multi-category classification tasks, it is crucial to evaluate model performance using metrics such as macro average, micro average, and weighted average. Macro averaging evaluates the model’s performance by calculating the arithmetic mean of precision, recall, and F1 score for each category, providing a measure of performance across all categories. Micro averaging aggregates the results from all categories to compute overall metrics. Weighted averaging adjusts for sample imbalances across categories by multiplying each category’s metrics by its sample proportion relative to the total sample size, then averaging the results. Given the imbalanced category distribution in the agricultural question dataset used in this study, weighted averages are employed for evaluating multiple comparative models. The formulas for weighted average precision (Precision_w), weighted average recall (Recall_w), and weighted average F1 score (F1_w) are as (23)–(26).

(23)$Precisio{n}_w={{\displaystyle \sum }}_i{w}_i{P}_i$

(24)$Recal{l}_w={{\displaystyle \sum }}_i{w}_i{R}_i$

(25)$F{1}_w={\displaystyle \frac{2\times Precisio{n}_w\times Recal{l}_w}{Precisio{n}_w+Recal{l}_w}}$

(26)$w_i={\displaystyle \frac{S_i}{S}}$

where S_i represents the number of i-th category, S denotes the total number of samples, and P_i and R_i are the precision and recall for the i-th category, respectively.

4.3. Hyperparameters Selection of the AES-BERCNN

To help the AES-BERCNN achieve better diagnostic accuracy, we compared different learning rates (LR) and dropouts to determine the appropriate parameters for the model.

• (1). Learning rate comparisons

Table 3 presents the results for weighted average precision (Precision_w), weighted average recall (Recall_w), weighted average F1 score (F1_w), and accuracy metrics under different learning rates. For ease of comparison, the best results are highlighted in bold.

As shown in Table 3, choosing the right learning rate greatly improves the model’s performance. The AES-BERCNN achieved its best accuracy and F1 score with a learning rate of 7 × 10⁻⁵.

• (2). Dropout comparisons

Table 4 shows the evaluation metrics for the AES-BERCNN with different dropout rates. Dropout decreases the number of intermediate features and enhances feature orthogonality, improving the model’s generalization and reducing overfitting. As depicted in Table 4, the model’s performance improved with higher dropout rates. Notably, setting the dropout rate to 0.2 optimized both the F1 score and accuracy.

4.4. Ablation Experiment

To evaluate the contribution of each component in our agricultural text classifier, we conducted the ablation experiment using the agricultural question dataset. The specific settings of the experiment are as follows: (1) BERT model, (2) ATE model, (3) ATE-BiGRU, and (4) ATE-BiGRU-CNN. The results of the ablation experiment on the agricultural question test sets are presented in Table 5.

Table 5 demonstrates that the AES-BERCNN achieved the best classification results, with an accuracy of 99.63% and a loss of 0.014. From the perspective of pre-training, the ATE not only requires less time compared to BERT but also achieves an F1 score of 99.20. Additionally, when considering hybrid models, ATE-BiGRU outperforms the ATE in classification performance. Therefore, the ATE-based models, particularly the ATE-BiGRU-CNN (AES-BERCNN), significantly enhance classification performance for agricultural short texts, achieving higher accuracy and F1 scores with lower loss and reasonable training times.

4.5. Comparative Experiments

To highlight the effectiveness of the proposed AES-BERCNN model, we compared it with eight widely used classification models from recent years. These models are categorized into three groups: single models, fusion models with BERT pre-training, and fusion models with an ATE.

• (1). Single models: TextCNN [46] utilizes convolutional neural networks to capture local features in the text. This model is known for its simplicity and effectiveness in capturing local dependencies in text. DPCNN [47] uses a series of convolutional and pooling layers to progressively reduce the dimensionality of the input while preserving critical information, making it effective for capturing long-range dependencies in text. BiLSTM [48] captures contextual information from both past and future states, which helps in understanding the context more comprehensively. This makes it particularly useful for tasks such as text classification, where understanding the context is crucial. BiGRU [49] processes text data in both forward and backward directions, capturing context from both past and future states. BiGRU offers a balance between performance and computational efficiency.

• (2). Fusion models with BERT pre-training: BERT-TextCNN [50] leverages BERT’s contextual understanding and TextCNN’s ability to detect local patterns, resulting in a robust text classification model. The BERT-DPCNN [51] combines the BERT’s contextual embeddings and the DPCNN’s hierarchical feature extraction, making this model highly effective for complex text classification tasks. BERT-BiLSTM [52] leverages BERT’s deep contextual understanding and BiLSTM’s ability to capture bidirectional dependencies, making it highly effective for text classification. BERT-BiGRU [53] combines BERT’s powerful embeddings and BiGRU’s efficient bidirectional processing, making this model well-suited for text classification tasks.

• (3). Fusion models with an ATE: this combines the proposed ATE with TextCNN, DPCNN, BiLSTM, and BiGRU.

Table 6 details the parameters of the comparative models, and the comparative experimental results on the test dataset are presented in Table 7.

Table 7 clearly demonstrates that the AES-BERCNN model outperforms other models, achieving a classification accuracy of 99.63% and a loss of 0.014 on the agricultural question test dataset. From the perspective of a single model, compared to TextCNN, BiLSTM, and BiGRU, DPCNN achieves the best F1 score of 97.84% and the lowest loss of 0.076. This superior performance is attributed to its ability to progressively extract high-level features from the text, thereby better understanding the semantic information. However, it is worth noting that an epoch of this model takes comparatively more time. Analyzing from the fusion model perspective, models incorporating BERT pre-training achieve the best classification results across all metrics compared to the single models. This demonstrates that BERT excels in understanding sentence semantics and capturing complex contextual dependencies, thereby significantly enhancing classification performance. However, in terms of time efficiency, the models BERT-TextCNN, BERT-DPCNN, BERT-BiLSTM, and BERT-BiGRU each take approximately 6 min per epoch, which poses a challenge to the timelines required in AESs. When examining the proposed ATE method, except for ATE-BiLSTM, the models ATE-TextCNN, ATE-DPCNN, and ATE-BiGRU exhibit higher accuracy, precision, recall, and F1 scores, along with lower loss values, compared to the BERT-based hybrid models. Additionally, the ATE series models require almost half the epoch time compared to the BERT series. The AES-BERCNN model proposed in this paper integrates the context embeddings generated by the ATE with the features extracted by the RCNN. This fusion allows the model to leverage rich feature representations from different levels, thereby enhancing overall classification performance. This effectively addresses the problem of insufficient features in short texts and the difficulty in learning semantic information.

To evaluate the convergence efficiency of the AES-BERCNN model against other models, we examined the changes in training loss values and accuracy over multiple training rounds. Figure 5 illustrates the comparison of accuracy and loss over 10 epochs on the agricultural questions training set.

As depicted in Figure 5a,b, the AES-BERCNN model exhibited a faster convergence rate compared to the other models, achieving the same classification accuracy more quickly. Additionally, the AES-BERCNN attained higher classification accuracy at the same convergence rate. These results clearly highlight the superiority of the AES-BERCNN in terms of convergence efficiency for classifying question text. Furthermore, as shown in Figure 5c, the AES-BERCNN requires less time than the BERT hybrid model, while still delivering improved classification performance.

4.6. Classification Experiment

To further investigate the differences and impacts between the models, we conducted experiments to analyze the precision, recall, and F1 scores of each model across various categories. The experimental outcomes are presented in Figure 6.

Figure 6 illustrates that all models exhibit high evaluation metrics in the categories of planting, breeding, fishery, and technology consulting. However, their performance declines in the processing and edible fungi categories, likely due to the larger data availability in the former categories and the smaller datasets in the latter. Notably, each model achieves a higher F1 score in the technology consulting category, particularly when using BERT and an ATE, as these models benefit from a richer vocabulary and stronger semantic support, leading to superior overall categorization results. Overall, the AES-BERCNN demonstrates optimal performance in the planting, breeding, fishery, and technology consulting categories. It also shows notable effectiveness and robustness in the processing and edible fungi categories with smaller sample sizes, underscoring its capability in agricultural short text classification.

To delve deeper into the classification performance of the models across different categories, a confusion matrix was created for each model’s test set results. Illustrated in Figure 7, the confusion matrix outlines the classification outcomes, where rows correspond to the predicted categories and columns to the true categories. The diagonal elements indicate the count of correctly classified question texts, whereas the off-diagonal elements denote the misclassified texts. The confusion matrix reveals that the AES-BERCNN surpasses other models in accurately identifying a variety of question classifications.

Figure 7 reveals that within the six categories, the classification results for edible fungi and technology consulting are relatively concentrated, exhibiting fewer misclassifications. In contrast, the categories of planting, breeding, processing, and fishery are more susceptible to cross-misclassification. Upon examining the question texts, it was observed that the boundaries between the planting, breeding, processing, and fishery categories are sometimes blurred in certain questions. For instance, the question “lettuce fresh-cut processing and management technology” involves both “management” and “fresh-cut processing”, which pertain to planting and processing, respectively. This overlap affects the classification accuracy to some extent.

4.7. Generalization Experiment

To assess the generalizability of the models presented in this paper, they were evaluated using the Tsinghua dataset. This dataset comprises 10 categories, finance, realty, stocks, education, science, society, politics, sports, games, and entertainment, with each category containing 20,000 texts, resulting in a balanced dataset. The maximum text length in the dataset is 38, with an average length of 18.8. For this experiment, the dataset, consisting of 200,000 texts, was divided into training (180,000 texts), validation (10,000 texts), and test (10,000 texts) sets.

To evaluate the convergence efficiency of the AES-BERCNN model relative to other models, we examined the changes in training loss values and accuracy over multiple training iterations. Figure 8 illustrates a comparison of accuracy and loss over 10 epochs using the Tsinghua training dataset.

Figure 8 illustrates that the AES-BERCNN model consistently surpasses other models on the Tsinghua news training dataset, attaining higher accuracy and lower loss values. However, overall, after 10 iterations on the Tsinghua data training set, the accuracy of the AES-BERCNN model exceeds 99%. In contrast, the accuracy on the validation and test sets is only around 91%, suggesting that the model may be experiencing overfitting. The overfitting could be attributed to the model’s complexity and the large number of parameters, which enable it to fit the noise and details in the training data well but hinder its ability to generalize to new data. Additionally, there might be distributional differences between the training and validation datasets, contributing to the observed performance discrepancy. This issue will be addressed in the next phase of the research.

A detailed comparison of key metrics on the Tsinghua news test dataset, including loss, weighted precision, recall, and F1 score, is provided in Table 8.

Table 8 demonstrates that while TextCNN, DPCNN, BiLSTM, and BiGRU are faster to train, they exhibit comparatively lower performance metrics. On the other hand, BERT-based models achieve higher accuracy, precision, recall, and F1 scores but demand significantly more training time compared to non-BERT models. ATE-based models, although showing slightly lower accuracy, precision, recall, and F1 scores than BERT-based models, require nearly half the training time, making them more efficient in terms of computational resources and time. In summary, the AES-BERCNN achieves the lowest loss at 0.28 and an accuracy of 91.14%. However, a comparison with Table 7 reveals that the model’s accuracy drops from 99.63% on the agricultural question dataset to 91.14% on the Tsinghua dataset. While the AES-BERCNN model performs well on the agricultural question dataset, its generalization ability is lacking when applied to the Tsinghua dataset, which encompasses a broader range of topics, greater categorical diversity, and more complex data. This indicates that the model requires further optimization and adaptation to enhance its adaptability and generalization performance across different types of datasets.

5. Discussion

This paper introduces an AES-BERCNN, a bidirectional encoder recurrent convolutional neural network specifically designed for the extraction and categorization of agricultural texts. Given the scarcity of publicly available datasets for such specialized applications, we meticulously developed an extensive agricultural question text dataset that encompasses multiple agricultural domains to facilitate comprehensive experimental validation. The AES-BERCNN effectively addresses challenges related to feature scarcity and the complexity of learning semantic information by enhancing feature extraction and improving classification accuracy for shorter agricultural texts.

Traditional NLP-based systems encounter significant hurdles due to the diversity, specialization, and dynamic nature of agricultural texts. By leveraging the strengths of both the Agricultural Text Encoder (ATE) and Recurrent Convolutional Neural Network (RCNN) architectures, the AES-BERCNN model tackles these issues through its bidirectional agricultural text representation and semantic extraction techniques. This dual approach ensures that the model comprehends complex jargon and contexts while continuously adapting to the evolving nature of agricultural information.

The AES-BERCNN model has demonstrated significant advancements in agricultural text classification; however, it is not without limitations. As demonstrated by the generalization experiments in Section 4.7, the AES-BERCNN’s ability to generalize is limited when handling datasets that encompass a broader range of topics, greater categorical diversity, and increased complexity. Specific reasons for this limitation may include:

• (1). Differences in Dataset Features: The agricultural question dataset likely contains more homogeneous topics and relatively simple linguistic patterns, whereas the Tsinghua dataset covers a wider array of topics and more complex linguistic structures. This discrepancy in features causes the model to perform less effectively on the Tsinghua dataset compared to the agricultural interrogative dataset.

• (2). Insufficient Dataset Diversity: The model heavily relies on the agricultural question dataset during training and may not have adequately learned the features and patterns necessary to handle more diverse data. Consequently, the model’s performance deteriorates when faced with the Tsinghua dataset.

• (3). Overfitting: The model’s excellent performance on the agricultural question dataset may be attributed to overfitting to the specific features of that dataset. When applied to a more complex dataset such as the Tsinghua dataset, the model struggles to adapt to new features and patterns, resulting in a decrease in accuracy.

• (4). Insufficient Domain Adaptation: The agricultural question dataset and the Tsinghua dataset may belong to different domains. The features learned by the model trained on the agricultural interrogative dataset are not fully applicable to the Tsinghua dataset, leading to the model’s limited generalization ability.

Another notable limitation is the absence of additional contextual data integration, such as market trends or environmental information, which could potentially enhance the model’s predictive capabilities. Our understanding of the model’s versatility and adaptability is currently constrained due to the lack of exploration in other domains with similarly complex textual material.

6. Conclusions

To address the limitations of existing single-depth models, this paper proposes a fused text classification model, AES-BERCNN, and validates it using a self-constructed agricultural question dataset. The experimental results demonstrate an accuracy rate of 99.63%, indicating high classification performance. The model effectively addresses issues such as the short length of agricultural texts, sparse features, and uneven distribution of classification samples. Additionally, the model’s generalization performance is validated using the Tsinghua dataset.

Future projects will plan to expand the model’s capabilities to accommodate a broader spectrum of agricultural texts and scenarios, ensuring its versatility across various agricultural contexts and languages. In addition, to address the limited generalization ability of the model on the Tsinghua dataset, we will implement improvements in the following areas:

• (1). Increase Data Diversity: Introduce more datasets from various fields and topics, allowing the model to access a broader range of language patterns and features. This will enhance its performance across different scenarios.

• (2). Optimize Model Structure: Further refine and improve the model architecture to boost feature extraction and semantic understanding capabilities, making the model more accurate and efficient in processing complex texts.

• (3). Prevent Overfitting: Employ regularization techniques and methods such as cross-validation to prevent overfitting, thereby improving the model’s generalization ability and ensuring robust performance on unseen data.

Through these enhancements, we expect to significantly improve the model’s generalization ability, enabling it to perform well in a wider range of practical applications.

Author Contributions

Conceptualization, Y.L. and L.L.; methodology, X.G. and J.W.; software, X.G. and J.Z.; validation, X.G., J.W. and G.G.; formal analysis, X.G.; investigation, G.G.; resources, J.Z. and L.L.; data curation, X.G.; writing—original draft preparation, X.G.; writing—review and editing, X.G., J.W. and Y.L.; visualization, X.G.; supervision, G.G.; project administration, X.G.; funding acquisition, X.G. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data used to support this study are available on GitHub; interested readers can search for files in https://github.com/guoxiaojuanhist/Agriculture-QuestionDataset (accessed on 29 April 2024) and download them. Any downloading, accessing, or use of this dataset for commercial or non-academic purposes is prohibited.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

AES	Agricultrural Expert System.
ATE	Agricultural Complex Text Encoder.
BERT	Bidirectional Encoder Representations from Transformer.
BiGRU	Bidirectional Gated Recurrent Unit.
BiLSTM	Bidirectional Long Short-Term Memory.
BiLSTM-Att	Bidirectional LSTM with Attention.
BiGRU-Att	Bidirectional GRU with Attention.
CNN	Convolutional Neural Networks.
DCNN	Dynamic Convolutional Neural Network.
DPCNN	Deep Pyramid Convolutional Neural Network.
FN	False Negative.
FP	False Positive.
GCN	Graph Convolutional Network.
GRU	Gated Recurrent Unit.
LSTM	Long Short-Term Memory.
NLP	Natural Language Processing.
Q&A	Question and Answer.
RCNN	Recurrent Convolutional Neural Network.
RNN	Recurrent Neural Network.
SVM	Support Vector Machine.
TextCNN	Text Convolutional Neural Network.
TF-IDF	Term Frequency-Inverse Document Frequency.
TN	True Negative.
TP	True Positive.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Preprocessing and construction dataset.

Figure 2. Structure of AES-BERCNN.

Figure 3. Structure of ATE.

Figure 4. Structure of the agricultural text classifier.

Figure 5. The accuracy, loss, and time of comparative models on agricultural question training set: (a) accuracy comparison, (b) loss comparison, and (c) time comparison.

Figure 6. Comparative model experimental results of precision, recall, and F1 on agricultural question test dataset: (a) precision comparison, (b) recall comparison, and (c) F1 comparison.

Figure 7. Confusion matrix of classification effect of each model: (a) BiLSTM, (b) BiGRU, (c) TextCNN, (d) DPCNN, (e) BERT-TextCNN, (f) BERT-DPCNN, (g) BERT-BiLSTM, (h) BERT-BiGRU, (i) ATE-DPCNN, (j) ATE-TextCNN, (k) ATE-BiLSTM, (l) ATE-BiGRU, and (m) ATE-BERCNN.

Figure 7. Confusion matrix of classification effect of each model: (a) BiLSTM, (b) BiGRU, (c) TextCNN, (d) DPCNN, (e) BERT-TextCNN, (f) BERT-DPCNN, (g) BERT-BiLSTM, (h) BERT-BiGRU, (i) ATE-DPCNN, (j) ATE-TextCNN, (k) ATE-BiLSTM, (l) ATE-BiGRU, and (m) ATE-BERCNN.

Figure 8. The accuracy and loss of comparative models on Tsinghua training set: (a) accuracy comparison and (b) loss comparison.

Table 1

Sample questions for 6 categories.

Category	Quantity	Theme
Breeding	16,818	Feeding, breeding, prevention and treatment techniques, treatment methods
Planting	65,564	Cultivation techniques, field management, high yield, diseases and pests
Processing	2676	Processing, production, storage and craftsmanship
Fishery	8059	Fish, breeding, ponds and aquatic products
Edible Fungi	2001	Shiitake, edible mushroom, flat mushroom and cultivation
Technology Consulting	5166	Science and technology, agriculture, development and technology

Table 2

AES-BERCNN parameter settings.

Parameters	Value	Parameters	Value
Batch_Size	128	Max_Position_Embedding	512
Maximum Sequence Length	32	Attention_Heads_Num	6
Hidden_Size	768	Hidden_Layers_Num	6
Epoch	10	Pooler_FC_Size	768
Learning_Rate	7 × 10−5	Pooler_Attention_Heads_Num	6
Dropout	0.2	Pooler_FC_Layers_Num	3
Filter_Sizes	(2, 3, 4)	Pooler_Perhead_Num	128
Num_Filters	256	Vocab_Size	18,000
Optimizer	Adam	Loss	CrossEntropy
RNN_Hidden	256	Activation	Softmax

Table 3

Results of AES-BERCNN with different LR.

LR	Loss	Accuracy(%)	Precisionw(%)	Recallw(%)	F1w(%)	Time of One Epoch
1 × 10−5	0.049	98.62	98.64	98.62	98.62	0:03:03
2 × 10−5	0.027	99.21	99.22	99.21	99.21	0:03:02
3 × 10−5	0.02	99.34	99.35	99.34	99.35	0:03:01
5 × 10−5	0.014	99.53	99.53	99.53	99.53	0:03:03
7 × 10−5	0.013	99.55	99.55	99.55	99.55	0:03:02
9 × 10−5	0.016	99.46	99.47	99.46	99.47	0:03:01

Table 4

Results of AES-BERCNN with different dropout.

Dropout	Loss	Accuracy(%)	Precisionw(%)	Recallw(%)	F1w(%)	Time of One Epoch
0.1	0.014	99.48	99.49	99.48	99.49	0:03:01
0.2	0.014	99.63	99.64	99.63	99.63	0:03:06
0.3	0.012	99.6	99.6	99.6	99.6	0:03:03
0.4	0.013	99.55	99.55	99.55	99.55	0:03:02
0.5	0.015	99.52	99.53	99.52	99.52	0:03:04

Table 5

Ablation experiment results.

Model	Loss	Accuracy(%)	Precisionw(%)	Recallw(%)	F1w(%)	Time of One Epoch
BERT	0.04	98.84	98.85	98.84	98.84	0:05:17
ATE	0.025	99.19	99.21	99.19	99.20	0:02:55
ATE-BiGRU	0.018	99.35	99.36	99.35	99.36	0:03:20
ATE-BiGRU-CNN(AES-BERCNN)	0.014	99.63	99.64	99.63	99.63	0:03:06

Table 6

Parameters of comparative models.

Model	Parameters	Value	Model	Parameters	Value
TextCNN	Batch_Size	128	DPCNN	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Dropout	0.4		Dropout	0.4
	Filter_Sizes	(2, 3, 4)		Num_Filters	250
	Num_Filters	256
BiLSTM	Batch_Size	128	BiGRU	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Dropout	0.4		Dropout	0.4
	Hidden_Size	128		Hidden_Size	128
BERT-TextCNN	Batch_Size	128	BERT-DPCNN	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Dropout	0.4		Hidden_Size	768
	Hidden_Size	768		Num_Filters	128
	Filter_Sizes	(2, 3, 4)
	Num_Filters	256
BERT-BiLSTM	Batch_Size	128	BERT-BiGRU	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Hidden_Size	768		Hidden_Size	768
	Dropout	0.4		Dropout	0.4
	Num_Filters	256		Num_Filters	256
	Rnn_Hidden	768		Rnn_Hidden	768
ATE-TextCNN	Batch_Size	128	ATE-DPCNN	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Dropout	0.4		Hidden_Size	768
	Num_Filters	256		Num_Filters	128
	Hidden_Size	768
	Filter_Sizes	(2, 3, 4)
ATE-BiLSTM	Batch_Size	128	ATE-BiGRU	Batch_Size	128
	Maximum Sequence Length	32		Maximum Sequence Length	32
	Epoch	10		Epoch	10
	Learning_Rate	3 × 10−5		Learning_Rate	3 × 10−5
	Dropout	0.4		Dropout	0.4
	Hidden_Size	768		Hidden_Size	768
	Num_Filters	256		Num_Filters	256
	Rnn_Hidden	768		Rnn_Hidden	768

Table 7

Comparative model experimental results on agricultural question test set.

Model	Loss	Accuracy (%)	Precisionw (%)	Recallw (%)	F1w (%)	Time of One Epoch
TextCNN	0.11	96.62	96.58	96.62	96.58	0:00:11
DPCNN	0.076	97.85	97.86	97.85	97.84	0:00:16
BiLSTM	0.17	95.43	95.43	95.43	95.37	0:00:08
BiGRU	0.15	95.63	95.62	95.63	95.61	0:00:07
BERT-TextCNN	0.02	99.28	99.29	99.28	99.29	0:05:53
BERT-DPCNN	0.029	99.07	99.1	99.07	99.08	0:05:50
BERT-BiLSTM	0.027	99.25	99.27	99.25	99.26	0:06:04
BERT-BiGRU	0.021	99.34	99.35	99.34	99.35	0:06:07
ATE-TextCNN	0.021	99.31	99.32	99.31	99.32	0:03:13
ATE-DPCNN	0.019	99.36	99.37	99.36	99.37	0:03:03
ATE-BiLSTM	0.025	99.22	99.24	99.22	99.23	0:03:24
ATE-BiGRU	0.018	99.35	99.36	99.35	99.36	0:03:20
AES-BERCNN	0.014	99.63	99.64	99.63	99.63	0:03:06

Table 8

Comparative experimental results on Tsinghua News test set.

Model	Loss	Accuracy (%)	Precisionw (%)	Recallw (%)	F1w (%)	Time of One Epoch
TextCNN	0.36	88.87	88.89	88.87	88.88	0:00:20
DPCNN	0.32	89.93	89.97	89.93	89.92	0:00:30
BiLSTM	0.43	86.44	86.51	86.44	86.42	0:00:17
BiGRU	0.45	85.32	85.28	85.32	85.27	0:00:16
BERT-TextCNN	0.29	91.28	91.34	91.28	91.28	0:13:33
BERT-DPCNN	0.31	90.74	90.92	90.74	90.76	0:13:18
BERT-BiLSTM	0.32	91.03	91.05	91.03	91.02	0:13:56
BERT-BiGRU	0.3	90.77	90.78	90.77	90.76	0:14:16
ATE-TextCNN	0.29	91.29	91.33	91.29	91.29	0:07:29
ATE-DPCNN	0.31	90.23	90.32	90.23	90.24	0:07:02
ATE-BiLSTM	0.32	90.85	90.86	90.85	90.84	0:07:54
ATE-BiGRU	0.31	90.67	90.75	90.67	90.7	0:07:47
AES-BERCNN	0.28	91.14	91.25	91.14	91.17	0:06:59

References

1. Wu, H.R.; Guo, W.; Deng, Y.; Wang, H.Q.; Han, X.; Huang, S.F. Review of Semantic Analysis Techniques of Agricultural Texts. Trans. Chin. Soc. Agric. Mach.; 2022; 53, pp. 1-16.

2. Lan, Y.B.; Guo, Y.Q.; Chen, Q.Z.; Lin, S.M.; Chen, Y.T.; Deng, X.L. Visual question answering model for fruit tree disease decision-making based on multimodal deep learning. Front. Plant Sci.; 2023; 13, 1064399. [DOI: https://dx.doi.org/10.3389/fpls.2022.1064399] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36684756]

3. Guo, Z.L.; Kuang, Z.; Wang, Y.; Zhao, Y.X.; Tao, Y.H.; Cheng, C.; Yang, J.; Lu, X.Y.; Hao, C. PmiREN: A comprehensive encyclopedia of plant miRNAs. Nucleic Acids Res.; 2020; 48, pp. D1114-D1121. [DOI: https://dx.doi.org/10.1093/nar/gkz894] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31602478]

4. Lagos-Ortiz, K.; Medina-Moreira, J.; Alarcón-Salvatierra, A.; Morán, M.F.; Del Cioppo-Morstadt, J.; Valencia-García, R. Decision Support System for the Control and Monitoring of Crops. Proceedings of the 2nd International Conference on ICTs in Agronomy and Environment (CITAMA); Guayaquil, Ecuador, 22–25 January 2019; pp. 20-28.

5. Li, C.; Liu, N.; Zheng, G.F.; Yang, J.; Dao, L. Research Review on Short Text Classification Method Based on Word Vector Model. J. Nanjing Norm. Univ. Eng. Technol. Ed.; 2024; Available online: https://link.cnki.net/urlid/32.1684.T.20240724.1706.002 (accessed on 25 July 2024).

6. Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv; 2013; arXiv: 1301.3781

7. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pretraining of deep bidirectional transformers for language understanding. arXiv; 2018; arXiv: 1810.04805

8. Haddoud, M.; Mokhtari, A.; Lecrop, T.; Abdeddaïm, S. Combining supervised term-weighting metrics for SVM text classification with extended term representation. Knowl. Inf. Syst.; 2016; 49, pp. 909-931. [DOI: https://dx.doi.org/10.1007/s10115-016-0924-1]

9. Feng, G.Z.; Guo, J.H.; Jing, B.Y.; Sun, T.L. Feature subset selection using naive Bayes for text classification. Pattern Recognit. Lett.; 2015; 65, pp. 109-115. [DOI: https://dx.doi.org/10.1016/j.patrec.2015.07.028]

10. Song, J.; Huang, X.L.; Qin, S.J.; Song, Q. A bi-directional sampling based on K-means method for imbalance text classification. Proceedings of the IEEE ACIS 15th International Conference on Computer and Information Science (ICIS); Okayama, Japan, 26–29 June 2016.

11. Wang, H.T.; He, J.; Zhang, X.H.; Liu, S.F. A Short Text Classification Method Based on N-Gram and CNN. Chin. J. Electron.; 2020; 29, pp. 248-254. [DOI: https://dx.doi.org/10.1049/cje.2020.01.001]

12. Ghosh, R. A recurrent neural network based deep learning model for text and non-text stroke classification in online handwritten Devanagari document. Multimed. Tools Appl.; 2022; 81, pp. 24245-24263. [DOI: https://dx.doi.org/10.1007/s11042-022-12767-6]

13. Sari, W.K.; Rini, D.P.; Malik, R.F. Text Classification Using Long Short-Term Memory. Proceedings of the 3rd International Conference on Electrical Engineering and Computer Science (ICECOS); Batam, Indonesia, 2–3 October 2019; pp. 150-155.

14. Zulqarnain, M.; Ghazali, R.; Hassim, Y.M.M.; Aamir, M. An Enhanced Gated Recurrent Unit with Auto-Encoder for Solving Text Classification Problems. Arab. J. Sci. Eng.; 2021; 46, pp. 8953-8967. [DOI: https://dx.doi.org/10.1007/s13369-021-05691-8]

15. Ramos, J. Using tf-idf to determine word relevance in document queries. Proceedings of the First Instructional Conference on Machine Learning; Los Angeles, CA, USA, 23–24 June 2003; Volume 242, pp. 29-48.

16. Wei, F.; Duan, Q.; Xiao, X.; Zhang, L. Classification technique of Chinese agricultural text information based on SVM. Trans. Chin. Soc. Agric. Mach.; 2015; 46, pp. 174-179.

17. Du, Y.R.; Sun, C.; Zhao, M. Learning of concepts and taxonomic relation for vegetable domain ontology based on Chinese wiki encyclopedia. Comput. Digit. Eng.; 2015; 43, pp. 1761–1766+1785.

18. Duan, Q.L.; Zhang, L.; Liu, Y.R. Automatic extraction method of hot words based on agricultural network information classification. Trans. Chin. Soc. Agric. Mach.; 2018; 49, pp. 160-167.

19. Zheng, L.; Qi, S.; Tian, L.; Yang, L. Entity relation extraction of news texts for food safety events. Trans. Chin. Soc. Agric. Mach.; 2020; 51, pp. 244-253.

20. Xu, J.L.; Hsu, Y.L. Analysis of agricultural exports based on deep learning and text mining. J. Supercomput.; 2022; 78, pp. 10876-10892. [DOI: https://dx.doi.org/10.1007/s11227-021-04238-w] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35125649]

21. Alami, N.; Meknassi, M.; En-Nahnahi, N. Enhancing unsupervised neural networks based text summarization with word embedding and ensemble learning. Expert Syst. Appl.; 2019; 123, pp. 195-211. [DOI: https://dx.doi.org/10.1016/j.eswa.2019.01.037]

22. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.S.; Dean, J. Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst.; 2013; pp. 3111-3119.

23. Xu, T.Y.; Zhao, D.; Zhou, Y.; Feng, S.; Wang, H. Research on method of intelligent Q & A for rice pests and diseases based on word2vec and Attention Seq2Seq. J. Shenyang Agric. Univ.; 2019; 50, pp. 378-384.

24. Chen, Y.; Chen, A.X.; Dong, Y.B.; Zhao, X.; Hou, W. Methods of food safety question answering system based on LSTM. Trans. Chin. Soc. Agric. Mach.; 2019; 50, pp. 380-384.

25. Wang, H.R.Q.; Zhu, H.J.; Wu, H.R.; Wang, X.M.; Han, X.; Xu, T.Y. A Densely Connected GRU Neural Network Based on Co-attention Mechanism for Chinese Rice-Related Question Similarity Matching. Agronomy; 2021; 11, 1307. [DOI: https://dx.doi.org/10.3390/agronomy11071307]

26. Xiao, Z.Y.; Yang, S.Q.; Duan, X.L.; Tang, D.Z.; Guo, Y.; Li, Z.Y. E3W-A Combined Model Based on Greedy Soup Weighting Strategy for Chinese Agricultural News Classification. Appl. Sci.; 2022; 12, 12059. [DOI: https://dx.doi.org/10.3390/app122312059]

27. Yang, G.F.; Yang, Y. Question classification of common crop disease question answering system based on BERT. J. Comput. Appl.; 2020; 40, pp. 1580-1586.

28. Yuan, P.S.; Li, R.L.; Wang, C.; Xu, H.L. Entity relationship extraction from rice phenotype knowledge graph based on BERT. Trans. Chin. Soc. Agric. Mach.; 2021; 52, pp. 151-158.

29. Wang, H.R.Q.; Wu, H.R.; Wang, Q.H.; Qiao, S.C.; Xu, T.Y.; Zhu, H.J. A Dynamic Attention and Multi-Strategy-Matching Neural Network Based on Bert for Chinese Rice-Related Answer Selection. Agriculture; 2022; 12, 176. [DOI: https://dx.doi.org/10.3390/agriculture12020176]

30. Cao, Y.; Sun, Z.X.; Li, L.; Mo, W.N. A Study of Sentiment Analysis Algorithms for Agricultural Product Reviews Based on Improved BERT Model. Symmetry; 2022; 14, 1604. [DOI: https://dx.doi.org/10.3390/sym14081604]

31. Chen, T.Y.; Qian, Y.Q.; Wang, Y.J.; Chen, X.J.; Di, O.Y.; Dong, S.H.; Li, X.; Zhao, J.B.; Huang, L. RoBERT-Agr: An Entity Relationship Extraction Model of Massive Agricultural Text Based on the RoBERTa and CRF Algorithm. Proceedings of the IEEE 8th International Conference on Big Data Analytics (ICBDA); Harbin, China, 3–5 March 2023; pp. 113-120.

32. Shi, L.; Duan, Q.G.; Ma, X.M.; Weng, M. The Research of Support Vector Machine in Agricultural Data Classification. Proceedings of the 5th International Conference on Computer and Computing Technologies in Agriculture (CCTA); Beijing, China, 29–31 October 2011.

33. Duan, Q.L.; Wei, F.F.; Zhang, L.; Xiao, X.Y. Automatic acquisition and classification system for agricultural network information based on Web data. Trans. Chin. Soc. Agric. Eng.; 2016; 32, pp. 172-178.

34. Cui, X.H.; Shi, D.Y.; Chen, Z.B.; Xu, F. Parallel Forestry Text Classification Technology Based on XGBoost in Spark Framework. Trans-Actions Chin. Soc. Agric. Mach.; 2019; 50, pp. 280-287.

35. Du, R.P.; Xian, G.J.; Kou, Y.T. Text Feature Extraction from Agricultural Science and Technology Literature Based on Improved TF-IDF-CHI Algorithm. Digital Library Forum.; 2019; 8, pp. 18-24.

36. Zhao, M.; Du, H.; Dong, C.; Chen, C. Diet health text classification based on word2vec and LSTM. Trans. Chin. Soc. Agric. Mach.; 2017; 48, pp. 202-208.

37. Zhao, M.; Dong, C.C.; Dong, Q.X.; Chen, Y. Question Classification of Tomato Pests and Diseases Question Answering System Based on BIGRU. Trans. Chin. Soc. Agric. Mach.; 2018; 49, pp. 271-276.

38. Zhang, M.Y.; Wu, H.R.; Zhu, H.J. Analysis of extraction of semantic feature in agricultural question and answer based on convolutional model. Trans. Chin. Soc. Agric. Mach.; 2018; 49, pp. 203-210.

39. Liang, J.; Cui, B.; Jiang, H.; Shen, Y.; Xie, Y. Sentence similarity computing based on word2vec and LSTM and its application in rice FAQ question-answering system. J. Nanjing Agric. Univ.; 2018; 41, pp. 946-953.

40. Zhou, H.; Guo, X.C.; Liu, C.Q.; Tang, Z.; Lu, S.H.; Li, L. Question Similarity Measurement of Chinese Crop Diseases and Insect Pests Based on Mixed Information Extraction. KSII Trans. Internet Inf. Syst.; 2021; 15, pp. 3991-4010.

41. Jin, N.; Zhao, C.J.; Wu, H.R.; Miao, Y.S.; Li, S.; Yang, B.Z. Classification Technology of Agricultural Questions Based on BiGRU_MulCNN. Trans. Chin. Soc. Agric. Mach.; 2020; 51, pp. 199-206.

42. Feng, S.; Xu, T.Y.; Zhou, Y.C.; Zhao, D.X.; Jin, N.; Wang, H.R.Q. Rice Knowledge Text Classification Based on Deep Convolution Neural Network. Trans. Chin. Soc. Agric. Mach.; 2021; 52, pp. 257-264.

43. Wang, X.M.; Wang, H.R.Q.; Zhao, G.C.; Liu, Z.C.; Wu, H.R. ALBERT over Match-LSTM Network for Intelligent Questions Classification in Chinese. Agronomy; 2021; 11, 1530. [DOI: https://dx.doi.org/10.3390/agronomy11081530]

44. Li, L.; Diao, L.; Tang, Z.; Bai, Z.; Zhou, H.; Guo, X.C. Question Classification Method of Agricultural Diseases and Pests Based on BERT_Stacked LSTM. Trans. Chin. Soc. Agric. Mach.; 2021; 52, pp. 172-177.

45. Wang, H.R.Q.; Wu, H.R.; Zhu, H.J.; Miao, Y.S.; Wang, Q.H.; Qiao, S.C.; Zhao, H.Y.; Chen, C.; Zhang, J.J. A Residual LSTM and Seq2Seq Neural Network Based on GPT for Chinese Rice-Related Question and Answer System. Agriculture; 2022; 12, 813. [DOI: https://dx.doi.org/10.3390/agriculture12060813]

46. Kim, Y. Convolutional Neural Networks for Sentence Classification. arXiv; 2014; arXiv: 1408.5882

47. Johnson, R.; Zhang, T. Deep pyramid convolutional neural networks for text categorization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017); Vancouver, BC, Canada, 30 July–4 August 2017.

48. Correa-Delval, M.; Sun, H.J.; Matthews, P.C.; Jiang, J. Appliance Classification using BiLSTM Neural Networks and Feature Extraction. Proceedings of the 11th IEEE-PES Innovative Smart Grid Technologies Europe (IEEE-PES ISGT Europe); Espoo, Finland, 18–21 October 2021; pp. 175-179. [DOI: https://dx.doi.org/10.1109/ISGTEUROPE52324.2021.9640061]

49. Chen, X.N.; Wang, C.Y.; Li, D.; Sun, X.M. A New Early Rumor Detection Model Based on BiGRU Neural Network. Discret. Dyn. Nat. Soc.; 2021; 2021, 2296605. [DOI: https://dx.doi.org/10.1155/2021/2296605]

50. Sun, K.; Shi, X.; Gao, H.; Kuang, H.; Ma, X.; Rong, G.; Shao, D.; Zhao, Z.; Zhang, H. Incorporating Pre-trained Transformer Models into TextCNN for Sentiment Analysis on Software Engineering Texts. Proceedings of the 13th Asia-Pacific Symposium on Internetware—A Software Paradigm for Internet Computing; Hohhot, China, 11–12 June 2022; pp. 127-136.

51. Guo, X.; Wang, J.; Gao, G.; Zhou, J.; Li, Y.; Cheng, Z.; Miao, G. Efficient Agricultural Question Classification with a BERT-Enhanced DPCNN Model. IEEE Access; 2024; 12, pp. 109255-109268. [DOI: https://dx.doi.org/10.1109/ACCESS.2024.3438848]

52. Xiong, Y.; Wei, N.Q.; Qiao, K.; Li, Z.Y.; Li, Z.W. Exploring Consumption Intent in Live E-Commerce Barrage: A Text Feature-Based Approach Using BERT-BiLSTM Model. IEEE Access; 2024; 12, pp. 69288-69298. [DOI: https://dx.doi.org/10.1109/ACCESS.2024.3399095]

53. Ma, R.N.; Cao, H.; Song, Z.H.; Wu, X.Y. Study on Chinese Semantic Entity Recognition Method for Cabin Utilizing BERT-BiGRU Model. IEEE Access; 2024; 12, pp. 56042-56049. [DOI: https://dx.doi.org/10.1109/ACCESS.2024.3386760]