DynGraph-BERT: Combining BERT and GNN Using Dynamic Graphs for Inductive Semi-Supervised Text Classification

Abstract

The combination of Bidirecional Encoder Representations from Transformers (BERT) and Graph Neural Networks (GNNs) has been extensively explored in the text classification literature, usually employing BERT as a feature extractor combined with heterogeneous static graphs. BERT transfers information via token embeddings, which are propagated through GNNs. Text-specific information defines a static heterogeneous graph. Static graphs represent specific relationships and do not have the flexibility to add new knowledge to the graph. To address this issue, we build a tied connection between BERT and GNN exclusively using token embeddings to define the graph and propagate the embeddings, which can force the BERT to redefine the GNN graph topology to improve accuracy. Thus, in this study, we re-examine the design spaces and test the limits of what this pure homogeneous graph using BERT embeddings can achieve. Homogeneous graphs offer structural simplicity and greater generalization capabilities, particularly when integrated with robust representations like those provided by BERT. To improve accuracy, the proposed approach also incorporates text augmentation and label propagation at test time. Experimental results show that the proposed method outperforms state-of-the-art methods across all datasets analyzed, with consistent accuracy improvements as more labeled examples are included.

1. Introduction

Text classification is a key task within natural language processing techniques. Many text classification tools rely on a large amount of labeled data to better recognize patterns in text classes, achieving better performance. However, obtaining an acceptable volume of labeled data is expensive. In response, unsupervised and semi-supervised methods have been developed to reduce this dependency, lowering costs while maintaining high accuracy levels.

Combining Bidirecional Encoder Representations from Transformers (BERT) with Graph Convolutional Networks (GCNs) has been explored in text classification tasks, especially using heterogeneous graphs [1,2,3,4]. These studies adopted heterogeneous graphs because of their rich representation and ability to capture complex data relationships [5,6]. However, the heterogeneous graphs introduce challenges, such as scalability, manual decisions to model relationships between types of nodes, and difficulties in propagating knowledge to the backbone [6,7].

Conversely, homogeneous graphs emerge as a promising alternative, offering structural simplicity and generalization capabilities, particularly when integrated with robust representations like BERT [8] embeddings. However, homogeneous graphs usually have less information, and they naturally have less expressiveness [5,6]. To alleviate this issue, the proposal introduces a series of text augmentations designed to enhance representational power. Text augmentation has emerged as a relevant strategy to improve semi-supervised learning, which uses unlabeled data to enhance model performance. By generating additional data through transformations of existing samples, data augmentation mitigates overfitting and enhances generalization capabilities [9]. This approach can benefit models based on dynamic graphs, where augmented graphs introduce subtle variations of tokens that enrich the training process, retaining the semantic meaning of the original text.

The semi-supervised learning literature often highlights BERT as a powerful feature extractor, with joint training alongside Graph Neural Networks (GNN) enhancing capabilities while keeping a static graph topology. This is especially true for Text Graph Convolutional Network (TextGCN) [10] and BertGCN [2] variants. The proposal better integrates BERT and GNN not only as a feature extractor but also as a guide for dynamic graph construction. This enables the proposal to capture richer semantic information and achieve a more efficient integration between representation learning and graph structure.

The literature reviews on GNNs in texts [11] and semi-supervised setups [12] indicate that many methods still rely on static graphs. Dynamic graph construction is an emerging and important area of interest [13,14,15], especially in computer vision [16,17,18]. In text classification, it is still not extensively explored. The method proposed in [17] performs graph-level classification for inductive learning by dynamically learning new node representations and adjusting edges. In contrast, the approaches in [19,20] introduce dynamicity only for the edges. Our method introduces an approach to dynamically adapt textual data into corpus-level graphs using BERT embeddings, where nodes represent documents and new representations and connections are generated. By combining augmented data with dynamic graphs, the increased diversity of data seen during training enhances the ability of the model to generalize, providing an advantage over other methods.

Another idea not widely explored in the text domain is label propagation during test time. The unsupervised information of the test batch can be used to improve the graph. The label propagation at the test time mechanism can access features of the test data during inference, improving the accuracy in graph-related tasks [21,22,23,24,25]. While this characteristic is inherent in some graph scenarios, we have explored a way to integrate these benefits into our proposal.

The main contribution of this study is a semi-supervised technique for text classification named Dynamic Graph with BERT (DynGraph-BERT). The proposal combines text augmentation, dynamic graph construction, and label propagation at test time. This unique combination allows for the design of an algorithm to compete with state-of-the-art semi-supervised text classification algorithms based on GNNs.

2. Related Work

Several works seek improvement for text classification on semi-supervised scenario. Recent methods in graph neural networks often use the TextGCN [10] as a starting point. The TextGCN represents the corpus as a graph, where nodes correspond to words and documents connected through relationships based on word co-occurrence and word occurrence in a document [10]. The model learns text representations by propagating information through graph connections, capturing global semantic relationships [10]. The Heterogeneous Graph Transformer (HGT) [26] was designed to model large-scale graphs using specific parameters for the node and edge types to perform dedicated attention. It also incorporates relative temporal encoding to capture dynamic dependencies. The HGSampling algorithm, inserted into the method, enables heterogeneous mini-batch sampling, allowing it to efficiently train on large-scale web graphs [26]. The Heterogeneous Graph Attention Network (HGAT) introduces a heterogeneous information network (HIN) to model short texts by incorporating additional information and capturing their relationships to mitigate semantic sparsity [27]. Based on the HIN graph, the method applies a two-level Graph Attention Network (GAT)—node-level and type-level—to learn the relationships between neighboring nodes, optimizing short-text classification in a semi-supervised scenario [27].

The Short-Text Heterogeneous Graph Network (STHGN) is a self-supervised method for short-text classification that addresses the lack of information in the corpus by modeling it as a heterogeneous graph [28]. It employs a self-attention-based heterogeneous graph neural network (SAHGNN) to learn embeddings for short texts. Additionally, it implements a self-supervised learning framework to explore both internal and external similarities among texts [28]. Heterogeneous Graph-Convolution-Network-Based Short-Text Classification (SHGCN) constructs a heterogeneous graph using nodes representing entities and words connected through mutual information and confidence between documents, words, and entities. The features are represented within a word graph, combined with BERT embeddings, and further enhanced using a Bidirectional Long Short-Term Memory (BiLSTM) [29].

The TextGCN, HGAT, STHGN, SHGCN, and Vocabulary Graph Convolutional Network (VGCN) process data using a static graph, limiting their ability to discover additional node patterns during training by enhancing knowledge between epochs. The HGT operates with dynamic graphs but ignores the embedding information of pretrained models.

Another interesting algorithm is InducT-GCN (InducTive Graph Convolutional Networks for Text classification) [30]. As an inductive approach, this study constructs the graph exclusively using training data, enabling the model to train without relying on any information from the test set. The graph nodes are divided into two types—nodes representing documents and those representing unique words in the training documents. The features are one-hot vectors for words and Term Frequency–Inverse Document Frequency (TF-IDF) for documents, enabling alignment between the two. The two types of edges—words with words based on Point-Wise Mutual Information (PMI) considering the co-occurrence of connected words and documents with words (if the words are in the text)—then connect them and are determined by TF-IDF values.

Induct-GCN relies solely on internal information and does not employ transfer learning to achieve higher accuracies. When unknown words are encountered, this information is excluded from the graph and the model, making it heavily dependent on the training data.

As already mentioned, most of the methods use static graphs, and the term frequency text representation is still the preferred strategy in the literature found to build the graphs. In this work, we seek to follow a different avenue. We re-examine the design spaces and test the limits of what a pure homogeneous graph can achieve. The proposal focuses on the expressiveness of BERT enhanced by data augmentation and the transductive at the test time approach. We also observe that the relationship of the documents at the batch has not been explored in the form of label propagation at test time in these methods. Considering these research gaps, we propose DynGraph-BERT.

3. Preliminaries

3.1. Graph Neural Network

Graph neural networks (GNNs) have gained popularity largely through the works of [31] on graph convolutional networks. This technique represents a combination of artificial neural networks tailored to handle structured data in graphs. The benefits of using graphs to structure data lie in the increased information gain compared to tabular data, as graphs are not confined to themselves; they enable the relationship between one data point and others, unveiling new associated or underlying patterns among them [32]. However, they introduce other challenges such as data heterogeneity within the graph [33,34]; the generalization of learned knowledge to new graphs; efficient data propagation; and the trade-off between model accuracy and computational efficiency [30,34,35,36,37], over-smoothing, under-reaching, and over-squashing [37].

A graph neural network is a machine learning framework that models and analyzes structured data through graphs. A graph G is an ordered pair $(V,E)$, where V represents a set of nodes, and E represents a set of edges that link the nodes. A GNN is a parametric model consisting of multiple layers, where each layer deals with the aggregation of information from neighboring nodes and edges.

The fundamental operation of the GNN involves aggregating information from the neighbors of a node and subsequently updating the representation of that node. This process is performed iteratively across l layers. At each layer, the representation of a node $v_i$, denoted as $h_i^{\left(l\right)}$, is updated based on the aggregation of information from its neighbors, $N\left(v_i\right)$, as defined in Equation (1):

(1)$\begin{array}{c}\hfill ag{g}_i^{\left(l\right)}=AGGREGAT{E}^{\left(l\right)}\left(\{h_{v_j}^{(l-1)}:\forall v_j\in N\left(v_i\right)\}\right)\end{array}$

The Equation (2) represents the node $v_i$ updated using the aggregated information from its neighbors [38]:

(2)$\begin{array}{c}\hfill h_i^{\left(l\right)}=UPDAT{E}^{\left(l\right)}(h_i^{(l-1)},ag{g}_i^{\left(l\right)})\end{array}$

The aggregate function can be a mean, sum, min, or max from its neighbor features’ values [39]. The update function is generally a feedforward neural network. In the case of the GCN, the update function takes the following Equation (3) to update the layers [31]:

(3)$h^{\left(l\right)}=f(A^{*}·h^{(l-1)}·W^{\left(l\right)})$

(4)$A^{*}=I_n+D^{-{\displaystyle \frac{1}{2}}}·A·D^{-{\displaystyle \frac{1}{2}}}$

where $W^{\left(l\right)}$ is the weight matrix of layer l, which will be learned during training, f is a non-linear activation function, and $A^{*}$ is the normalized adjacency matrix of the graph, as shown in the Equation (4). D is the diagonal degree matrix of the layer l, where $D_{ii}={\sum }_j{A}_{ij}$. The normalization by $D^{-{\displaystyle \frac{1}{2}}}$ avoids scaling issues of values during propagation. The identity matrix $I_n$ includes self-loop connections. The first $h^{\left(0\right)}$ layer constitutes the features of the data.

Through multiple layers, a GCN can effectively learn to incorporate information from the neighborhoods of nodes, allowing the capture of complex features of data structured in a graph.

The loss function is defined as the cross-entropy error over all labeled documents [31]:

(5)$\begin{array}{c}\hfill Z=softmax\left(h_i^{\left(l\right)}\right)\end{array}$

(6)$\begin{array}{c}\hfill L=-\sum _{d\in Y_{tr}}\sum _{f=1}^C{Y}_{df}ln{Z}_{df}\end{array}$

where $Y_{tr}$ is the set of text indices that have labels, and C is the dimension of the output features, which is equal to the number of classes. Y is the label indicator matrix.

The graph neural network diffuses the features of neighboring nodes so that the layers of the network are fed with more details by operation $A^{*}·h^{l-1}$, and each layer has more diffuse details than the previous one. Bringing this to our context of text classification, when a node represents a text, its diffused features will have more features or excerpts from its neighbors, if the graph constructed is based on some similarity or co-occurrence of words contained in the documents, encoded in adjacency matrix A.

3.2. BERT

BERT (Bidirectional Encoder Representations from Transformers) changes the way to process textual data by leveraging bidirectional context and learning rich representations through the attention mechanism [40]. The model undergoes a large-scale pretraining phase on vast amounts of text, capturing deep relationships between words and generating contextualized embeddings that effectively encode semantic and syntactic aspects of language. In the fine-tuning phase, these embeddings are further adapted to specific tasks.

During training, BERT uses special tokens to structure input sequences: the CLS token marks the beginning of the text, the EOS token indicates the end of the text, and the SEP token separates sentences or ends text too. Some of these tokens are for the Next Sentence Prediction (NSP) pretraining task. For text classification, the CLS token serves as a representation of the entire input, aggregating relevant information, and is fed into a fully connected layer whose output size matches the number of classes.

In DynGraph-BERT, we utilize two types of embeddings from BERT: the CLS token representation and the average of the final layer. The CLS embedding captures task-specific information for classification, while the averaged embeddings provide a more generalized representation of the text, independent of its class.

Since the introduction of BERT, several models have addressed its limitations and improved its effectiveness. Notable advancements include RoBERTa [41], DeBERTa [42], XLNet [43], MPNet [44], and ModernBERT [45], which enhance embedding quality and performance in downstream tasks. In our study, we use BERT as a baseline to ensure a fair comparison with other models that also adopt it as a reference.

3.3. Transductive Versus Inductive

In a node classification problem of a graph, let V be the set of nodes and E be the set of edges of a graph $G=(V,E)$. In inductive learning, we are interested in learning a function such that $f:V\to Y$, where Y is the set of labels for each node.

The training space of an inductive model is defined by a graph $G_{tr}=(V_{tr},E_{tr})$ such that $V_{tr}\subseteq V$ and $E_{tr}\subseteq E$, where $V_{tr}$ and $E_{tr}$ are, respectively, the sets of training nodes and the set of edges that connect nodes in $V_{tr}$. The set of edges $E_{tr}$ can be defined as $E_{tr}=\{(u,v)\in E|u\in V_{tr}\wedge v\in V_{tr}\}$. In the transductive setting, the model has access to all the data (train and test) features except the class values for testing.

Thus, the training input for the GNN network in an inductive approach is only the graph $G_{tr}$ and $Y_{tr}$. Compared to the transductive approach, the training input graph for the GNN network is formed by $G=(V,E)$ and $Y_{tr}$. The advantage space is created by having full access to V and E during training. The adjacency matrix during training in the inductive mode is sized as $|V_{tr}| \times |V_{tr}|$, while in transductive mode, it is $|V_{tr}+V_{te}| \times |V_{tr}+V_{te}|$.

At the end of training, an inductive model allows for the creation of edges $(v_i,v_j)\in E$, where $v_i\in V_{tr}$ and $v_j\in V_{te}$, or the addition of new nodes $v_j\in V$ (representing new samples from the real world). In contrast, the transductive model has already determined the classes for all nodes. If new data are introduced to a transductive model, the model must be retrained with these data.

The transductive scenario occurs when the unlabeled test data are at our disposal during the training stage. The access to unlabeled test data usually gives some advantage to the transductive in terms of accuracy when compared with the inductive. This handy improvement comes with a price. Transductive models cannot predict future data, only present and past data. Therefore, a transductive setup is less common in real-world scenarios. Inductive methods can be applied in more general setups and applications.

Much related work has been done on the transductive setup, not on the inductive setup. It is not always trivial to convert a transductive graph method to an inductive graph method. In most cases, transductive methods incorporate graph-building mechanisms within their training procedures, making the direct conversion to inductive methods difficult. For this reason, we designed the proposal in an inductive setup. However, we took advantage of some ideas from the transductive setup using label propagation at test time.

4. DynGraph-BERT

The DynGraph-BERT is a welcome combination of three ideas: data augmentation, dynamic graph construction, and label propagation at test time. In the following sections, we will dive into these topics, followed by the proposal algorithm.

4.1. Data Augmentation

To address the lack of labeled data, we employed two data augmentation techniques. The first involves augmenting data at the text level by modifying words and their structure using the library nlpaug (https://github.com/makcedward/nlpaug (accessed on 12 February 2025)) [46]. The second technique is applied to the model-trained embeddings by adding noise to the embedding vectors. Further details about the first technique are provided in the following paragraphs, while the second is discussed at the end of this section.

The nlpaug library provides various strategies for text modification. These techniques increase the variability of the dataset without introducing significant semantic bias, enhancing the robustness of the model when handling noise and variations in text. Specifically, we applied four main approaches:

• Swap: replacing words with synonyms;

• Omitting: omitting arbitrary words;

• Typos: introducing typos;

• Change order: altering the order of words in a sentence.

Table 1 presents examples of these transformations, demonstrating how these techniques preserve the overall meaning while enhancing dataset diversity. The selection of these techniques balances computational costs and performance improvement [47,48]. Data augmentation with nlpaug has two application points, as illustrated in Figure 1 within the cylinder labeled Aug. Train Data. The first application contributes to creating the Global Context Graph, which is represented in the gray space in Figure 1. The second is actively integrated into the training process through batches, which is represented in the red space in Figure 1.

The data augmentation process yields two outcomes: the augmented data, which are incorporated into the batch training process, and the augmented graph, comprising unlabeled training data alongside augmented unlabeled data. An evaluation of both outcomes will be conducted in the ablation study section.

Noise augmentation data are introduced to produce augmented embeddings, improving the robustness of embeddings, as denoted by ${\tilde{\mathbf{v}}}_i$ [49,50]. This noise is a small variation applied to each dimension of the embeddings, represented by a vector of the same size as the embeddings. The adjustments in each dimension are minor and follow a normal distribution, as described in Equation (7):

(7)${\tilde{\mathbf{v}}}_i={\mathbf{v}}_i+\mathcal{N}(\mu ,{\sigma }^2)$

where ${\mathbf{v}}_i$ represents the i-th original embedding, and $\mathcal{N}$ is the normal distribution with mean $\mu$ and variance ${\sigma }^2$. The noise is applied to both CLS and AVG embeddings during training steps, and the usage will be explained in detail later. The CLS embedding is a vector representation for the CLS token inserted at the beginning of documents, with the aim of condensing information from the text to be classified. The AVG embedding is the average of token embeddings in the text, including the CLS and SEP tokens; the EOS token is not applicable in this task.

4.2. Dynamic Graph

The dynamic graph plays a critical role in both the training process and the prediction of nodes within the graph. Its construction begins with what we call the Global Context Graph. In this graph, nodes represent texts, and their features are embeddings generated by a frozen BERT model denoted as $BER{T}_e$. This ensures that the initial feature representations remain consistent throughout the construction phase.

Textual data are represented as $t_i$, where i is the index of the text in the training dataset and is processed by $BER{T}_e$ with all layers frozen, that is, its parameters are locked. The encoder output produces the feature representation ${\mathbf{v}}_i$, as described in Equation (8). Each resulting embedding vector is associated with a node $v_i$, $\forall v_i\in V_{tr}$. The embeddings used in the graph construction are average embeddings (AVG embeddings) derived from the output of the last BERT layer, including the CLS token. Including the CLS token is justified, as BERT encodes class information within this embedding, making it a valuable feature for downstream tasks [51].

The connections are based on their embeddings and nearest neighbors. Edges are added under two conditions: a specified number of nearest neighbors k and a minimum similarity threshold m, which are both configurable hyperparameters. The similarity is quantified as the inverse of the cosine distance between the text embeddings, as defined in Equation (9). The resulting graph is weighted, with each edge’s weight corresponding to the calculated similarity:

(8)$\begin{array}{cc}\hfill {\mathbf{v}}_i& =BER{T}_e\left(t_i\right)\hfill \end{array}$

(9)$\begin{array}{cc}\hfill sim({\mathbf{v}}_i,{\mathbf{v}}_j)& =1-{\displaystyle \frac{cosine\_dist({\mathbf{v}}_i,{\mathbf{v}}_j)}{2}}\hfill \end{array}$

(10)$\begin{array}{cc}\hfill cosine\_dist({\mathbf{v}}_i,{\mathbf{v}}_j)& =1-{\displaystyle \frac{{\mathbf{v}}_i·{\mathbf{v}}_j}{\parallel {\mathbf{v}}_i\parallel \times \parallel {\mathbf{v}}_j\parallel }}\hfill \end{array}$

where $cosine\_dist(·)$ is a function that brings similar vectors closer to a value of 1, representing maximum similarity. In contrast, dissimilar vectors are assigned a value closer to 0.

In the training process, the dynamic nature of the graph is introduced. During each epoch, the model processes data in batches. For each batch, new embeddings are computed by a trainable BERT model, which is fine-tuned for the classification task. These embeddings are perturbed with augmented noise and added to the Global Context Graph following the same edge rules. As a result, a unique batch-specific graph is generated for training, as illustrated in Figure 2a. This batch graph is processed by a GCN, which refines its weights based on the classification task.

At the end of each epoch, the embeddings in the Global Context Graph are recalculated using the updated BERT model. This process creates a new Global Context Graph ($G_1,G_2,\dots$) at the beginning of each subsequent epoch. Each graph incorporates the knowledge gained during the previous epoch, allowing the model to adapt dynamically and integrate new insights. Figure 3 depicts the evolution of the Global Context Graph across epochs.

This iterative approach offers several advantages. First, it enables the model to incorporate dynamic changes in the representation of textual data, improving its ability to generalize. Second, it facilitates the creation of transductive graphs during the test time, leveraging information from the training process.

4.3. Label Propagation at Test Time

The graph is built following the same creation rules mentioned earlier, adhering to the specified number of k neighbors and a minimum similarity threshold. Label propagation occurs when the unlabeled test data are included in the graph. This allows any test example to contribute to the batch being predicted. In Figure 2b, this is illustrated with the graph of $batc{h}_1$, which includes unlabeled test data in yellow, unlabeled training data in blue, and the batch examples to be predicted in green.

The information propagation now also flows using unlabeled test data (yellow dots). In this way, the unlabeled test data can propagate the information through the graph, helping the GCN to improve the classification task (see the Ablation section).

4.4. DynGraph-BERT Algorithm

DynGraph-BERT is a graph-based model integrating the GCN and BERT in an inductive setup with a homogeneous graph and semi-supervised learning. Its semi-supervised nature emerges from graph construction without labels or external data during both the training and testing phases. Algorithm 1 outlines the procedural steps involved in the training process, whereas Algorithm 2 delineates the methodology for evaluating the model.

To recap the steps with data augmentation, at lines 1 and 2, the graph augmentation is built with the rules of k and m at line 4 in Algorithm 1. The encoder $BER{T}_{init}$, at line 5, is activated for training, as indicated by a flame icon in Figure 1. Each text segment is encoded, generating CLS position embeddings for classification, as in a traditional BERT model. Additionally, AVG embeddings are incorporated into the training graph, following the k-neighbor constraint and a minimum similarity threshold m. This process is depicted as a build k-NN graph in Figure 1.

Algorithm 1 DynGraph-BERT training process.

Require:  Training data $D_{train}$, Test data $D_{test}$, k-neighbor constraint, similarity threshold m, initial encoder $BER{T}_{init}$, augment data per technique a. Ensure:  Model DynGraph-BERT   1:. $D_{aug}=augment\_techniques(D_{train},a)$   2:. $D_{train}^{\prime }=D_{train}\cup D_{aug}$   3:. ${\mathbf{V}}_{cls},{\mathbf{V}}_{avg}=encode(D_{train}^{\prime },BER{T}_{init})$   4:. $G_0=build\_graph({\mathbf{V}}_{avg},k,m)$   5:. $BER{T}_0=enable\_training\left(BER{T}_{init}\right)$   6:. for each epoch e do   7:.     Training:   8:.     for each batch B in $D_{train}^{\prime }$ do   9:.        ${\mathbf{v}}_{CLS},{\mathbf{v}}_{AVG}=encode(B,BER{T}_e)$ 10:.        ${\mathbf{v}}_{CLS}={\mathbf{v}}_{CLS}+noise\left({\mathbf{v}}_{CLS}\right)$ 11:.        $Z_{BERT}=FC\left({\mathbf{v}}_{CLS}\right)$ 12:.        ${\mathbf{v}}_{AVG}={\mathbf{v}}_{AVG}+noise\left({\mathbf{v}}_{AVG}\right)$ 13:.        $G_{e,B}=mirror\_and\_add({\mathbf{v}}_{AVG},G_e,k,m)$ 14:.        $Z_{GCN}=GCN\left(G_{e,B}\right)$ 15:.        $\lambda =\sigma \left(\right(Z_{BERT}\left|\right|Z_{GCN}\left)W_{L.L.}\right)$ 16:.        $Z=\lambda Z_{GCN}+(1-\lambda )Z_{BERT}$ 17:.        Update network weights via backpropagation 18:.     end for 19:.     Graph Update: 20:.     $BER{T}_e=disable\_training\left(BER{T}_e\right)$ 21:.     ${\mathbf{V}}_{cls},{\mathbf{V}}_{avg}=encode(D_{train}^{\prime },BER{T}_e)$ 22:.     $G_e=build\_graph({\mathbf{V}}_{avg},k,m)$ 23:.     $BER{T}_{e+1}=enable\_training\left(BER{T}_e\right)$ 24:. end for 25:. return Model DynGraph-BERT

After graph construction, the learning flow resumes, updating network weights through batches. In the standard BERT flow, the CLS embedding passes through a fully connected (FC) layer, resulting in BERT logits. In the graph-based flow, AVG embeddings serve as graph features processed by two GNN layers, producing GNN logits. The final logits are computed as a weighted combination of the two. To regulate model learning, a Lambda Learner (L.L.) mechanism is integrated. This mechanism determines $\lambda$ using a linear layer that learns the optimal value for each text segment. The input to this mechanism comprises embeddings from both flows, which are concatenated as described in Equation (12):

(11)$\begin{array}{cc}\hfill f_{L.L.}& =\sigma \left(\right(Z_{BERT}\left|\right|Z_{GCN}\left)W_{L.L.}\right)\hfill \end{array}$

(12)$\begin{array}{cc}\hfill \lambda & =f_{L.L.}(Z_{BERT}\left|\right|Z_{GCN})\hfill \end{array}$

(13)$\begin{array}{cc}\hfill Z& =\lambda Z_{GCN}+(1-\lambda )Z_{BERT}\hfill \end{array}$

where $Z_{BERT}$ and $Z_{GCN}$ represent logits for the BERT and GCN flows, respectively, weighted by $\lambda$. $f_{L.L.}$ denotes the Lambda Learner function, and $\sigma$ is the sigmoid activation. At the end of each epoch, the BERT model is frozen, disabling its parameters for training while encoding new embeddings and updating the graph based on insights from the preceding epoch (Figure 3).

Algorithm 2 DynGraph-BERT evaluation process.

Require:  Training data $D_{train}$, test data $D_{test}$, k-neighbor constraint, similarity threshold m, encoder $BER{T}_e$. Ensure:  Predicted labels for $D_{test}$   1:. Prediction:   2:. Freeze all layers and remove noise addition   3:. $D=D_{train}^{\prime }\cup D_{test}$ # join unlabeled data   4:. ${\mathbf{V}}_{CLS},{\mathbf{V}}_{AVG}=encode(D,BER{T}_e)$   5:. $G=build\_graph({\mathbf{V}}_{AVG},k,m)$   6:. $Predictions=⌀$   7:. for each batch B in $D_{test}$ do   8:.     ${\mathbf{v}}_{CLS},{\mathbf{v}}_{AVG}=encode(B,BER{T}_e)$   9:.     $G_B=mirror\_and\_add({\mathbf{v}}_{AVG},G,k,m)$ 10:.     $Z_{BERT}=FC\left({\mathbf{v}}_{CLS}\right)$ 11:.     $Z_{GCN}=GCN\left(G_B\right)$ 12:.     $\lambda =\sigma \left(\right(Z_{BERT}\left|\right|Z_{GCN}\left)W_{L.L.}\right)$ 13:.     $Z=\lambda Z_{GCN}+(1-\lambda )Z_{BERT}$ 14:.     $add\_list\left(argmax\right(Z),Predictions)$ 15:. end for 16:. return Predicted labels for $D_{test}$

For evaluation, the examples to be predicted are processed in batches, all layers are frozen, and noise addition is omitted. The model generates AVG embeddings for graph construction, which are used to create new AVG and CLS embeddings for each test batch. The graph construction for prediction includes both training data and evaluation data, maintaining the same k and m values. For each batch in the evaluation set, CLS embeddings are classified through an FC layer, while AVG embeddings are processed by a GNN. The logits from both flows are combined using the $\lambda$ value for each batch, as shown in Figure 4. The final prediction is obtained by an ArgMax function on the weighting of the logits.

The model selected for the prediction of the test set corresponds to the best epoch during training based on the highest accuracy achieved on the validation set. This model is indicated as $BER{T}_{best}$ in Figure 4 and replaces the encoder $BER{T}_e$ in Algorithm 2.

5. Results

Experiments were conducted on five well-known text classification datasets, following the experimental setup of [28]. Ohsumed [52] contains titles and abstracts of medical articles, R8 [10] is a subset of the Reuters news dataset, Snippets [53] comprises search keywords, AGNews [54] is another news dataset, and DBLP [55] consists of text about scientific articles. The dataset size, number of classes, and distributions of examples across training, validation, and test sets are detailed in Table 2.

We processed the text with a BERT tokenizer using all available words and characters, because BERT was trained with raw text, which was also done in the work of [8]. For each dataset, we set the maximum number of tokens based on a percentile of 90% of the length of texts. To build the graph, we used $k=3$ and $m=0.5$ for the threshold after a Bayesian hyperparameters search based on the best validation accuracy. The Table 3 shows the other hyperparameters. The learning rates for BERT and GNN were established with the best values of [2]. We used AdamW as an optimizer and applied the warm-up following [2].

To evaluate the performance of DynGraph-BERT, we selected several baseline methods that use graphs in a semi-supervised setting for text classification, including TextGCN [10], HGT [26], HGAT [27], and STHGN [28]. Additionally, we included classical CNNs [56] for comparison purposes.

All benchmark results were sourced from [28]. Ten rounds of experiments were conducted as in [28], and the average accuracy is reported in Table 4. Ref. [28] did not provide the code or random seeds used to generate the training and test subsets. It was indicated that the subsets were chosen randomly from the corresponding sets. To ensure reproducibility, we attempted to replicate this process in a similar way and recorded the seeds used in the experiments.

Evaluating the methods, DynGraph-BERT surpassed the second-best competitor by more than 3% on the Ohsumed and DBLP datasets and by more than 20% on the Snippets dataset. Even in larger datasets such as Snippets and DBLP, DynGraph-BERT consistently achieved superior performance, with an advantage of over 10% compared to the other methods.

Numerically, DynGraph-BERT shows a higher classification accuracy in all datasets tested. We performed a significance test using a t-test with a p-value < 0.05 to compare our proposal with the state-of-the-art STHGN. The results indicate that we can reject the null hypothesis that DynGraph-BERT and STHGN have similar results with p < 0.05 in four out of five datasets. Statistical analysis was conducted for six methods with 50 paired experiments, applying 10 for each dataset. We removed the HGAT because we were unable to run it due to lack of memory.

Therefore, these results indicate that the proposal has higher accuracy in all datasets tested. When comparing DynGraph-BERT and STHGN using the t-test, we can reject the null hypothesis in four out of five datasets, showing superior results of the proposal.

6. Ablation

This section explores approaches to assess stability and identify key components contributing to model performance. We conducted three experiments: analyzing performance with varying numbers of training examples, evaluating performance by method components, and examining hyperparameter variations.

6.1. Training Sample Size Variation

We first evaluated the model’s performance on five datasets with different training sample sizes. Following the experimental setup from [28], we varied the number of examples to 5%, 10%, and 20%, in addition to those already reported in Table 4, and plot the respective accuracies in Figure 5. The results for the other methods were also extracted from [28]. This study examines the impact of data quantity on the generalization capability of the model.

As shown in Figure 5, the proposal (black lines with stars) yielded higher accuracy in all tested datasets in all training size percentages. A notable result is observed on the Ohsumed dataset, where increasing the training examples from 2.5% to 20% led to a 22-point accuracy improvement. Meanwhile, other methods failed to surpass the model’s performance achieved with only 5% of the training data.

6.2. DynGraph-BERT Components

To evaluate the impact of different components in our model, we conducted experiments with specific variations, removing or adjusting each component separately using the same train size reported in Table 2. The average results from 10 runs are reported in Table 5. The variations and components are as follows:

• BERT: The accuracy of the BERT model significantly decreased when dealing with a small number of samples per class.

• model v1: This experiment examined the BERT model enhanced with augmented data. Although this data augmentation contributed to model improvement, the interrelationship between datasets yielded superior results compared to subsequent versions of the models.

• model v2: The model was evaluated without data augmentation, using only the original data. It was observed that the DynGraph-BERT model experienced a performance drop, suggesting that augmented data helps with the model’s regularization and robustness.

• model v3: Noisy data were removed during training. The results were quite similar to those of the full method, suggesting that the Aug. Noise is the element with less contribution to the model’s accuracy.

• model v4: The label propagation at test time (L.P.T.T.) was absenced from DynGraph-BERT. The results show a slight loss in accuracy. This indicates that label propagation in testing remains an advantageous factor in capturing relationships between data, including test data.

• model v5: Data augmentation was performed without additional labeling of the augmented data. Here, the performance was close to that of the full method, showing that this component can be omitted, allowing more unlabeled data to be included while maintaining accuracy.

• model v6: In this case, we evaluated the impact of the dynamic graph. We froze the BERT model, resulting in the same graph across all training epochs.

The most effective and contributing component was the graph construction and GNN processing, followed by augmented data. The label propagation at test time yielded a modest improvement in overall average performance. Labeling the augmented data or adding noise provided minimal improvement. Together, these components make the model more robust than BERT, even when using labeled augmented data.

6.3. Hyperparameter Evaluation

To understand the influence of hyperparameters on the method, we evaluated the minimum similarity level and the maximum number of k neighbors in experiments with the Snippets dataset. The results, averaged over 10 runs of the DynGRaph-BERT, are shown in Figure 6a,b. Figure 6c shows the variation of fixed values $\lambda$ without the Lambda Learner mechanism, indicating a slight performance improvement when $\lambda$ is closer to 1 or greater than 0.5. In the studies presented in Figure 6, the algorithm demonstrates high stability, with no significant changes in accuracy when varying the hyperparameter values.

7. Discussion

DynGraph-BERT demonstrated superior performance compared to the models examined in this work for semi-supervised text classification, particularly across the five datasets evaluated. The combination of GCN [31] and BERT [8] in DynGraph-BERT deepened the relationships established by the graphs, resulting in substantial improvements in prediction rates. While models like RoBERTa [41], XLNet [43], and MPNet [44] may provide additional advances, the embeddings generated by transformers are notably effective at capturing semantic patterns, even in short sentences [57]. DynGraph-BERT becomes more competitive by integrating these embeddings and models, especially in low-data scenarios. Including unlabeled and augmented data further enhanced the performance of our technique, reducing the dependency on labeled data.

Regarding graph construction, DynGraph-BERT differs from other text classification methods, such as Text-GCN [10], InducT-GCN [30], and BertGCN [2], which use words as nodes and form heterogeneous graphs. New words cannot be inserted into the graph, limiting the adaptability of these models. Moreover, these methods are constrained to static graphs, preventing embeddings from adapting to new contexts or relationships between nodes. DynGraph-BERT, on the other hand, constructs graphs in batches, expanding its applicability to real-world scenarios. This batch-based construction resembles the advantages observed in Cluster-GCN [36], which also leverages batch-based training for computational efficiency.

DynGraph-BERT demonstrated strong performance in low-resource scenarios, where only a few labeled nodes per class are available. While its runtime efficiency decreased with smaller text sizes, it remained more robust than baseline methods when increasing the number of labeled nodes, as shown in Figure 5. When dealing with longer texts, within BERT’s maximum input length, DynGraph-BERT became even more advantageous due to its dynamic graph adaptation.

In multi-class classification scenarios, our model effectively separates different classes and groups similar ones. Table 4 highlights improvements of 22 and 3.7 percentage points in datasets with 8 and 23 classes (Snippets and Ohsumed), which are more complex than the 10-point gain observed in AGNews, a dataset with only 4 classes. This suggests that DynGraph-BERT is particularly well suited for tasks with diverse class distributions and complex relationships between labels.

The impact of example ordering during training remains an open question in our approach. The TCIL [58] method in incremental learning suggests that structured ordering can improve performance, which we plan to explore in future work.

A challenge for DynGraph-BERT is handling graphs with numerous nodes, which is a difficulty also encountered by other graph-based methods [59], with HGAT [27] serving as an example in this domain. To alleviate this limitation, we manipulated the graph using the FAISS library [60] to optimize memory usage and computational performance.

The most costly operation is the graph building procedure; it has a complexity of $O\left(\right|D_{train}{|}^{2}d)$, where d is the dimension of embedding. Using FAISS, this complexity goes down to $O\left(\right|D_{train}|log|D_{train}\left|\right)$, and BERT has $O\left(BL(n^{2}d+n{d}^2)\right)$, where B is the batch size, n is the max number of tokens in sentences, and L is the number of layers. The operation to insert nodes in the graph is $O\left(Bkd\right)$, with k being the number max of neighbors. Therefore, the proposal has complexity to train for each epoch: $O\left(\right|D_{train}|log|D_{train}\left|\right)+|D_{train}|/B\ast (O\left(BL(n^{2}d+n{d}^2)\right)+O\left(Bkd\right))$. The TextGCN has the complexity time: $O\left(\right(|D_{train}+D_{test}|+\mu )d)$, where $\mu$ is the vocabulary size [61]. For the STHGN, the complexity is $O\left(L\right|D_{train}|F{d}^{\prime }+L|E\left|d^{\prime 2}\right)$, where F is the original feature dimension, $d^{\prime }=d/M$, and M is the number of attention heads [28]. Other works did not report the complexity time of their models.

Another limitation of DynGraph-BERT is its dependency on the test set size. If the test set grows at the same rate as the training set, graph-based methods that create nodes for documents, such as TextGCN, HGAT, and BertGCN, may have problems with scalability. In our method, this could lead to higher memory consumption. However, the DynGraph-BERT remains consistent with increasing batch sizes.

To mitigate this problem, we will explore adaptive strategies for selecting specific subsets of the test set and inserting them into the graph as future research. This could help in two aspects: keep or improve accuracy while reducing runtime and memory overhead.

8. Conclusions

We present DynGraph-BERT, an inductive semi-supervised model that combines language models and graph neural networks through dynamic graph construction. Experiments have shown that the proposed approach outperformed state-of-the-art methods, especially in scenarios with few labeled examples, showing greater stability and generalization capacity.

Our analysis revealed that the graph structure is essential for performance, being the main factor for improving text classification. Furthermore, the use of data augmentation proved to be fundamental to increase the robustness of the model, while label propagation in the test contributed to refine the predictions without introducing direct dependence on the test data. But, the addition of noise in the embeddings had a reduced impact, suggesting that the gains of DynGraph-BERT come from the dynamic graph integration between BERT and GNN.

Author Contributions

Conceptualization, E.L.S.P. and E.T.M.; Methodology, E.L.S.P. and E.T.M.; Software, E.L.S.P.; Validation, E.L.S.P., M.C.d.S., J.d.A.S. and E.T.M.; Formal analysis, E.L.S.P., M.C.d.S., J.d.A.S. and E.T.M.; Investigation, E.L.S.P., J.d.A.S. and E.T.M.; Resources, E.L.S.P. and E.T.M.; Writing—Original Draft, E.L.S.P.; Writing—Review and Editing, E.L.S.P., M.C.d.S., J.d.A.S. and E.T.M.; Visualization, E.L.S.P., M.C.d.S., J.d.A.S. and E.T.M.; Supervision, E.L.S.P., M.C.d.S., J.d.A.S. and E.T.M.; Project Administration, E.L.S.P. and E.T.M.; Funding Acquisition, E.L.S.P. and E.T.M. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The original data presented in the study are openly available at https://github.com/yao8839836/text_gcn, https://pytorch.org/text/0.12.0/datasets.html#ag-news, https://github.com/FKarl/short-text-classification, accessed on 12 February 2025.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Overview of the DynGraph-BERT training process for a single epoch.

Figure 2. Illustration of the dynamic graph generation for each batch during training and testing. The nodes and cylinders colors indicate the link between them. Best view in color.

Figure 3. Overview of the DynGraph-BERT training process across epochs. Colors in the graph represent node identification.

Figure 4. Testing procedure of DynGraph-BERT: label propagation at test time.

Figure 5. Accuracy results for training data increments.

Figure 6. Plots for ablation study. (a) Maximum number of neighbors for graph construction on the Snippets dataset. (b) Minimum similarity for graph construction on the Snippets dataset. (c) Variation of [Forumla omitted. See PDF.] in the model’s loss function for the Ohsumed and AGNews datasets.

Table 1

Examples of text generated using the augmentation techniques applied in DynGraph-BERT.

Augmenter	Text
Original	The weather is beautiful today.
Swap	The climate is gorgeous today.
Omitting	The weather is today.
Typos	The weatger is biutyful today.
Change order	The weather is today beautiful.

Table 2

Summary statistics of datasets.

Dataset	# Train	# Per Class	%	# Val	# Test	# Class	Avg. Len.
Ohsumed	207	9	2.80	1000	6193	23	135.82
R8	80	10	1.04	1000	6594	8	65.72
AGNews	40	10	0.50	1000	6960	4	26.85
Snippets	64	8	0.52	1000	11,276	8	17.73
DBLP	120	20	0.50	1000	22,880	6	8.51

The symbol # indicates number of samples or number of.

Table 3

Hyperparameters.

Hyperparameters	Value
hidden size	256
embedding dimension size	768 (BERT)
aug. data	2
batch size	16
dropout	0.66
number of layers	2
epochs	20
learning rate BERT	1 $\times {10}^{-5}$
learning rate GNN	1 $\times {10}^{-3}$

Table 4

Classification accuracy on the test set based on the training percentages indicated in the parenthesis at the table heading. The best results and second best are in bold and underlined, respectively. The asterisk (*) indicates a significant difference (p < 0.05) with STHGN using a t-test.

Method	Graph Type	Ohsumed (2.8%)	R8 (1.04%)	AGNews (0.5%)	Snippets (0.52%)	DBLP (0.5%)
CNN-rand	No graph	34.07±0.57	79.52±1.07	39.04±3.74	42.41±2.47	36.94±1.40
CNN-pretrain	No graph	22.77±0.56	72.41±1.33	33.00±1.61	52.56±0.95	28.64±0.67
TextGCN	Heterogeneous	28.77±0.39	76.15±0.45	56.81±1.48	35.80±0.59	40.01±0.58
HGT	Heterogeneous	42.44±0.38	80.88±1.10	31.52±2.66	45.42±0.94	43.07±0.45
HGAT	Heterogeneous	39.42±1.26	86.45±1.33	63.40±4.97	OOM	OOM
STHGN	Heterogeneous	44.64±0.56	90.20±0.73	73.41±2.29	55.81±1.49	46.39±0.81
DynGraph-BERT	Homogeneous	48.34 *±1.54	90.93 ±1.71	83.56 *±1.97	77.87 *±3.90	58.96 *±2.01

OOM is out of memory.

Table 5

Table with ablation study.

	Aug. Graph	Dynamic Graph	Aug. Train	L.P.T.T.	Aug. Noise	Ohsumed	AGNews
BERT	-	-		-		9.53	52.12
model v1	-	-	✓	-		37.30	81.79
model v2		✓		✓	✓	43.12	81.98
model v3	✓	✓	✓	✓		47.31	83.46
model v4	✓	✓	✓		✓	44.06	80.37
model v5	✓	✓		✓	✓	45.69	81.97
model v6	✓		✓	✓	✓	29.20	82.45
proposal	✓	✓	✓	✓	✓	48.34	83.56

Notes: Aug. Graph means augmented graph. Aug. Train means training data were augmented. L.P.T.T. means label propagation at test time. The bold values represent the best results for the dataset.

References

1. Li, H.; Yan, Y.; Wang, S.; Liu, J.; Cui, Y. Text classification on heterogeneous information network via enhanced GCN and knowledge. Neural Comput. Appl.; 2023; 35, pp. 14911-14927. [DOI: https://dx.doi.org/10.1007/s00521-023-08494-0]

2. Lin, Y.; Meng, Y.; Sun, X.; Han, Q.; Kuang, K.; Li, J.; Wu, F. BertGCN: Transductive Text Classification by Combining GNN and BERT. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021; Online Event, 1–6 August 2021; Zong, C.; Xia, F.; Li, W.; Navigli, R. Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 1456-1462. [DOI: https://dx.doi.org/10.18653/v1/2021.findings-acl.126]

3. Li, M.; Xie, Y.; Yang, W.; Chen, S. Multistream BertGCN for Sentiment Classification Based on Cross-Document Learning. Quantum Eng.; 2023; 2023, 3668960. [DOI: https://dx.doi.org/10.1155/2023/3668960]

4. Xue, B.; Zhu, C.; Wang, X.; Zhu, W. An Integration Model for Text Classification Using Graph Convolutional Network and BERT. J. Phys. Conf. Ser.; 2021; 2137, 012052. [DOI: https://dx.doi.org/10.1088/1742-6596/2137/1/012052]

5. Yang, X.; Yan, M.; Pan, S.; Ye, X.; Fan, D. Simple and Efficient Heterogeneous Graph Neural Network. Proc. AAAI Conf. Artif. Intell.; 2023; 37, pp. 10816-10824. [DOI: https://dx.doi.org/10.1609/aaai.v37i9.26283]

6. Zhao, J.; Wang, X.; Shi, C.; Hu, B.; Song, G.; Ye, Y. Heterogeneous Graph Structure Learning for Graph Neural Networks. Proc. AAAI Conf. Artif. Intell.; 2021; 35, pp. 4697-4705. [DOI: https://dx.doi.org/10.1609/aaai.v35i5.16600]

7. Bing, R.; Yuan, G.; Zhu, M.; Meng, F.; Ma, H.; Qiao, S. Heterogeneous graph neural networks analysis: A survey of techniques, evaluations and applications. Artif. Intell. Rev.; 2023; 56, pp. 8003-8042. [DOI: https://dx.doi.org/10.1007/s10462-022-10375-2]

8. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Minneapolis, MN, USA, 2–7 June 2019; Burstein, J.; Doran, C.; Solorio, T. Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4171-4186. [DOI: https://dx.doi.org/10.18653/v1/N19-1423]

9. Chen, J.; Tam, D.; Raffel, C.; Bansal, M.; Yang, D. An empirical survey of data augmentation for limited data learning in nlp. Trans. Assoc. Comput. Linguist.; 2023; 11, pp. 191-211. [DOI: https://dx.doi.org/10.1162/tacl_a_00542]

10. Yao, L.; Mao, C.; Luo, Y. Graph Convolutional Networks for Text Classification. Proc. AAAI Conf. Artif. Intell.; 2019; 33, pp. 7370-7377. [DOI: https://dx.doi.org/10.1609/aaai.v33i01.33017370]

11. Wang, K.; Ding, Y.; Han, S.C. Graph neural networks for text classification: A survey. Artif. Intell. Rev.; 2024; 57, 190. [DOI: https://dx.doi.org/10.1007/s10462-024-10808-0]

12. Duarte, J.M.; Berton, L. A review of semi-supervised learning for text classification. Artif. Intell. Rev.; 2023; 56, pp. 9401-9469. [DOI: https://dx.doi.org/10.1007/s10462-023-10393-8] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36743267]

13. Zhang, G.; Hu, Z.; Wen, G.; Ma, J.; Zhu, X. Dynamic graph convolutional networks by semi-supervised contrastive learning. Pattern Recognit.; 2023; 139, 109486. [DOI: https://dx.doi.org/10.1016/j.patcog.2023.109486]

14. Bao, P.; Li, J.; Yan, R.; Liu, Z. Dynamic Graph Contrastive Learning via Maximize Temporal Consistency. Pattern Recognit.; 2024; 148, 110144. [DOI: https://dx.doi.org/10.1016/j.patcog.2023.110144]

15. Zou, M.; Gan, Z.; Wang, Y.; Zhang, J.; Sui, D.; Guan, C.; Leng, S. UniG-Encoder: A universal feature encoder for graph and hypergraph node classification. Pattern Recognit.; 2024; 147, 110115. [DOI: https://dx.doi.org/10.1016/j.patcog.2023.110115]

16. Munir, M.; Avery, W.; Rahman, M.M.; Marculescu, R. GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Los Alamitos, CA, USA, 17–21 June 2024; pp. 6118-6127. [DOI: https://dx.doi.org/10.1109/CVPR52733.2024.00585]

17. Chen, Y.; Wu, L.; Zaki, M. Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings. Proceedings of the Advances in Neural Information Processing Systems; Online, 6–12 December 2020; Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; Lin, H. Curran Associates, Inc.: Red Hook, NY, USA, 2020; Volume 33, pp. 19314-19326.

18. Sankar, A.; Wu, Y.; Gou, L.; Zhang, W.; Yang, H. Dynamic Graph Representation Learning via Self-Attention Networks. arXiv; 2019; arXiv: 1812.09430

19. Wang, Z.; Wang, C.; Zhang, H.; Duan, Z.; Zhou, M.; Chen, B. Learning Dynamic Hierarchical Topic Graph with Graph Convolutional Network for Document Classification. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics; Online, 26–28 August 2020; Chiappa, S.; Calandra, R. PMLR: Cambridge, MA, USA, Volume 108, pp. 3959-3969.

20. Piao, Y.; Lee, S.; Lee, D.; Kim, S. Sparse Structure Learning via Graph Neural Networks for Inductive Document Classification. Proc. AAAI Conf. Artif. Intell.; 2022; 36, pp. 11165-11173. [DOI: https://dx.doi.org/10.1609/aaai.v36i10.21366]

21. Gasteiger, J.; Bojchevski, A.; Günnemann, S. Combining Neural Networks with Personalized PageRank for Classification on Graphs. Proceedings of the International Conference on Learning Representations; New Orleans, LA, USA, 6–9 May 2019.

22. Sun, Y.; Wang, X.; Liu, Z.; Miller, J.; Efros, A.A.; Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. Proceedings of the 37th International Conference on Machine Learning, JMLR.org, ICML’20; Virtual Event, 13–18 July 2020.

23. Niu, S.; Wu, J.; Zhang, Y.; Chen, Y.; Zheng, S.; Zhao, P.; Tan, M. Efficient Test-Time Model Adaptation without Forgetting. Proceedings of the 39th International Conference on Machine Learning; Baltimore, MD, USA, 17–23 July 2022; PMLR: Cambridge, MA, USA, Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; Sabato, S. Volume 162, pp. 16888-16905.

24. Jin, W.; Zhao, T.; Ding, J.; Liu, Y.; Tang, J.; Shah, N. Empowering Graph Representation Learning with Test-Time Graph Transformation. Proceedings of the Eleventh International Conference on Learning Representations; Kigali, Rwanda, 1–5 May 2023.

25. Ju, M.; Zhao, T.; Yu, W.; Shah, N.; Ye, Y. GraphPatcher: Mitigating Degree Bias for Graph Neural Networks via Test-time Augmentation. Proceedings of the Thirty-Seventh Conference on Neural Information Processing Systems; New Orleans, LA, USA, 10–16 December 2023.

26. Hu, Z.; Dong, Y.; Wang, K.; Sun, Y. Heterogeneous Graph Transformer. Proceedings of the Web Conference 2020; New York, NY, USA, 7–12 February 2020; pp. 2704-2710. [DOI: https://dx.doi.org/10.1145/3366423.3380027]

27. Yang, T.; Hu, L.; Shi, C.; Ji, H.; Li, X.; Nie, L. HGAT: Heterogeneous Graph Attention Networks for Semi-supervised Short Text Classification. ACM Trans. Inf. Syst.; 2021; 39, pp. 1-29. [DOI: https://dx.doi.org/10.1145/3450352]

28. Cao, M.; Yuan, J.; Yu, H.; Zhang, B.; Wang, C. Self-supervised short text classification with heterogeneous graph neural networks. Expert Syst.; 2023; 40, e13249. [DOI: https://dx.doi.org/10.1111/exsy.13249]

29. Hua, J.; Sun, D.; Hu, Y.; Wang, J.; Feng, S.; Wang, Z. Heterogeneous Graph-Convolution-Network-Based Short-Text Classification. Appl. Sci.; 2024; 14, 2279. [DOI: https://dx.doi.org/10.3390/app14062279]

30. Wang, K.; Han, S.C.; Poon, J. InducT-GCN: Inductive Graph Convolutional Networks for Text Classification. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR); Los Alamitos, CA, USA, 21–25 August 2022; pp. 1243-1249. [DOI: https://dx.doi.org/10.1109/ICPR56361.2022.9956075]

31. Welling, M.; Kipf, T.N. Semi-supervised classification with graph convolutional networks. Proceedings of the International Conference on Learning Representations (ICLR 2017); Toulon, France, 24–26 April 2016.

32. Zhou, J.; Cui, G.; Hu, S.; Zhang, Z.; Yang, C.; Liu, Z.; Wang, L.; Li, C.; Sun, M. Graph neural networks: A review of methods and applications. AI Open; 2020; 1, pp. 57-81. [DOI: https://dx.doi.org/10.1016/j.aiopen.2021.01.001]

33. Chen, M.; Huang, C.; Xia, L.; Wei, W.; Xu, Y.; Luo, R. Heterogeneous Graph Contrastive Learning for Recommendation. Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining; New York, NY, USA, 27 February–3 March 2023; pp. 544-552. [DOI: https://dx.doi.org/10.1145/3539597.3570484]

34. Fu, D.; Hua, Z.; Xie, Y.; Fang, J.; Zhang, S.; Sancak, K.; Wu, H.; Malevich, A.; He, J.; Long, B. VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections. Proceedings of the Twelfth International Conference on Learning Representations; Vienna, Austria, 7–11 May 2024.

35. Hamilton, W.; Ying, Z.; Leskovec, J. Inductive Representation Learning on Large Graphs. Proceedings of the Advances in Neural Information Processing Systems; Long Beach, CA, USA, 4–9 December 2017; Curran Associates, Inc.: Red Hook, NY, USA, 2017; Volume 30.

36. Chiang, W.L.; Liu, X.; Si, S.; Li, Y.; Bengio, S.; Hsieh, C.J. Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; New York, NY, USA, 4–8 August 2019; pp. 257-266. [DOI: https://dx.doi.org/10.1145/3292500.3330925]

37. Alon, U.; Yahav, E. On the Bottleneck of Graph Neural Networks and its Practical Implications. Proceedings of the International Conference on Learning Representations; Vienna, Austria, 4 May 2021.

38. Fan, X.; Gong, M.; Wu, Y.; Qin, A.K.; Xie, Y. Propagation Enhanced Neural Message Passing for Graph Representation Learning. IEEE Trans. Knowl. Data Eng.; 2023; 35, pp. 1952-1964. [DOI: https://dx.doi.org/10.1109/TKDE.2021.3102964]

39. Pellegrini, G.; Tibo, A.; Frasconi, P.; Passerini, A.; Jaeger, M. Learning Aggregation Functions. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence Organization; Montreal, QC, Canada, 19–27 August 2021; pp. 2892-2898.

40. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems; Long Beach, CA, USA, 4–9 December 2017; Curran Associates, Inc.: Red Hook, NY, USA, 2017; Volume 30.

41. Zhuang, L.; Wayne, L.; Ya, S.; Jun, Z. A Robustly Optimized BERT Pre-training Approach with Post-training. Proceedings of the 20th Chinese National Conference on Computational Linguistics; Huhhot, China, 13–15 August 2021; Li, S.; Sun, M.; Liu, Y.; Wu, H.; Liu, K.; Che, W.; He, S.; Rao, G. Springer: Cham, Switzerland, 2021; pp. 1218-1227.

42. He, P.; Gao, J.; Chen, W. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. Proceedings of the Eleventh International Conference on Learning Representations; Kigali, Rwanda, 1–5 May 2023.

43. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.R.; Le, Q.V. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings of the Advances in Neural Information Processing Systems; Vancouver, BC, Canada, 8–14 December 2019; Wallach, H.; Larochelle, H.; Beygelzimer, A.; d’Alché-Buc, F.; Fox, E.; Garnett, R. Curran Associates, Inc.: Red Hook, NY, USA, 2019; Volume 32.

44. Song, K.; Tan, X.; Qin, T.; Lu, J.; Liu, T.Y. MPNet: Masked and permuted pre-training for language understanding. Proceedings of the 34th International Conference on Neural Information Processing Systems; Red Hook, NY, USA, 6–12 December 2020.

45. Warner, B.; Chaffin, A.; Clavié, B.; Weller, O.; Hallström, O.; Taghadouini, S.; Gallagher, A.; Biswas, R.; Ladhak, F.; Aarsen, T. et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv; 2024; arXiv: 2412.13663

46. Ma, E. NLP Augmentation. 2019; Available online: https://github.com/makcedward/nlpaug (accessed on 12 February 2025).

47. Pellicer, L.F.A.O.; Ferreira, T.M.; Costa, A.H.R. Data augmentation techniques in natural language processing. Appl. Soft Comput.; 2023; 132, 109803. [DOI: https://dx.doi.org/10.1016/j.asoc.2022.109803]

48. Li, B.; Hou, Y.; Che, W. Data augmentation approaches in natural language processing: A survey. AI Open; 2022; 3, pp. 71-90. [DOI: https://dx.doi.org/10.1016/j.aiopen.2022.03.001]

49. Wei, J.; Zou, K. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Hong Kong, China, 3–7 November 2019; pp. 6382-6388.

50. Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning; Virtual, 13–18 July 2020; PMLR: Cambridge, MA, USA, Daumé III, H.; Singh, A. Volume 119, pp. 1597-1607.

51. Galal, O.; Abdel-Gawad, A.H.; Farouk, M. Rethinking of BERT sentence embedding for text classification. Neural Comput. Appl.; 2024; 36, pp. 20245-20258. [DOI: https://dx.doi.org/10.1007/s00521-024-10212-3]

52. Moschitti, A.; Basili, R. Complex Linguistic Features for Text Classification: A Comprehensive Study. Proceedings of the Advances in Information Retrieval; Sunderland, UK, 5–7 April 2004; McDonald, S.; Tait, J. Springer: Berlin/Heidelberg, Germany, 2004; pp. 181-196.

53. Phan, X.H.; Nguyen, L.M.; Horiguchi, S. Learning to classify short and sparse text & web with hidden topics from large-scale data collections. Proceedings of the 17th International Conference on World Wide Web; New York, NY, USA, 21 April 2008; pp. 91-100. [DOI: https://dx.doi.org/10.1145/1367497.1367510]

54. Zhang, D.; Xu, H.; Su, Z.; Xu, Y. Chinese comments sentiment classification based on word2vec and SVMperf. Expert Syst. Appl.; 2015; 42, pp. 1857-1863. [DOI: https://dx.doi.org/10.1016/j.eswa.2014.09.011]

55. Tang, J.; Zhang, J.; Yao, L.; Li, J.; Zhang, L.; Su, Z. ArnetMiner: Extraction and mining of academic social networks. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; New York, NY, USA, 24–27 August 2008; pp. 990-998. [DOI: https://dx.doi.org/10.1145/1401890.1402008]

56. Kim, Y. Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); Doha, Qatar, 25–29 October 2014; Moschitti, A.; Pang, B.; Daelemans, W. Association for Computational Linguistics: Stroudsburg, PA, USA, 2014; pp. 1746-1751. [DOI: https://dx.doi.org/10.3115/v1/D14-1181]

57. Karl, F.; Scherp, A. Transformers are Short-Text Classifiers. Proceedings of the Machine Learning and Knowledge Extraction: 7th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2023; Benevento, Italy, 29 August–1 September 2023; Springer: Berlin/Heidelberg, Germany, 2023; pp. 103-122. [DOI: https://dx.doi.org/10.1007/978-3-031-40837-3_7]

58. Huang, B.; Chen, Z.; Zhou, P.; Chen, J.; Wu, Z. Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning. Proc. AAAI Conf. Artif. Intell.; 2023; 37, pp. 908-916. [DOI: https://dx.doi.org/10.1609/aaai.v37i1.25170]

59. Taha, K.; Yoo, P.D.; Yeun, C.; Taha, A. Text Classification: A Review, Empirical, and Experimental Evaluation. arXiv; 2024; arXiv: 2401.12982

60. Johnson, J.; Douze, M.; Jégou, H. Billion-Scale Similarity Search with GPUs. IEEE Trans. Big Data; 2021; 7, pp. 535-547. [DOI: https://dx.doi.org/10.1109/TBDATA.2019.2921572]

61. Ragesh, R.; Sellamanickam, S.; Iyer, A.; Bairi, R.; Lingam, V. HeteGCN: Heterogeneous Graph Convolutional Networks for Text Classification. Proceedings of the 14th ACM International Conference on Web Search and Data Mining; New York, NY, USA, February 2021; pp. 860-868. [DOI: https://dx.doi.org/10.1145/3437963.3441746]