Exploring Named Entity Recognition via MacBERT-BiGRU and Global Pointer with Self-Attention

Abstract

Named Entity Recognition (NER) is a fundamental task in natural language processing that aims to identify and categorize named entities within unstructured text. In recent years, with the development of deep learning techniques, pre-trained language models have been widely used in NER tasks. However, these models still face limitations in terms of their scalability and adaptability, especially when dealing with complex linguistic phenomena such as nested entities and long-range dependencies. To address these challenges, we propose the MacBERT-BiGRU-Self Attention-Global Pointer (MB-GAP) model, which integrates MacBERT for deep semantic understanding, BiGRU for rich contextual information, self-attention for focusing on relevant parts of the input, and a global pointer mechanism for precise entity boundary detection. By optimizing the number of attention heads and global pointer heads, our model achieves an effective balance between complexity and performance. Extensive experiments on benchmark datasets, including ResumeNER, CLUENER2020, and SCHOLAT-School, demonstrate significant improvements over baseline models.

1. Introduction

Named Entity Recognition (NER), a fundamental problem in natural language processing, aims to identify and categorize named entities within unstructured text. This technique is essential for applications such as knowledge graph construction [1], information retrieval [2], and question-answering systems [3]. Traditional NER methods use handcrafted features and rule-based systems. Although initially effective, they struggled with scalability and adaptability. Additionally, early machine learning-based NER required extensive word-level annotations on large datasets, making the process labor-intensive [4].

Following the significant progress made in deep learning, particularly the introduction of pre-trained language models like BERT [5], the NER models have achieved great success. Zheng et al. [6] utilized Word2Vec to pre-train word embeddings, which were then used in the encoding layer to improve NER performance. Additionally, GloVe [7], ELMo [8] and RoBERTa [9] are commonly employed as pre-trained word embeddings in NER tasks. By leveraging large-scale text corpora, these models have demonstrated a remarkable ability to capture rich semantic representations. However, there are still three main challenges in NER tasks: (1) Conventional NER models often struggle to capture diverse semantic information and long-distance dependencies due to their limited pre-training or reliance on handcrafted features. This limitation hinders the model’s ability to understand the complex meanings and relationships within the text. (2) The accurate identification of nested entities, where one entity is embedded within another, remains a significant challenge for many NER models. This limitation often leads to reduced accuracy and increased complexity in entity recognition tasks. (3) Achieving an optimal balance between model complexity and computational efficiency is still a challenge.

To address these challenges, in this study, we propose a model named MB-GAP (MacBERT-BiGRU-Self Attention-Global Pointer), which consists of semantic representation, sequence modeling, and label decoding. Specifically, our model leverages MacBERT, a pre-trained language model specifically designed for Chinese text, to provide deep semantic understanding. This is complemented by a Bidirectional GRU (BiGRU) networks, which capture rich contextual information in both the forward and backward directions. Additionally, a Self-Attention mechanism is incorporated to dynamically adjust the weights of different positions, allowing the model to focus on the most relevant parts of the input sequence. For precise entity boundary detection, we employ a Global Pointer layer, which treats named entities as a whole and connects every pair of characters in the text, forming an upper triangular matrix that represents all potential candidate entities. This combination of MacBERT, BiGRU, Self-Attention, and Global Pointer mechanisms in the MB-GAP model enables a comprehensive and effective approach to NER tasks.

The contributions of our paper are summarized as follows:

• •. We propose a novel MB-GAP (MacBERT-BiGRU-Self Attention-Global Pointer) model, which combines MacBERT for deep semantic understanding, BiGRU for rich contextual information, Self-Attention for focusing on relevant parts, and the Global Pointer for precise entity boundary detection.

• •. We conduct a thorough investigation into the impact of variations in the number of attention and global pointer heads on the performance of the MB-GAP model.

• •. We perform extensive experiments on benchmark datasets, including ResumeNER, CLUENER2020, and SCHOLAT-School, to validate the effectiveness of our proposed model. The results demonstrate that our approach significantly outperforms baseline models.

2. Related Work

Named Entity Recognition (NER) is a technique used to detect real-world named entities from natural language text and classify them into predefined categories [10]. In this section, we mainly review the following aspects of the related works: named entity recognition, and the use of pre-trained models for text representation.

2.1. Named Entity Recognition

In recent years, numerous efforts have been made to improve the performance of NER from various perspectives, including rule-based, unsupervised, supervised (feature-based), and deep learning-based named entity recognition. These models can be categorized into four categories, as follows:

• •. Rule-based models consist of expert-created rules, primarily derived from linguistic and domain-specific knowledge. One notable work is that of Collins et al. [11], who presented an approach to named entity classification using a small set of seed rules and unlabeled data. Their method leverages the redundancy in the data, where both the spelling of a name and its context can indicate its type. Hanisch et al. [12] designed a system called ProMiner that uses a preprocessed synonym dictionary to identify potential names appearing in biomedical texts. By matching the identified text with identifiers in biomedical databases, the system resolves entity disambiguation problems to find the most relevant matches. Other examples of rule-based NER systems include FASTUS [13], SAR [14], and Facile [15].

• •. Unsupervised learning models are applied to unlabeled data, commonly using association and clustering techniques. Etzioni et al. [16] introduced the KNOWITALL system, which focuses on extracting information without hand-labeled training data. They detailed three unsupervised named entity recognition methods—pattern learning, subclass extraction, and list extraction—to enhance recall and extraction rates while maintaining high precision. Zhang et al. [17] presented an unsupervised approach to biomedical named entity recognition in the clinical and biological texts. This method avoids reliance on handcrafted rules or annotated data by using a noun phrase chunker and an inverse document frequency-based filter to extract candidate entities.

• •. Supervised learning models use labeled data to train models, leveraging features such as word embeddings and syntactic cues to accurately identify and classify named entities. Zhou et al. [18] applied the Hidden Markov Model (HMM) to the task of named entity recognition, integrating four types of internal and external features: simple deterministic internal features of words, internal semantic features of important leading words, internal features from a place name dictionary, and external features of macro-context. Conditional Random Fields (CRF) [19,20]-based NER introduced prior information to the labeling sequence, enhancing the recognition of entity sequences.

• •. Deep learning models use neural networks to automatically learn features from raw text, achieving high accuracy by capturing complex patterns and contextual information. Typical approaches to deep learning-based NER involve RNN, CNN, and hybrid systems that integrate these techniques with other supervised learning methods, such as SVM. Huang et al. [21] used various LSTM-based sequence labeling models for named entity recognition tasks. Lample et al. [22] proposed a BiLSTM-CRF to extract character-level representations of words. This model views text sentences as sequences of characters and computes word entity labels based on character computations. Zhang et al. [23] introduced an improved model called Lattice LSTM, which uses a lattice structure to encode input character sequences and all possible latent words. Compared to character-based methods, this model explicitly utilizes the sequential information between words.

2.2. Pre-Trained Models for Text Representation

Pre-trained language models have made significant progress in numerous NLP tasks. Owing to their strong semantic representation capabilities, pre-trained language models have shown great success in NER tasks as well. Li et al. [24] redefined the NER task as a machine reading comprehension (MRC) problem, addressing it by fine-tuning the BERT model. This approach also resolves the issue of entity overlap in nested entity recognition. As the BERT model consists of stacked multi-layer Transformer encoders, and CRF is the most commonly used decoder for output label sequences, the BERT-CRF architecture is the most prevalent in deep learning-based named entity recognition, for instance, BERT-IDCNN-CRF [25].

3. Problem Definition

Named entities generally refer to entities in the text that have specific meanings or referential properties, which can be considered instances of a concept in the real world, such as personal names, organizational names and location names. For example, “university” is a concept, whereas “Tsinghua University” is an instance of a “university”, also known as an entity. NER involves identifying these entity terms and their types from the text.

To be more specific, NER is a natural language processing subtask that involves the use of sequence labeling and segment classification to identify entities in a text and categorize them into predefined categories, as illustrated in Figure 1. Formally, this subtask can be expressed as follows: Given a sequence of tokens $s=[s_1,s_2,\dots ,s_n]$ and a predefined set of entity types T, NER outputs a list of triplets in the form $(p_s,p_e,t)$. Here, $p_s,p_e\in [1,n]$ represent the start and end positions of the recognized named entity in the original sequence, and $t\in T$ is the type of entity.

4. Method

4.1. Overview of MB-GAP

The MB-GAP (MacBERT-BiGRU-Self Attention-Global Pointer) model consists of four main components: MacBERT, BiGRU, Self-Attention, and Global Pointer. The interaction between these components is designed to leverage their unique strengths, providing a comprehensive approach to named entity recognition (NER). Figure 2 shows the architecture of the MB-GAP model, illustrating the flow between each component.

• •. The MacBERT [26] module is an improved BERT with a novel MLM as the correction pre-training task, which mitigates the discrepancies in pre-training and fine-tuning. MacBERT uses a large amount of Chinese text data for pre-training to obtain a deep semantic understanding of the Chinese language. This layer helps to convert text into semantic representations for downstream tasks.

• •. The BiGRU module is located at the output end of the text representation layer, serving as an improved form of RNN that can capture richer contextual information in the text sequence data. Through the BiGRU layer, our model can better capture long-distance lexical information, thereby enhancing the ability to extract text features.

• •. The self-attention module is responsible for modeling the input text sequence. The self-attention layer allows for the model to establish relationships between characters or words with different positions in the sequence, focusing on the semantic connections at various locations. This helps to enhance the model’s understanding of long-distance dependencies and the internal structure of the sequences, improving its performance in determining entity boundaries and extracting semantic information.

• •. The Global Pointer module aims to decode and recognize named entities within the text. The Global Pointer leverages the textual features and contextual information learned from the preceding encoding network. Using a pointer network, the model delineates the start and end positions of named entities within the text, facilitating the identification and tagging of these entities.

4.2. Semantic Representation Layer

In common NLP tasks, the semantic representation layer usually employs pre-trained language models such as ELMo, BERT, or GPT to derive the semantic representations of textual data. The semantic information learned during the pre-training phase from the vast corpora of text data is utilized to enhance model performance in downstream tasks. To alleviate inconsistencies between the pre-training and fine-tuning tasks for downstream applications, we utilize MacBERT in our semantic representation layer, as depicted in Figure 3. The improvement in MacBERT involves altering the masking strategy in BERT’s Masked Language Model (MLM) pre-training task. A Corrective Masked Language Model (MLM as correction, Mac) is used for the pre-training task, which is more suitable for Chinese characters and words.

One of the problem with BERT’s MLM masking strategy is that the [MASK] token used to replace masked words does not pass over to downstream tasks and is only used during the pre-training phase. This fixed token reduces the model’s ability to handle the context around the token during downstream fine-tuning tasks, leading to a mismatch between the pre-training and fine-tuning. MacBERT addresses this problem by replacing masked words with words that have similar semantics instead of using the fixed [MASK] token. MacBERT uses Word2Vec to find similar replacement words. When no similar replacement word is found, a random word from the vocabulary is used to ensure consistency between the pre-training and downstream tasks. Furthermore, MacBERT uses Whole Word Masking (WWM) to ensure that a complete word is masked and employs N-gram masking techniques to identify multi-word phrases with complete semantics.

4.3. Sequence Modeling Layer

The sequence modeling layer consists of a BiGRU network and a self-attention mechanism. The BiGRU network captures contextual information through its bidirectional structure, while the self-attention mechanism dynamically adjusts the weights of different positions, allowing the model to focus on the more important parts of the sequence. The combination of these two components enables a more comprehensive modeling of text sequences, providing more accurate information for subsequent modules.

4.3.1. Bidirectional Gated Recurrent Unit Network

BiGRU is a bidirectional recurrent neural network whose recurrent structure units are called GRUs (Gated Recurrent Units). The main components of the GRU are as follows:

• •. Update Gate: Controls the amount of information from the previous hidden state that is passed to the current time step;

• •. Reset Gate: Determines how much of a hidden state should be forgotten;

• •. Current Memory: Computes the new memory based on the current input, the previous hidden state, and the reset gate information;

• •. State Update: Calculates the current state based on the update gate and the current memory.

The calculation of the update gate in the GRU at time step t in layer l is shown as follows:

(1)$U_t^l=\sigma ((W_u·A_{t-1}^l)+(S_u·A_{t-1}^l)+B_r)$

where $A_t^l$ represents the output at time step t in layer l, $\sigma$ is the Sigmoid activation function, $W_u$ denotes the weight matrix of the update gate in the previous layer, $S_u$ represents the weight matrix of the update gate from the previous time step, and $B_r$ is the bias matrix of the update gate.

Similarly, the calculation of the reset gate in the GRU at time step t in layer l is shown as follows:

(2)$R_t^l=\sigma ((W_r·A_{t-1}^l)+(S_r·A_{t-1}^l)+B_r)$

where $W_r$ represents the weight matrix of the reset gate in the previous layer, $S_r$ denotes the weight matrix of the reset gate from the previous time step, and $B_r$ is the bias matrix of the reset gate.

The output of the GRU is defined as follows:

(3)$\begin{array}{cc}\hfill & {\tilde{A}}_t^l=Tanh\left(\left(W·A_t^{l-1}\right)+\left(R_t^l\odot \left(S·A_{t-1}^l\right)\right)+B\right)\hfill \\ \hfill & A_t^l=U_t^l\odot A_{t-1}^l+\left(1-U_t^l\right)\odot {\tilde{A}}_t^l\hfill \end{array}$

where $A_t^l$ represents the output at time step t in layer l, Tanh is the activation function, W denotes the weight matrix in the previous layer, S represents the weight matrix with the previous time step, and B is the bias matrix.

When the value of the reset gate approaches 0, the influence of the output from the previous time step will be “forgotten”. The update gate also controls the amount of information from the previous time step that is retained through the values of the weight matrix. Therefore, each unit in the network can capture important past information while forgetting unimportant past information.

4.3.2. Scaled Dot-Product Multi-Head Self-Attention

The attention mechanism calculates the relevance of the relationship between every pair of characters in the text sequence at different positions. Our model uses the scaled dot-product multi-head self-attention mechanism. We introduce the attention mechanism, self-attention, and multi-head attention as follows.

Attention Mechanism: The input of the attention mechanism typically consists of three components: the Query, the Key, and the Value. The Query is a tensor that represents the current position or information of the model, obtained through a learnable neural network, which is usually the output of the final hidden layer at each sequence step of the preceding neural network. The Key is a tensor that represents different positions or information in the input sequence. The Value is a tensor representing the information associated with each Key, which is determined by the model’s task and context.

The output of the attention mechanism consists of attention weights and attention output. Attention weights are tensors representing the importance weights allocated by the model between the queries and keys. The attention mechanism calculates the similarity between queries and keys, and then normalizes them using Softmax to obtain the attention weights. The attention output is a tensor representing the weighted sum computed based on queries, keys, and values.

Firstly, our model use the attention function to compute the similarity between a query and each key as follows:

(4)$S_i=AttnF(Q,K_i),i=1,2,...,l$

where Q is the query, K is the key, $AttF$ is the attention function, S represents the similarity between the query and the key, and l denotes the length of the target sequence.

Then, we normalize the similarity to obtain the attention weights W.

(5)$W_i=Softmax\left(S\right)={\displaystyle \frac{e^{S_i}}{{\sum }_{j=1}^l{e}^{S_j}}},i=1,2,...,l$

Finally, we obtain the attention output O.

(6)$O_i=W_{i}V$

We use the Scaled Dot-Product Attention mechanism [27] as the attention mechanism in the MB-GAP model. The equation is shown as follows:

(7)$AttnF(Q,K)={\displaystyle \frac{Q{K}^{\top }}{\sqrt{di{m}_K}}}$

The computation of the Scaled Dot-Product Attention mechanism is shown as follows:

(8)$\begin{array}{cc}\hfill W& =Softmax(AttnF(Q,K\left)\right),\hfill \\ \hfill O& =Attention(Q,K,V)=WV\hfill \\ \hfill & =Softmax\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{{dim}_K}}}\right)V\hfill \end{array}$

Self-Attention Mechanism

When the three inputs to the attention mechanism are the same tensor, a variant of the attention mechanism, called Self-Attention, is formed. The design of the self-attention mechanism reduces dependence on external information, allowing our model to focus more on the internal structure of the sequence when processing it, and thereby enhancing our model’s ability to extract intrinsic data structures. This can be represented as follows:

(9)$SelfAttention\left(A\right)=Attention(A,A,A)$

where the query (Q), key (K), and value (V) are all the same tensor A.

Multi-Head Attention Mechanism Multi-Head Attention is used to enhance the model’s ability to capture the global dependencies within sequences. Multi-Head Attention can simultaneously learn multiple sets of attention weights, each of which is referred to as a head. This allows the model to focus on different parts of the input sequence in different attention subspaces, thereby better capturing complex patterns within the sequence. The computation process of Multi-Head Attention is as follows:

• •. Linear transformations are applied to the input Q, K, and V to map them to different attention subspaces:

  (10)$\begin{array}{cc}\hfill Q_i& =U_{Q_i}·Q\hfill \\ \hfill K_i& =U_{Ki}·K\hfill \\ \hfill V_i& =U_{Vi}·V\hfill \end{array}$

  where $i=1,2,...,\mathit{num}\_\mathit{heads}$ means the number of attention heads, while $U_{Q_i},U_{K_i},$ and $U_{V_i}$ are the weight matrices.

• •. Multi-head attention is calculated, and different attention mechanisms are applied to each attention head:

  (11)$O_i=Attention(Q_i,K_i,V_i)$

• •. Multi-head attention outputs are combined, and the outputs of each attention head are concatenated:

  (12)$O=Concat(O_1,O_2,...,O_{num\_heads})$

• •. A linear transformation is applied to the concatenated multi-head outputs to obtain the final output:

  (13)$MultiHead(Q,K,V)=U_O·O$

  where $U_O$ means a linear transformation.

4.4. Label Decoding Layer

Inspired by the global pointer proposed by Su et al. [28], our Global Pointer marks entity boundaries. Unlike traditional methods, it views entities as a whole. Consider the text, “Apple is an American computer and consumer electronics company”. Our method forms an upper triangular matrix by connecting character pairs, with each element representing a candidate entity (for length $n=9$, there are $n(n+1)/2=45$ candidates). We use the context feature vector c, (from the encoder-modeled text) and for each entity type t, the $t_{th}$ global pointer head carries out linear transformations and uses the dot products on c to obtain scores for the subsequences. After threshold binarization, scores above the threshold are considered entities (type determined by the pointer). However, it has a limitation: entity length is not considered. Therefore, we use rotary position encoding (RoPE) to add position information, enhancing boundary detection accuracy.

Subsequence Score: The context feature vector $c=[c_1,c_2,...,c_n]$ is obtained from the text sequence after being modeled by the encoder and serves as the input for the global pointer layer. For the t-th type of entity, the t-th global pointer head applies two different linear transformations to c, and then performs a dot product operation on the resulting vectors. This is shown as follows:

(14)$\begin{array}{cc}\hfill Q_t& =U_{Q_t}·c\hfill \\ \hfill K_t& =U_{K_t}·c\hfill \end{array}$

(15)$S_t(i,j)=Q_t\left[i\right]·K_t{\left[j\right]}^{\top }$

where $U_Q$ and $U_K$ represent the weights of the linear transformations, Q and K are the sequence vectors of length n after mapping, and $M\left[i\right]$ denotes the i-th element of the sequence vector M. The $S(i,j)$ indicates the score (logits) of the continuous subsequence from i to j in the original text sequence, which is a named entity of type t.

Finally, threshold binarization is performed on the upper triangular matrix, where units less than the threshold are set to 0, and units greater than the threshold are set to 1. Units with value 1 represent a recognized named entity, and the type of the named entity can be determined according to the global pointer.

Position Information: The global pointer predicts the starting and ending boundaries of entities together, without consider the length of the entities. This means that the calculation process for the global pointer remains the same. Although this approach avoids the issue of inconsistent boundary recognition, it also results in reduced sensitivity to entity length. For instance, when two short entities of the same type are in close proximity, the absence of entity length information can lead the global pointer to mistakenly identify the head of the first entity and the tail of the second entity as a single valid entity. However, such a prediction is considered correct by the model, since the global pointer accurately marks the head and tail of the entities along with their types. To address this problem, we use rotary position encoding (Rotary Position Embedding, RoPE) [29] to add sequential position information to the global pointer.

5. Experiments

In this section, we outline the experimental setup and discuss the results. We perform an extensive evaluation to highlight the effectiveness of our MB-GAP model. Specifically, we seek to address the following research questions: (RQ1) How does our proposed MB-GAP model perform against the baselines? (RQ2) How do the different components (i.e., BiGRU, Rotary Position Embedding, and the attention module) influence the performance of the proposed MB-GAP model? (RQ3) How does the number of attention heads and global pointer heads influence the performance of the MB-GAP model?

5.1. Datasets

To validate the effectiveness of the proposed model, we conducted experiments on three different kinds of NER datasets: ResumeNER [23], CLUENER2020 https://github.com/CLUEbenchmark/CLUENER2020 (accessed on 10 August 2024), and the SCHOLAT-School https://www.scholat.com/scholarsearch-school.html (accessed on 10 August 2024). For evaluation metrics, we used the micro-averaged precision, recall, and F1-score. The details of these datasets are as follows:

• •. ResumeNER is a Sina Finance resume dataset, containing the summary of 1027 executive resumes from various companies in China. The training set consists of 3822 sentences with 13,438 named entities; the validation set contains 464 sentences with 1497 named entities; and the test set includes 478 sentences with 1630 named entities.

• •. CLUENER2020 dataset is extracted from THUCTC, a Chinese text classification dataset from Tsinghua University. After the extraction, the dataset consists of 8598 sentences in the training set, containing 19,239 named entities; 2150 sentences in the validation set, containing 4731 named entities; and 1343 sentences in the test set, containing 3072 named entities.

• •. SCHOLAT-School dataset contains 2756 named entities, including the full and abbreviated names of Chinese universities, which serve as the question targets. The sample questions encompass 14 different inquiry intentions, each with several question formats. An example of the data is shown in Table 1. Each sample includes a question, the manually annotated subject entity and its type, and the type of relationship implied in the question.

5.2. Experimental Settings

The hyperparameters were determined through an empirical search of the values listed in Table 2. We implemented all the approaches using PyTorch 1.13.0 with CUDA 11.7, based on Python 3.10.9. All comparison models used the pre-trained BERT-large. For the MB-GAP model, we used a batch size of 32 to balance computational efficiency and model stability. We used AdamW as the optimizer, with a learning rate of 5 × 10⁻⁵, which was found to provide the best convergence during our initial experiments. Each experiment was run for 10 epochs with a sequence length of 256 to ensure that the model could capture long-range dependencies. The dropout rate was set to 0.1 across all models to prevent overfitting. The MB-GAP model used three BiGRU layers with 1024 units, 32 attention heads, and 64 global pointer heads.To run our experiments, we used two NVIDIA V100 GPUs with 32 GB memory to handle our model.

5.3. Baselines

We selected the baseline models to provide a comprehensive comparison across three types of architectures: (1) traditional models, such as CRF modules; (2) Transformer-based models, which enhance the pre-trained MacBERT framework with additional components such as CRF and BiLSTM; and (3) state-of-the-art models specifically designed for handling complex NER tasks. The baselines used for comparison are as follows:

• •. BiLSTM-CRF [21]. This is a well-established traditional model for named entity recognition. We included it to compare the performance of our model against a basic sequential model that uses long-short-term memory and conditional random fields.

• •. MacBERT-Softmax. This model introduces a Softmax layer in MacBERT [26] to convert the semantic information from the text context into a probability distribution of entity categories, thereby constructing an efficient and accurate NER model.

• •. MacBERT-CRF. This is a BERT-CRF [30] model obtained by adding a CRF decoding layer, which provides more prior information about the label sequence and can improve the accuracy of the predicted labels.

• •. MacBERT-BiLSTM-CRF: This model introduces a layer of bidirectional LSTM to the MacBERT-CRF framework to enhance performance.

• •. NFLAT [31]. NFLAT (Non-Flat Lattice Transformer) is an improved version of FLAT [32] for Chinese named entity recognition. NFLAT introduces a new lexical enhancement method named InterFormer, which effectively reduces computational and memory costs by constructing a non-flat lattice.

• •. CPLL [33]. This model combines prior and posterior confidences and utilizes the Expectation-Maximization algorithm to learn the true posterior probability and confidence scores.

• •. SLRBC [34]. SLRBC combines SoftLexicon and RoBERTa for better representation and uses BiLSTM-CRF to capture context.

• •. LLM-DER [35]. LLM-DER uses large language models to generate relationship lists and evaluate entities for domain-specific NER.

• •. FGAT [36]. FGAT uses graph attention networks to interactively fuse character and lexical information, enhancing feature representation with techniques like residual connections and layer normalization.

• •. MVT [37]. MVT defines four views to capture different token interaction structures, constructs view-aware visible matrices, and designs three variants to fuse multi-view features.

6. Results

6.1. Overall Performance (RQ1)

We conducted a comprehensive evaluation of the MB-GAP model by comparing it with a diverse set of baselines, including traditional methods, transformer-based approaches, and recent models designed to handle nested entities and long-range dependencies. Our goal was to provide a rigorous and thorough assessment of MB-GAP’s performance. To ensure a fair comparison, all models were trained under the same conditions, using the same dataset splits, training algorithms, and hyperparameter tuning strategies. This allowed us to isolate the impact of the model architectures and techniques used in MB-GAP. The overall experimental results are presented in Table 3, Table 4 and Table 5. Our proposed MB-GAP model consistently exhibited a superior performance across all three datasets.

On ResumeNER, MB-GAP attained the highest precision, outperforming the second-best model (MVT) by 1.16%, and achieved an F1-score of 96.25%, which is 0.47% higher than MVT’s. However, its recall was lower than some baselines. This may be because the model focuses on precision, and is therefore more conservative in identifying entities. It needs stronger evidence, reducing false positives and ensuring better precision but potentially missing some entities, causing lower recall. For CLUENER2020, similar trends were observed. MB-GAP has the best precision (1.24% over MVT) and an F1-score of 80.81% (slightly better than MVT). The recall, while still competitive, is again slightly lower than that of a few baselines. A possible reason for this could be the same as in the ResumeNER case. The model’s focus on precise identification might lead to a more conservative stance in terms of entity recognition, resulting in a trade-off between recall and the other metrics. On SCHOLAT-School dataset, MB-GAP, stands out with a perfect F1-score of 96.81%, surpassing that of FGAT and MVT. It also exhibits high precision and a relatively good recall. In this case, our proposed model appears to achieve a better balance between precision and recall compared to the other datasets. This could imply that the nature and characteristics of the SCHOLAT-School dataset are more favorable for the MB-GAP model’s entity identification mechanism. The distribution of entity types and the context used for this dataset might be more in line with the model’s capabilities, enabling it to accurately identify entities without a significant compromise in recall.

To further validate the significance of these results, we conducted multiple runs of the experiments. We trained and evaluated each model five times with different random initializations. We then performed a t-test to compare the means and standard deviations of the precision, recall, and F1-score of MB-GAP with those of the baseline models. The results of the t-test confirm that the improvements in precision and F1-score achieved by MB-GAP are statistically significant ($p<0.05$) across all three datasets.

In conclusion, the observed trade-off in the MB-GAP model’s performance, characterized by a higher precision and F1-score but a slightly lower recall in some instances, can be mainly ascribed to its design, which emphasizes precision. This leads to a more conservative stance in terms of entity identification, which brings benefits in terms of reduced false positives but also has drawbacks in its potentially overlooking some entities. Additionally, the different behaviors of the three datasets suggest that the dataset properties play a role in determining how effectively the model can balance these factors.

6.2. Ablation Study (RQ2)

To answer RQ2, we conducted an ablation study, as presented in Table 6. “w/o” means the module was excluded from MB-GAP. We compared different variants of our MB-GAP model to evaluate the impact of the MacBERT, BiGRU, the attention, and the Rotary Position Embedding (RoPE) module on the performance of MB-GAP. From Table 6, we can observe the following:

• •. MacBERT: Excluding the MacBERT module led to a significant drop in all metrics (F1 score decreased by 1.42%). This demonstrates the importance of using a robust pre-trained language model for capturing complex semantic patterns, which are critical for NER tasks. MacBERT’s deep semantic representations allowed the model to understand nuanced language features effectively.

• •. BiGRU: Removing the BiGRU module resulted in a slight decrease in the F1 score (by 0.54%), indicating that while BiGRU enhances the model’s contextual understanding, its contribution is less significant than that of MacBERT. The bidirectional processing of sequences by BiGRU helps capture the dependencies across tokens, complementing the self-attention mechanism.

• •. Attention Mechanism: The absence of the attention module led to a higher recall but lower precision. This suggests that while attention enhances the model’s focus on relevant parts of the sequence, its removal forces the model to rely more on broader input features, potentially increasing recall at the expense of precision. This trade-off highlights the need for a balanced attention configuration.

• •. RoPE (Rotary Position Embedding): Excluding RoPE resulted in the most substantial decline across all metrics (F1 score decreased by 1.70%). This indicates the critical role of positional embeddings in providing a structural context, which is essential for accurately identifying entity boundaries in sequential data.

• •. BiGRU and Attention Modules Excluded: When both BiGRU and Attention modules were excluded, the model’s performance declined across all metrics, although not as drastically as when MacBERT or RoPE were excluded. This suggests that while BiGRU and Attention enhance the model’s performance by improving sequence processing and focus, the core performance relies heavily on the robust semantic understanding provided by MacBERT and the structural context from RoPE.

• Impact of BiGRU Layer Exploration

To further investigate the impact of hyperparameter settings, we briefly explored the effect of varying the number of BiGRU layers and their dimensions. While this exploration was limited, preliminary observations indicated that increasing the number of BiGRU layers from one to three improved the F1 score slightly, suggesting enhanced sequence modeling capabilities. However, increasing the layers beyond three led to diminishing returns, likely due to the increased model complexity and the risk of overfitting.

Similarly, varying the hidden dimensions of the BiGRU layer suggested that the use of 1024 units provided the best balance between capturing contextual information and avoiding excessive computational overhead. We propose a more comprehensive tuning of these parameters as a direction for future research to better understand their impact on model performance.

• Key Contributions of RoPE and Components in MB-GAP

The MB-GAP model’s performance is driven by the interplay of its key components, with RoPE (Rotary Position Embedding) standing out as the most crucial. RoPE provides positional awareness, enabling the model to capture the relationships between tokens based on their relative positions in a sequence. This is essential for handling tasks involving overlapping or nested entities, where positional context defines the entity boundaries. Without RoPE, the model’s ability to distinguish between such entities significantly diminishes, leading to a notable drop in precision and recall. As presented in Figure 4, each component contributes uniquely to the model’s overall effectiveness. (1) RoPE: Identified as the most critical component, RoPE encodes the positional information within input sequences, ensuring the model captures the order and structure of the data. This encoding allows the model to represent the relative positions of tokens, a vital ability for sequence-based tasks such as NER. (2) MacBERT: This pre-trained language model provides the semantic foundation for MB-GAP by leveraging an extensive pre-training on large corpora. MacBERT delivers rich contextual embeddings that capture semantic nuances and syntactic structures, significantly enhancing the model’s ability to understand and generate natural language. (3) Attention Mechanism: By focusing dynamically on relevant parts of the input sequence, the attention layer enables the model to weigh token importance effectively. This prioritization enhances precision by emphasizing significant information while filtering out irrelevant details. (4) BiGRU: The bidirectional processing capability of BiGRU captures dependencies spanning across sequences, enriching contextual understanding and improving predictions. This dual-direction modeling complements the outputs from other components, further refining the model’s predictions.

6.3. Impact of Attention and Global Pointer Heads Number (RQ3)

The tuning of hyperparameters plays a crucial role in optimizing the model’s performance. In this section, we investigate the impact of the number of attention and global pointer heads on the performance of the MB-GAP model for named entity recognition (NER). Their effect on model performance is shown in the detailed results presented in Figure 5 and Figure 6.

Impact of Number of Attention Heads: As shown in Figure 5, we can observe that, with only one attention head, the model achieved a precision of 94.92%, a recall of 93.87%, and an F1 score of 94.39%. With eight attention heads, the performance slightly improved, suggesting that multiple attention heads allow the model to capture a broader range of features and dependencies. At 32 attention heads, the model performance peaked, with a precision of 96.70% and recall of 95.80%, and the highest F1 score of 96.25%. This indicates that 32 heads provide an optimal balance of feature capture and model complexity. When the attention heads increased to 64, a decline can be observed in precision and F1 score, as well as a slight decline in recall. We believe this is due the fact that to too many attention heads might introduce redundant or less useful features, potentially complicating the model without proportional benefits.

Impact of Number of Global Pointer Heads: As shown in Figure 6, the results demonstrate that, with eight global pointer heads, the model shows lower precision (84.79%) but high recall (95.26%), and an F1 score of 89.72%. When the number of heads increased to 16, precision improved to 95.79% and recall slightly improved, to 95.97%, yielding a high F1 score of 95.88%. With 32 heads, precision, recall, and decreased slightly. At 64 heads, both precision and F1 score were maximized, reaching 96.70 % and 96.25%, respectively. This configuration achieved the best balance and overall performance. However, increasing the number of global pointer heads to 96 resulted in a slight decrease in precision (95.33%) and recall (95.15%), with an F1 score of 95.24%. This reduction could be due to overfitting or an overcomplicated model.

The results from both Figure 5 and Figure 6 illustrate that tuning the number of attention and global pointer heads is crucial for optimizing the MB-GAP model’s performance. The optimal configurations are 32 attention heads and 64 global pointer heads. These settings strike a balance between capturing complex patterns and maintaining computational efficiency.

In conclusion, the experimental results clearly demonstrate the effectiveness of the MB-GAP model. The overall superior performance of our model underscores its robustness and the successful integration of its components: MacBERT, BiGRU, self-attention, and the global pointer mechanism.

7. Discussion

The MB-GAP model demonstrated a strong performance in named entity recognition (NER) tasks, achieving a notably higher precision compared to baseline models. The model’s precision, especially on datasets like ResumeNER and CLUENER2020, indicates that it effectively identifies entities with fewer false positives. However, the slightly lower recall suggests a trade-off, where the model may overlook some entities. This behavior may be attributed to overfitting, as indicated by the high precision but reduced recall. In this context, the model appears conservative in identifying entities, prioritizing confidence and reducing false positives at the cost of missing certain instances.

While we optimized hyperparameters such as the dropout rate through an empirical search, further evaluations using additional regularization techniques could be beneficial. Approaches like early stopping or k-fold cross-validation could help improve recall without compromising precision. These strategies are particularly important for datasets with complex, nested entities, where the model might struggle to generalize effectively. We propose these as potential directions for future work.

The architecture of MB-GAP, while effective, involves a high level of complexity due to its multi-layer BiGRU and attention mechanisms. This complexity might lead to diminishing returns with an increase in the number of attention heads, as observed in our experiments. While MB-GAP excels in capturing long-range dependencies and nested entities, the efficiency could be further improved for practical applications requiring faster processing.

Compared to previous approaches, the MB-GAP model’s robust performance and ability to handle complex NER tasks make it a strong candidate for further research and optimization.

8. Conclusions

In this paper, we presented the MacBERT-BiGRU-Self Attention-Global Pointer (MB-GAP) model, a novel solution for the NER task. Our model leverages MacBERT for deep semantic understanding, BiGRU for capturing rich contextual information, self-attention for focusing on relevant parts of the input, and a global pointer mechanism for precise entity boundary detection. The experimental results demonstrate that MB-GAP outperforms other baselines on benchmark datasets such as ResumeNER, CLUENER2020, and SCHOLAT-School.

The ablation study highlights the importance of MacBERT and RoPE. Despite its high precision, the lower recall shown on some datasets may suggest overfitting. Future work will explore the use of regularization and hyperparameter tuning, like BiGRU layers and dimensions, to enhance robustness and generalizability. Also, regarding its multilingual applicability, given its good Chinese performance, we will collect various language corpora, optimize the architecture, explore multilingual training and transfer learning, and build a testing platform to make MB-GAP more versatile for NER across different languages.

Author Contributions

All authors conceived of the paper. C.Y. drafted the initial manuscript. F.T., W.S. and J.L. developed the software and ran the analyses. R.L. and C.M. participated in the discussion of theories. C.S. supervised the study and edited the manuscript. All authors reviewed the article. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

Both datasets used in this research are publicly available and can be accessed via the following links: CLUENER2020 https://github.com/CLUEbenchmark/CLUENER2020 and SCHOLAT https://www.scholat.com/scholarsearch-school.html (both accessed on 10 August 2024).

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The introduction of the NER task.

Figure 2. The framework of the MB-GAP model.

Figure 3. The semantic representation layer of the MB-GAP model.

Figure 4. Precision, recall, and F1 score for different variants of MB-GAP.

Figure 5. Performance when using different numbers of attention heads.

Figure 6. Performance when using different numbers of global pointer heads.

Table 1

Examples from the SCHOLAT-School dataset.

Question	Subject Entity	Relationship Type	Corresponding Knowledge Graph Triple
Where is South China Normal University?	(South China Normal University, University)	Location	(South China Normal University, Location, Guangdong Province)
What is the college code of Nanchang University?	(Nanchang University, University)	College Code	(Nanchang University, College Code, 10403)
Does Peking University belong to Project 985 universities?	(Peking University, University)	Classification	(Peking University, Classification, Project 985)
When was Tsinghua founded?	(Tsinghua, University Abbreviation)	Establishment Date	(Tsinghua University, Establishment Date, 1911)

Table 2

Hyperparameters’ search space and the best values.

Parameter	Range	MB-GAP
max epochs	[10, 50]	10
optimizer	[SGD, AdamW]	AdamW
batch size	[4, 32, 64, 128]	32
learning rate	[$1\times {10}^{-6}$, $1\times {10}^{-4}$]	$5\times {10}^{-5}$
drop out	[0.1, 0.8]	0.1
BiGRU Units	[128, 1024]	1024
sequence length	[64, 256]	256
attention head	[1, 8, 32, 64]	32
global pointer head	[8, 16, 32, 64, 96]	64

Table 3

Experimental results for ResumeNER compared to baselines ($p<0.05$ under t-test), bold values indicate the best performance, while underlined values indicate the second-best performance.

Models	Precision (%)	Recall (%)	F1 (%)
BiLSTM-CRF	$94.41\pm 0.5$	$93.25\pm 0.7$	$93.83\pm 0.6$
MacBERT-Softmax	$94.93\pm 0.4$	$94.87\pm 0.5$	$94.90\pm 0.45$
MacBERT-CRF	$94.87\pm 0.35$	$\underline{95.98}\pm 0.4$	$95.42\pm 0.38$
MacBERT-BiLSTM-CRF	$95.27\pm 0.3$	$95.86\pm 0.35$	$95.56\pm 0.32$
NFLAT	$94.54\pm 0.45$	$95.52\pm 0.5$	$95.03\pm 0.48$
CPLL	-	-	96.07
SLRBC	$94.34\pm 0.4$	$95.61\pm 0.5$	$94.97\pm 0.45$
LLM-DER	-	-	93.94
FGAT	95.47 ± 0.3	95.84 ± 0.35	95.65 ± 0.32
MVT	95.54 ± 0.35	96.01 ± 0.4	95.78 ± 0.38
MB-GAP	96.70 ± 0.2	95.80 ± 0.3	96.25 ± 0.25

Table 4

Experimental results for CLUENER2020 compared to baselines ($p<0.05$ under t-test), bold values indicate the best performance, while underlined values indicate the second-best performance.

Models	Precision (%)	Recall (%)	F1 (%)
BiLSTM-CRF	73.73 ± 0.8	69.79 ± 1.0	71.71 ± 0.9
MacBERT-Softmax	79.16 ± 0.6	80.63 ± 0.7	79.89 ± 0.65
MacBERT-CRF	79.38 ± 0.5	80.44 ± 0.6	79.91 ± 0.55
MacBERT-BiLSTM-CRF	78.97 ± 0.4	82.00 ± 0.5	80.46 ± 0.45
NFLAT	76.38 ± 0.7	78.74 ± 0.8	77.54 ± 0.75
CPLL	-	-	-
SLRBC	75.60 ± 0.6	78.22 ± 0.7	76.89 ± 0.65
LLM-DER	-	-	-
FGAT	79.15 ± 0.4	81.86 ± 0.5	80.48 ± 0.45
MVT	81.21 ± 0.45	80.38 ± 0.5	80.79 ± 0.48
MB-GAP	82.45 ± 0.3	79.23 ± 0.4	80.81 ± 0.35

Table 5

Experimental results for SCHOLAT-School compared to baselines ($p<0.05$ under t-test), bold values indicate the best performance, while underlined values indicate the second-best performance.

Models	Precision (%)	Recall (%)	F1 (%)
BiLSTM-CRF	93.60 ± 0.4	93.60 ± 0.4	93.60 ± 0.4
MacBERT-Softmax	95.72 ± 0.3	95.21 ± 0.35	95.47 ± 0.32
MacBERT-CRF	95.58 ± 0.35	94.45 ± 0.4	95.01 ± 0.38
MacBERT-BiLSTM-CRF	95.63 ± 0.3	95.56 ± 0.35	95.59 ± 0.32
NFLAT	94.52 ± 0.4	94.74 ± 0.45	94.63 ± 0.42
CPLL	-	-	-
SLRBC	94.12 ± 0.35	94.98 ± 0.4	94.53 ± 0.38
LLM-DER	-	-	-
FGAT	96.63 ± 0.25	96.27 ± 0.3	95.97 ± 0.28
MVT	96.32 ± 0.3	96.48 ± 0.35	96.41 ± 0.32
MB-GAP	96.81 ± 0.15	96.81 ± 0.2	96.81 ± 0.18

Table 6

The precision recall and F1 score performance of different variants in our proposed MB-GAP, bold values indicate the best performance.

Model	MacBERT	BiGRU	Attention	RoPE	Precision (%)	Recall (%)	F1 Score (%)
w/o MacBERT	✗	✓	✓	✓	94.91	94.75	94.83
w/o BiGRU	✓	✗	✓	✓	95.63	95.80	95.71
w/o Attention	✓	✓	✗	✓	95.20	96.15	95.67
w/o RoPE	✓	✓	✓	✗	94.83	94.28	94.55
w/o BiGRU+Attention	✓	✗	✗	✓	95.17	95.56	95.36
MB-GAP	✓	✓	✓	✓	96.70	95.80	96.25

References

1. He, Q.; Wu, L.; Yin, Y.; Cai, H. Knowledge-Graph Augmented Word Representations for Named Entity Recognition. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence; New York, NY, USA, 7–12 February 2020; pp. 7919-7926.

2. Brandsen, A.; Verberne, S.; Lambers, K.; Wansleeben, M. Can BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology Domain. ACM J. Comput. Cult. Herit.; 2022; 15, pp. 51:1-51:18. [DOI: https://dx.doi.org/10.1145/3497842]

3. Yin, D.; Cheng, S.; Pan, B.; Qiao, Y.; Zhao, W.; Wang, D. Chinese named entity recognition based on knowledge based question answering system. Appl. Sci.; 2022; 12, 5373. [DOI: https://dx.doi.org/10.3390/app12115373]

4. Ma, Y.; Zhang, Y.; Sangaiah, A.K.; Yan, M.; Li, G.; Wang, T. Active Learning for Name Entity Recognition with External Knowledge. ACM Trans. Asian Low-Resour. Lang. Inf. Process.; 2023; [DOI: https://dx.doi.org/10.1145/3593023]

5. Devlin, J.; Chang, M.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL-HLT 2019; Minneapolis, MN, USA, 2–7 June 2019; Burstein, J.; Doran, C.; Solorio, T. Association for Computational Linguistics: Minneapolis, MN, USA, 2019; pp. 4171-4186.

6. Zheng, S.; Wang, F.; Bao, H.; Hao, Y.; Zhou, P.; Xu, B. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. Proceedings of the ACL 2017; Vancouver, BC, Canada, 30 July–4 August 2017; Barzilay, R.; Kan, M. Association for Computational Linguistics: Vancouver, BC, Canada, 2017; Volume 1: Long Papers pp. 1227-1236.

7. Pennington, J.; Socher, R.; Manning, C.D. Glove: Global Vectors for Word Representation. Proceedings of the EMNLP 2014; Doha, Qatar, 25–29 October 2014; Moschitti, A.; Pang, B.; Daelemans, W. A Meeting of SIGDAT, a Special Interest Group of the ACL ACL: Doha, Qatar, 2014; pp. 1532-1543.

8. Peters, M.E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep Contextualized Word Representations. Proceedings of the NAACL-HLT 2018; New Orleans, LA, USA, 1–6 June 2018; Walker, M.A.; Ji, H.; Stent, A. Association for Computational Linguistics: New Orleans, LA, USA, 2018; Volume 1 (Long Papers) pp. 2227-2237.

9. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv; 2019; arXiv: 1907.11692

10. Li, J.; Sun, A.; Han, J.; Li, C. A Survey on Deep Learning for Named Entity Recognition. IEEE Trans. Knowl. Data Eng.; 2022; 34, pp. 50-70. [DOI: https://dx.doi.org/10.1109/TKDE.2020.2981314]

11. Collins, M.; Singer, Y. Unsupervised Models for Named Entity Classification. Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP 1999; College Park, MD, USA, 21–22 June 1999; Fung, P.; Zhou, J. Association for Computational Linguistics: Baltimore, MD, USA, 1999.

12. Hanisch, D.; Fundel, K.; Mevissen, H.T.; Zimmer, R.; Fluck, J. ProMiner: Rule-based Protein and Gene Entity Recognition. BMC Bioinform.; 2005; 6, S14. [DOI: https://dx.doi.org/10.1186/1471-2105-6-S1-S14] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/15960826]

13. Appelt, D.E.; Hobbs, J.R.; Bear, J.; Israel, D.J.; Kameyama, M.; Martin, D.L.; Myers, K.L.; Tyson, M. SRI International FASTUS system: MUC-6 test results and analysis. Proceedings of the 6th Conference on Message Understanding, MUC 1995; Columbia, MD, USA, 6–8 November 1995; ACL: Baltimore, MD, USA, 1995; pp. 237-248.

14. Aone, C.; Halverson, L.; Hampton, T.; Ramos-Santacruz, M. SRA: Description of the IE2 System Used for MUC-7. Proceedings of the MUC 1998 Seventh Message Understanding Conference: Proceedings of a Conference; Fairfax, VA, USA, 29 April–1 May 1998.

15. Black, W.J.; Rinaldi, F.; Mowatt, D. FACILE: Description of the NE System Used for MUC-7. Proceedings of the MUC 1998 Seventh Message Understanding Conference: Proceedings of a Conference; Fairfax, VA, USA, 29 April–1 May 1998.

16. Etzioni, O.; Cafarella, M.; Downey, D.; Popescu, A.M.; Shaked, T.; Soderland, S.; Weld, D.S.; Yates, A. Unsupervised Named-entity Extraction from the Web: An Experimental Study. Artif. Intell.; 2005; 165, pp. 91-134. [DOI: https://dx.doi.org/10.1016/j.artint.2005.03.001]

17. Zhang, S.; Elhadad, N. Unsupervised Biomedical Named Entity Recognition: Experiments with Clinical and Biological Texts. J. Biomed. Inform.; 2013; 46, pp. 1088-1098. [DOI: https://dx.doi.org/10.1016/j.jbi.2013.08.004] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23954592]

18. Zhou, G.; Su, J. Named Entity Recognition using an HMM-based Chunk Tagger. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics; Philadelphia, PA, USA, 6–12 July 2002; pp. 473-480.

19. Lafferty, J.D.; McCallum, A.; Pereira, F.C.N. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. Proceedings of the 18th International Conference on Machine Learning; San Francisco, CA, USA, 28 June–1 July 2001; pp. 282-289.

20. Settles, B. Biomedical Named Entity Recognition using Conditional Random Fields and Rich Feature Sets. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications; Geneva, Switzerland, 28–29 August 2004; pp. 104-107.

21. Huang, Z.; Xu, W.; Yu, K. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv; 2015; arXiv: 1508.01991

22. Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; Dyer, C. Neural Architectures for Named Entity Recognition. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; San Diego, CA, USA, 12–17 June 2016; pp. 260-270.

23. Zhang, Y.; Yang, J. Chinese NER Using Lattice LSTM. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics; Melbourne, Australia, 15–20 July 2018; pp. 1554-1564.

24. Li, X.; Feng, J.; Meng, Y.; Han, Q.; Wu, F.; Li, J. A Unified MRC Framework for Named Entity Recognition. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; pp. 5849-5859.

25. Cai, X.; Sun, E.; Lei, J. Research on Application of Named Entity Recognition of Electronic Medical Records Based on BERT-IDCNN-CRF Model. Proceedings of the 6th International Conference on Graphics and Signal Processing; Chiba, Japan, 1–3 July 2022; pp. 80-85.

26. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Wang, S.; Hu, G. Revisiting Pre-trained Models for Chinese Natural Language Processing. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020; Online, 16–20 November 2020; pp. 657-668.

27. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.U.; Polosukhin, I. Attention is All you Need. Proceedings of the 31st International Conference on Neural Information Processing Systems; Long Beach, CA, USA, 4–9 December 2017; Volume 30, pp. 5998-6008.

28. Su, J.; Murtadha, A.; Pan, S.; Hou, J.; Sun, J.; Huang, W.; Wen, B.; Liu, Y. Global Pointer: Novel Efficient Span-based Approach for Named Entity Recognition. arXiv; 2022; arXiv: 2208.03054

29. Su, J.; Ahmed, M.; Lu, Y.; Pan, S.; Bo, W.; Liu, Y. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing; 2024; 568, 127063. [DOI: https://dx.doi.org/10.1016/j.neucom.2023.127063]

30. Wang, X.; Jiang, Y.; Bach, N.; Wang, T.; Huang, Z.; Huang, F.; Tu, K. Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing; Online, 1–6 August 2021; pp. 1800-1812.

31. Wu, S.; Song, X.; Feng, Z.; Wu, X. NFLAT: Non-Flat-Lattice Transformer for Chinese Named Entity Recognition. arXiv; 2022; arXiv: 2205.05832

32. Li, X.; Yan, H.; Qiu, X.; Huang, X. FLAT: Chinese NER Using Flat-Lattice Transformer. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; pp. 6836-6842.

33. Xiong, L.; Zhou, J.; Zhu, Q.; Wang, X.; Wu, Y.; Zhang, Q.; Gui, T.; Huang, X.; Ma, J.; Shan, Y. A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023; Toronto, ON, Canada, 9–14 July 2023; pp. 1375-1386.

34. Cui, X.; Yang, Y.; Li, D.; Qu, X.; Yao, L.; Luo, S.; Song, C. Fusion of SoftLexicon and RoBERTa for Purpose-Driven Electronic Medical Record Named Entity Recognition. Appl. Sci.; 2023; 13, 13296. [DOI: https://dx.doi.org/10.3390/app132413296]

35. Xiao, L.; Xu, Y.; Zhao, J. LLM-DER: A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domain. arXiv; 2024; arXiv: 2409.10077

36. Wang, Y.; Wang, Z.; Yu, H.; Wang, G.; Lei, D. The interactive fusion of characters and lexical information for Chinese named entity recognition. Artif. Intell. Rev.; 2024; 57, 258. [DOI: https://dx.doi.org/10.1007/s10462-024-10891-3]

37. Xiao, Y.; Ji, Z.; Li, J.; Han, M. MVT: Chinese NER Using Multi-View Transformer. IEEE/ACM Trans. Audio Speech Lang. Process.; 2024; 32, pp. 3656-3668. [DOI: https://dx.doi.org/10.1109/TASLP.2024.3426287]