Measuring the Inferential Values of Relations in Knowledge Graphs

Abstract

Knowledge graphs, as an important research direction in artificial intelligence, have been widely applied in many fields and tasks. The relations in knowledge graphs have explicit semantics and play a crucial role in knowledge completion and reasoning. Correctly measuring the inferential value of relations and identifying important relations in a knowledge graph can effectively improve the effectiveness of knowledge graphs in reasoning tasks. However, the existing methods primarily consider the connectivity and structural characteristics of relations, but neglect the semantics and the mutual influence of relations in reasoning tasks. This leads to truly valuable relations being difficult to fully utilize in long-chain reasoning. To address this problem, this work, inspired by information entropy and uncertainty-measurement methods in knowledge bases, proposes a method called Relation Importance Measurement based on Information Entropy (RIMIE) to measure the inferential value of relations in knowledge graphs. RIMIE considers the semantics of relations and the role of relations in reasoning. Specifically, based on the values of relations in logical chains, RIMIE partitions the logical sample set into multiple equivalence classes, and generates a knowledge structure for each relation. Correspondingly, to effectively measure the inferential values of relations in knowledge graphs, the concept of relation entropy is proposed, and it is calculated according to the knowledge structures. Finally, to objectively assess the effectiveness of RIMIE, a group of experiments are conducted, which compare the influences of the relations selected according to RIMIE and other patterns on the triple classifications by knowledge graph representation learning. The experimental results confirm what is claimed above.

1. Introduction

Knowledge graph (KG) is a kind of multi-relational graph, in which the relations contain rich semantic information. The relations in knowledge graphs are vital to effectiveness and availability for downstream tasks [1]. As shown in Figure 1, in a knowledge graph, different relations play different roles, and identifying important relations in the knowledge graph can benefit many reasoning applications such as search, recommendation, and question answering.

In most reasoning applications, every step in a logical or reasoning path is very important. When a path reaches an entity with multiple outgoing relations, choosing different relations can lead to completely different outcomes. For example, in Figure 1, in a just two-hop reasoning path, choosing the relation “final date” results in an entity unrelated to the correct answer. If such biases are placed in a longer chain of reasoning, in which each step affects subsequent reasoning, noises will be constantly amplified and the reasoning result will seriously deviate from the correct answers. It can be considered that different relations have different influences on the reasoning process, and identifying the inferential value of relations is conducive to the acquisition of inference answers.

In fact, as a kind of specific information carrier with knowledge semantics, the relations in knowledge graphs have uneven information values which have uneven impact on reasoning. As shown in Figure 1, the relation “plays for” connects player entities to club entities and describes key knowledge information about the player’s work, significantly affecting the completeness of knowledge graphs. On the other hand, the relation “birthday” describes personal information about a player and contains less critical information on the player’s work. Therefore, when an inference path is selected, it is essential to seriously consider the semantic characteristics and significance of different relations. However, most current multi-step reasoning approaches overlook this aspect. In this study, we believe that for long-chain logical reasoning, it is crucial to properly evaluate and identify the inferential values of different relations, which will contribute to completing the knowledge graph and improving its performance in downstream tasks.

A closely related issue for the inferential value of relations in knowledge graphs is the importance metric of relations. Most current methods of measuring relation importance focus on the topological features of graphs, using metrics such as the degree of relation or the degrees of tied nodes on a relation [2,3]. These methods primarily consider the connectivity and structural characteristics of relations, but neglect the semantics and the mutual influence of relations in reasoning tasks. And thus, they cannot solve the problem of long-chain logical reasoning discussed above. To fully consider and utilize the role of relations in reasoning, this work proposes a model of measuring the importance of relations based on information entropy (RIMIE), by which the information entropy of relations, defined as the relation entropy, is calculated while considering the role of relations in logical inference chains. The physical meaning of relation entropy is as follows: given a logical inference chain l and a relation r, how much does r contribute to inferring the correct answer, or in a certain step of reasoning, how much does it cost if r is ignored? Based on this physical meaning and idea, this study fully considers the reasoning information on the logical inference chains in knowledge graphs, where the importance of relations is measured effectively. Thereby, the inferential values of relations in KGs are revealed in this study, and a potential driving-force factor for the long-chain reasoning is found.

The remainder of this work is organized as follows. In Section 2, we review related work on the identification of important nodes and edges. Section 3 presents the main ideas and framework of our approach. Section 4 introduces the symbols and definitions used in this work, explains the concept of relation entropy, and the RIMIE method for computing relation entropy. In Section 5, we describe the triple classification experiment and analyze and discuss the experimental results, which demonstrate the effectiveness of the RIMIE. Finally, Section 6 summarizes the research content of this work.

2. Related Work

In our work, RIMIE measures the importance of the relations in knowledge graphs from the perspective of inferential value. Since knowledge graphs can be viewed as a special class of complex networks carrying semantics [4], we first discuss some related work on the classic importance measurements of nodes and edges in complex networks, and then the measures of knowledge importance from a semantic perspective are discussed.

In complex networks, the nodes and edges with significant influence are referred to as the key spreaders [5,6,7]. The identification of important spreaders [5] has significant implications in lots of real-world applications such as epidemic transmission [8], cybersecurity [9], bioinformatics [10], social networks [11,12], etc. In the field of complex networks, there are many classic algorithms for identifying important nodes and edges. Of these, the measurements of edge importance are mainly based on the node importance. They usually either use node importance directly to compute edge importance [13], or convert edges to nodes to compute their importance [14]. Therefore, in this section, we mainly discuss the measurement methods of node importance.

Those traditional methods are mainly built based on the topological characteristics of complex networks [2,3,15,16], such as the distributions of nodes, the features of paths, and the characteristics of adjacency matrices. The method of degree centrality (DC) [17] considers the nodes with higher degrees to have greater influence in the network than others, focusing on the local information in a network. The k-shell decomposition [18] determines the importance of nodes based on their position in a network, assuming that the nodes closer to the center have greater influence. The method of H-index [19] considers nodes with neighbors having higher degrees to be more important. In addition, some methods utilize the features of paths in networks to measure the importance of nodes [20]. Closeness centrality [21] evaluates the importance of a node based on its distance to other nodes, considering global information within the network. Betweenness centrality [22,23] calculates the importance of a node based on the number of shortest paths passing through it. There are also some methods employing the characteristics of adjacency matrices to mirror the importance of nodes, and the most typical of these are the methods based on the eigenvectors of adjacency matrices. For example, the method of eigenvector centrality [24] updates the representation of a node based on the representations of its neighbors; PageRank [25] updates the probability of visiting each node through random walks, where the probability represents the PageRank value of the node and its importance.

Besides the classic methods above, some other state-of-the-art algorithms of node importance have also been explored. For example, inspired by the law of universal gravitation, the Gravity Model (GM) algorithm [26] treats the degree of nodes as mass and the shortest paths between nodes as distance to compositely calculate the importance of nodes. Furthermore, ref. [27] improves the performance of the GM algorithm by reducing the computational complexity and noise, which may arise from long-distance interactions. Li and Pan [28] introduced the concept of structural entropy for analyzing the structured data such as graphs and networks. Structural entropy not only defines the dynamic complexity of networks but also detects the natural or real structures from noisy data in the network.

Since knowledge graphs can be viewed as a special class of complex networks, many methods for identifying important nodes in complex networks seems to be usable for knowledge graphs. However, in fact, most measures of node importance in complex networks mainly consider structural features and lack consideration of knowledge semantic information, making it difficult to directly apply to most knowledge graphs. For example, refs. [29,30] use Graph Neural Networks (GNNs) to predict node importance by utilizing both internal and external information from the knowledge graph, where the node importance is regarded as a predictable label but it has to require some rich external information. Another significant difference between knowledge graphs and the traditional complex networks is that the edges in knowledge graphs also carry more semantics, such as clear type labels or physical information [31], rather than merely representing the connections between nodes. Additionally, the relation types are usually much more than the node types. These phenomena indicate that the importance of relations in knowledge graphs have more specificity and should be paid more attention than that the traditional complex networks. Therefore, the traditional methods measuring edge importance [32], which are usually based on node importance, are no longer applicable to knowledge graphs.

Besides the structural importance of networks and graphs, one effort related to the importance identification of knowledge semantics is the attribute reduction in knowledge bases [33,34]. Attribute reduction [35] aims to identify the most valuable and meaningful attributes for some downstream tasks. This process shares many similarities with measuring the importance of relations in knowledge graphs. Attribute reduction considers the semantics of attributes and aims to identify the attributes that best represent the samples. These attributes play a decisive role in downstream tasks such as classification and clustering. The partition of knowledge structures can be used to identify the highly knowledgeable attributes, called decision attributes [36]. In addition, Li et al. [37] used multiple metrics such as information entropy, knowledge granularity, knowledge quantity, and rough entropy to quantify the amount of information or uncertainty contained in a knowledge structure. In those studies, attribute reduction only considers the feature information of a single sample. However, the inference tasks in knowledge graphs usually involve multiple entities and relations [38]. In other words, an inference process can be regarded as a composition of multiple triples, corresponding to multiple inference steps, and thus more semantic and structural characteristics of triples need to be considered.

3. The Main Idea of RIMIE

To effectively reveal the inferential values of relations in KGs, a new measurement of relation importance needs to be explored. This work addresses the issue that the existing methods only consider the structural features of graphs, while ignoring the semantics in KGs and the roles of relations in inferences. Thus, inspired by uncertainty-measurement methods in knowledge bases, this study proposes a relation importance measurement based on information entropy to mirror the inferential values of relations in KGs.

The main idea of RIMIE is to measure the inferential value of a relation in terms of how well the relation removes the inference uncertainty in an inference chain, and this value is called relation entropy. In a knowledge graph, an inference process can be viewed as an inference chain composed of a series of fact triplets (i.e., some entities and relations). That is, a story consists of a series of facts. Each fact contributes differently to the advancement of a story line, and finding those key facts is more conducive to discovering the truth of the story. In this process, the relations carrying semantics are the core elements of those key facts, and can be regarded as the key attributes of a story or an inference process. The corresponding relation values can be characteristic of relations in an inference chain, such as some physical or numeric semantics on relations, the credibility of relations, the relative position of relations in a chain, the occurrence frequency of a specific relation in a chain, etc. We believe the changes in the values of different relations have different influences on an inference chain. That is, as shown in Figure 2, if a change in the value of a relation can heavily influence the result of reasoning on a chain, this relation can be considered to be high-inference value. In an inference task, catching the high-inference value relations facilitates the whole reasoning process. It is as if, in a complex case, it is a small number of key clues, not all the information, that is really decisive for the detection of the case.

4. The Proposed Approach

This section first introduces some symbol definitions and concepts RIMIE, and then describes the concept of the logical relation information table in knowledge graphs and the calculation of relation entropy in RIMIE.

4.1. Some Preliminaries of RIMIE

Definition 1. 

Information Table . In a knowledge base, an information table is a tuple $\left(U,A\right)$, where $U=\{x_1,x_2,\dots ,x_n\}$ represents a non-empty finite sample set of inference chains based on knowledge information, and $A=\{a_1,a_2,\dots ,a_n\}$ is a non-empty finite attribute set on inference chains.

There exists a mapping from the sample space to the attribute value space, denoted as $f_a:U\to V_a$. For each sample $x\in U$, there exists a value $f_a\left(x\right)\in V_a$, where $V_a=\left\{f_a\left(x\right)\right|x\in U\}$ is called the codomain of a.

Remark 1. 

According to Definition 1, in a knowledge graph G, an inference chain can be viewed as a subgraph or chain composed of a series of fact triplets or 4-tuples, for example, (head, relation: value, tail) or (head, relation, tail, value). The relations can be regarded as the attributes of an inference chain.

According to Definition 1 and Remark 1, in Figure 3, there are three inference chains, $x_1$, $x_2$ and $x_3$, the shaded part, where “conflict” is a relation in the inference chains.

Definition 2. 

Equivalence Class. The sample set U can be divided by an attribute a into some disjoint subsets, which are called a group of equivalence classes. Every equivalence class on a is expressed by a set as the following

(1)${\left[x\right]}_a=\left\{y\right|f_a\left(y\right)=f_a\left(x\right),y\in U\},$

where ${\left[x\right]}_a$ is a set of samples, the elements in which have equal values on attribute a. ${\left[x\right]}_a$ is also called the equivalence class obtained by the partitioning with attribute a. Different equivalence classes are disjoint, and the union of all equivalence classes forms a partition of the sample set U.

Definition 3. 

Knowledge Structure  [37]. A partition of the sample set U on attribute a, which is composed of the equivalence classes such as ${\left[x\right]}_a$, is called a knowledge structure χ on attribute a, defined as the following

(2) $\chi \left(a\right)={\displaystyle \frac{U}{a}}=\left\{{\left[x\right]}_a\right|x\in U,a\in A\}.$

For example in Figure 3, a knowledge structure on the relation attribute “conflict” generated by partitioning the sample set U can be expressed as

(3)$\chi \left(r\right)={\displaystyle \frac{U}{r}}=\{(x_1,x_2),\left(x_3\right)\},$

where r represents the relation “conflict”, and $(x_1,x_2),\left(x_3\right)$ are two equivalence classes.

4.2. Relation Entropy for Knowledge Inference

In the Shannon information theory, the uncertainty of a discrete event x with probability $p\left(x\right)$, called the self-information of x, can be measured by

(4)$h\left(x\right)=-logp\left(x\right).$

The average level of uncertainty of an information system including a group of discrete events $X=\{x_1,x_2,\cdots ,x_n\}$, called the entropy, is defined as

(5)$H\left(X\right)=\sum _{i=1}^n(p\left(x_i\right)\ast h\left(x_i\right)).$

In this study, entropy is employed to measure the levels of the uncertainties of the relations on inference chains in knowledge graphs. The concept of relation entropy is proposed. Relation entropy considers the inferential values of the relation in logical inference. If the entropy of a relation r is larger than others, it can eliminate more uncertainty in an inference chain; in other words, r has a greater impact on the inference than other relations. According to this idea, relation entropy is defined as follows.

Definition 4. 

Relation Entropy . In a knowledge graph, the relation entropy of a knowledge structure $\chi \left(r\right)$ on a relation r is defined as

(6) $\mathcal{H}\left(\chi \left(r\right)\right)=-\sum _{X_i\in \chi \left(r\right)}(P\left(X_i\right)logP\left(X_i\right)),$

where $X_i$ is an equivalence class in $\chi \left(r\right)$, and $P\left(X_i\right)=|X_i|/|U|$. $P\left(X_i\right)$ is the ratio of the number of samples in the equivalence class $X_i$ to the total number of samples in the sample set U.

Remark 2. 

The definition of relation entropy abides by the basic principle of Shannon information entropy. For a knowledge structure with n samples on a relation r, when each sample has a different value on the relation r, the number of equivalence classes in the knowledge structure equals the number of samples, n. In this case, the value of the relation entropy is maximized at $logn$. Conversely, if all samples have the same value for this relation r, the knowledge structure contains only one equivalence class. According to Definition 4, the entropy in this situation is minimized at 0. In most cases, the value of relation entropy is between these two extremes.

4.3. Calculation of Relation Entropy in RIMIE

To calculate the relation entropy, the concept of logical relation information table (LRIT) needs to be proposed firstly, which is the data basis for RIMIE. LRIT is essentially a dataset in the form of $(U,R)$ according to Definition 1, in which U is composed of the inference chains and R is a set of the relation. For example, Table 1 shows an instance of LRIT, in which each inference chain has the same series of relations; however, the values of inference chains may differ on the relations. The value of the relation may be some characteristic of the relation in the inference chain. For instance, in Table 1, $r_1\left(x_1\right)=2$ represents that the relation $r_1$ has a value of 2 on logical inference chain $x_1$. More specifically, in Figure 2, the relation “monitored” represents the two relation values of “2024-006-21 09:12:00” and “2024-06-22 13:14:00”.

RIMIE employs the uncertainty measurement methods in knowledge bases to assess the inferential values of relations in KGs. In inference processes, an inference chain can be seen as a series of steps, in which an error at any step may lead to the results completely divergent from the correct answers. Therefore, considering the effect of relations in each logical inference chain is crucial for accurately measuring their inferential values. On the other hand, if a relation r in a knowledge graph is more important for the downstream logical inference, the uncertainty associated with r should be greater. This means that at each step of a logical inference chain, considering r should bring greater benefits, and ignoring r may incur more losses.

RIMIE proposes a measurement method of relation inferential value based on relation entropy. In an inference chain, the inferential value of a relation r depends on the relation dimensions of r in a knowledge graph. RIMIE partitions the samples in the LRIT based on the attribute values of r, and a knowledge structure, denoted by $\chi \left(r\right)=\{X_1,X_2,\dots \}$, is obtained. For each equivalence class $X_i\in \chi \left(r\right)$, the samples in $X_i$ have the same attribute values on relation r, and the number of equivalence classes represents the attribute dimensions of the relation. The probability distribution of equivalence classes in $\chi \left(r\right)$ can be constructed by

(7)$P=\left\{P\left(X_i\right)\right|X_i\in \chi \left(r\right)\}.$

Correspondingly, the relation entropy is calculated by

(8)$\mathcal{H}\left(\chi \left(r\right)\right)=-\sum _{P\left(X_i\right)\in P}\left(P\left(X_i\right)logP\left(X_i\right)\right),$

where $P\left(X_i\right)=|X_i|/|U|$. An instance of the calculation process of relation entropy is shown in Figure 4.

In this study, the relation entropy reflects the importance of the relation from the inferential value. Its physical meaning is as follows: given a relation r, how much uncertainty does relation r introduce to the logical reasoning, and how much uncertainty can be reduced by removing relation r. The finer the partitioning of the sample set U by a specific relation, the greater the number of equivalence classes, and then relation r has more feature dimensions and a larger range of values. Correspondingly, the determination of a specific relation (i.e., the selection of an inference step) has greater uncertainty, or there is greater relation entropy.

According to Remark 2, the value of relation entropy should be between 0 and $logn$, corresponding to the two extreme cases. One case is when the number of equivalence classes is equal to the number of samples; this indicates that relation r has different values in different logical inference chains. According to Formula (8), the maximum entropy is $logn$. Another case is that the relation r has the same value on every logical inference chain. This means that one equivalence class covers all samples and thus the relation has the same effect on different inference chain. In this scenario, the relation entropy of r is 0, that is, r has the least inferential value.

The calculation process of relation entropy is shown in Algorithm 1. Given a logic-relation information table T based on the knowledge graph, the sample set U is divided according to the value of relation r on the logical inference chain, and the knowledge structure $\chi \left(r\right)$ is obtained (line 4 to line 12 in Algorithm 1). Subsequently, each relation is traversed and iterates the above process; the relation entropy of each relation is calculated (line 13 and line 14 in Algorithm 1).

Algorithm 1 Relation entropy

Input:  $T=(U,R),$ a logic-relation information table T composed of logical inference chain set U and relation set R. Output:  The relation entropy set $RE\left(R\right)$ 1:. $RE\left(R\right)=\varphi$ 2:. for each r in R do 3:.    $\chi \left(r\right)=\varphi$ 4:.    for $l\in U$ do 5:.      $n=\mathrm{r}\left(l\right)$ 6:.      if $\chi {\left(r\right)}_n=\varphi$ then 7:.         $\chi {\left(r\right)}_n=\left\{l\right\}$ 8:.         $\chi {\left(r\right)}_n\in \chi \left(r\right)$ 9:.      else 10:.         $\chi {\left(r\right)}_n=\chi {\left(r\right)}_n\cup \left\{l\right\}$ 11:.      end if 12:.    end for 13:.    Get $P=\left\{P\left(X_i\right)\right|X_i\in \chi \left(r\right)\}$ by Formula (7) 14:.    Calculate $re\left(r\right)$ by Formula (8) 15:.    $RE\left(R\right)\supseteq re\left(r\right)$ 16:. end for 17:. Return $RE\left(R\right)$

5. Experimental

In this section, the effectiveness of the RIMIE is validated through a triple classification experiment, and it is demonstrated that RIMIE can effectively measure the inferential values of relations. We first introduce the dataset construction, experimental configuration and implementation, and then the experimental results are analyzed and discussed.

5.1. Dataset Construction

Dataset. This work conducts the triple classification experiments on three commonly used knowledge graph benchmark datasets: FB15K-237, NELL-995, and YAGO3-10 [39]. They are common knowledge graph datasets that are applied to knowledge graph embedded learning tasks. FB15K-237 is extracted from the Freebase knowledge graph project, eliminating redundant relations and the type of relation is relatively basic, focusing on direct relations between entities. YAGO3-10 is a subset of the dataset YAGO3, covering parts of Wikipedia, Wikidata, and WordNet, and contains the entities associated with at least ten different relations. The relations are richer than FB15K-237, especially in the academic field and in the field of time and historical events, and are suitable for more complex reasoning tasks. NELL-995 is derived from the dataset “The Never Ending Language Learner (NELL) System”, representing a subset of NELL’s 95th iteration. The entities in NELL-995 are more diverse, with relations related to Internet information and social behavior being more common, covering multiple domains of knowledge automatically learned from the Internet. The statistics of the dataset are shown in Table 2.

Construction of Information Table. It must be mentioned that since our study is the first exploration about the inferential value measurement of relations in knowledge graphs, there is still no specialized dataset released on this research, as of the time of our submission. Therefore, the dataset of knowledge graph inference for experiments has to be built on the generalized datasets. Considering the logical inference chains in the datasets of knowledge graphs are usually implicit and require manual annotations, which is costly, this work utilizes the shortest paths as the logical inference chains to construct the logical relation information table. The attribute values in the information table represent the frequencies of occurrence of relations on the shortest paths, i.e., the logical inference chains. Based on this, the relation entropy is calculated. Generally, there are a large number of entities in a knowledge graph, and computing the shortest path between every pair of entities is very time-consuming, especially for the large-scale knowledge graphs. Fortunately, as long as there are enough logical inference chains and they are diverse enough, RIMIE can work effectively. Therefore, we only need to extract a sufficient number of shortest paths, rather than the shortest paths between all entities, to construct the information table.

Relation Combinations. The experiment of triple classification on LRIT is designed to verify the impact of different relations on the completeness of the knowledge graph. This work mainly compares the inferential value of three different sets of relations:

• Relation-Entropy: the top@k relations selected based on relation entropy.

• Relation-Max: the top@k relations with the highest occurrences in the knowledge graph.

• Relation-Random: the k relations selected randomly.

Construction of Test Set. The triple classification experiment also requires constructing the test set, denoted as $F_{test}$, containing the logical inference chains. Each sample in the test set is a logical inference chain. Subsequently, the triplets are sampled from these logical inference chains, and a given number of negative examples are constructed for each triplet for use in the classification experiment.

5.2. Experimental Configuration and Implementation

Triplet Classification. This study employs three embedding models, TransE [40], ComplEx [41], and VLP [42], to obtain the vector representations of entities and relations of knowledge graphs. The introduction to the models is displayed in Appendix A. Three sets of relations are selected: Relation-Entropy, Relation-Max and Relation-Random, introduced above. A new training set is generated by removing the triplets containing these relations from the original training set. Correspondingly, four different training sets are generated: the original complete training set, and three training sets obtained by deleting the relations from each of the three selected groups.

Remark 3. 

We are actually aware that the triplet classification experiments on TransE/ComplEx, compared with the real-wold long-chain reasoning (such as crime investigating and suspense stories), are far from exciting. We still construct these experiments for two reasons, as follows. Firstly, as the classical task and model, respectively, although triplet classification and TransE/ComplEx are no longer so fashionable, they are still the simplest and most popular forms of knowledge reasoning and embedding, respectively. Their basic idea, especially that of TransE, has been embedded in almost all knowledge graph embedding tasks. Secondly, as of the time of our submission, there has not been any released long-chain reasoning task dataset to be used. Fortunately, it is known that scaling the triplet-based classification is the lowest common denominator in scaling more complex reasoning applications. Since any long-chain reasoning on knowledge graphs is composed of a series of triplets, the correct classification or not of each triplet is crucial for the whole reasoning process. To sum up, scaling the triplet classification experiments on TransE/ComplEx is therefore the lowest common denominator in scaling more complex knowledge reasoning. Our work provides a baseline for further research on more complex knowledge reasoning.

Implementation. The logical relation information table is constructed, and the relation entropy $r_e$ for each relation is calculated by using RIMIE. This work needs to compare the effects of different relation combinations on the experimental results. Given a relation set R, we remove the triplets containing relation $r\in R$ from the original training set F, and this results in a new training set $F_{f_R}^{-}$. The knowledge graph embedding models are used to obtain the vector representations of entities and relations on the new training set $F_{f_R}^{-}$. Then, the triplet classification experiments are performed by using the obtained entity and relation vectors on the test set $F_{test}$. Finally, the impacts of different relation combinations are observed and compared on the triplet classification experiments. In this process, to ensure the credibility of results, the same test set is used for each group of experiments.

Evaluation Criteria. To assess the experimental results, this work employs the classical evaluation metrics from classification tasks: Precision-Recall (PR) curve and Receiver Operating Characteristic (ROC) curve, as well as the area under the PR curve (PR AUC) and the area under the ROC curve (ROC AUC). PR and ROC curves not only assess the classification ability of the model but also demonstrate its generalization ability and the impact of different classification thresholds on the models. Therefore, PR and ROC curves are used as the evaluation metrics.

Experimental Environment. All experiments are conducted on the PyTorch deep learning framework with a Linux cluster (CPU: Intel(R) Core(TM) i7-7800X CPU @ 3.50 GHz, GPU:NVIDIA GeForce RTX 3090 Ti).

5.3. Experimental Results and Analyses

In this section, we explain the experimental results and analyze some of the potential symptoms and causes behind them.

Triple Classification. The PR curves and ROC curves of the triplet classification experiment are shown in Figure 5, Figure 6 and Figure 7, respectively. The sizes of the four training sets obtained after removing the four groups of relations are shown in Table 3. The closer the PR curve is to the upper right corner and the ROC curve to the upper left corner, the better the experimental results are considered. The area ratios (%) under the PR curve and ROC curve (AUC) are shown in Table 4, Table 5 and Table 6. It can be seen that compared with the experimental results on the original training set “Raw”, all the other three groups have shown varying degrees of declines. “REntropy” declines more than “Random”; this verifies the hypothesis that relations with higher relation entropy have greater inferential value. According to Table 3, the train dataset of “REntropy” is smaller than “Random”, so we add “Max” to analyze the contact between the inferential value and the triples contained in the relation. Among these, the “Max” and “REntropy” experiments exhibited more significant decreases than others, and this indicates that these two sets of relations have higher reasoning value and greater impact on model training and learning.

According to the hypothesis of this study, the experimental results should have the most significant decrease after removing the relations with high relation entropy. However, the “Max” experiment achieved results similar to those of the “REntropy” experiment. This is because the relations in “Max” are the top@k relations with the highest frequency in the dataset, and they cover a large number of triples. After these relations are removed, the amount of data in the knowledge graph drops sharply, and this results in a significant loss of knowledge graph structure and semantics. Thus, this leads to a “qualitative change induced by quantitative change”, causing the model-learning performance to deteriorate on the training set.

It can also be seen that relation frequency is not the only determining factor affecting relation inferential value by data, as shown in Figure 8, where the numbers of triples in the training set obtained after removing the four groups of relations are displayed.

• Comparing Figure 5, Figure 6 and Figure 7 with Figure 8, it can be observed that the triples in the “Random” group are more than the other two groups, and the experimental results on “Random” exceed those on “Max” and “REntropy”.

• As listed in Table 4, Table 5 and Table 6, as well as the number of triples in Table 3, the triples in “Max” are less than in “REntropy”, but its corresponding experimental results are better than or close to those of “REntropy”.

• Even in YAGO3-10, “Max” has only one-fifth of the triples of “REntropy”, but its experimental results are better than “REntropy” and even better than “Random”.

These observations indicate that the occurrence frequency of a relation in a knowledge graph is not the sole determining factor in evaluating its inferential values, despite it having influence on the inferential value of relations. In other words, different relations have different impacts on a knowledge graph, and relation inferential value is the result of multiple factors. Through RIMIE, the truly important relations in the knowledge graph can be identified beyond just their occurrence frequencies.

Remark 4. 

Our experimental hypothesis is that the experimental results obtained on the training set formed by deleting $Top@k$ relations with the highest entropy are lower than those obtained on the training set formed by randomly deleting k relations. As can be seen from Figure 5, Figure 6 and Figure 7, “REntropy” is lower than “Random”. This is consistent with our conjecture. Further, we compare and analyze the relationship between “Max” and “REntopy” to show that the number of occurrences of the relation is not the determining factor of the experimental results.

5.4. Analysis of Relation Connectivity

As stated above, RIMIE employs the relation entropy to measure the inferential values of relations, and the high-entropy relations are considered crucial to the inference tasks on knowledge graphs. It is well known that in complex networks, the connectivity of nodes and edges is regarded as an indicator of importance, mirroring the information-spreading capacity. Since knowledge graphs are a special class of complex networks, and knowledge reasoning can also be seen as a special kind of information spreading, it is necessary to explore the relationship between relation entropy and connectivity to further illustrate the effectiveness of RIMIE. The connectivity of a triple is defined as the sum of the degrees of the head and tail entities connected by the relation in the triple. Then, the connectivity of relations can be defined. The connectivity of a relation can be seen as the average connectivity of triples covering the relation. Triple connectivity and relation connectivity can be calculated by

(9)$C\left(\left(h,r,t\right)\right)=degree\left(h\right)+degree\left(t\right)$

and

(10)$C\left(r\right)={\displaystyle \frac{\sum (h,r,t)\in F^{r}C\left(\left(h,r,t\right)\right)}{|F^r|}},$

respectively, where $F^r$ is the set of triples covering relation r. For a set of relations, the average connectivity of relations in the set is defined as the connectivity of the relation set. The connectivity of the relation sets “Random”, “Max”, and “REntropy” is shown in Table 7.

It can be observed that the connectivity of selected relations based on entropy is greater than that of the randomly selected ones, and it is also greater than the most frequently appearing relations in FB15k-237 and NELL-995. However, on the dataset YAGO3-10, it is only less than the connectivity of relations in the “Max” group. This is because YAGO3-10 contains a special relation called “hasGender”. This relation has only two tail entities, namely “male” and “female”, which are associated with all character entities in the graph. As a result, these two nodes have very high degrees, leading to a very high connectivity of the “hasGender” relation. Moreover, the “hasGender” relation increases the overall connectivity of relations in the “Max” group. Therefore, the connectivity of relations in the “Max” group in YAGO3-10 is greater than the other two groups. After the “hasGender” relation is removed, the connectivity of relations in the “Max” group is 626, which is less than the connectivity of relations in the “REntropy” group. Thus, it can be seen that the connectivity of relations selected based on entropy is greater, which is more conducive to information propagation. This indicates that RIMIE can also effectively mirror the inferential value of relations from the perspective of connectivity.

6. Conclusions

This work presents a new model, called RIMIE, for evaluating the inferential values of relations in knowledge graphs. RIMIE introduces the concept of relation entropy, and defines relation entropy as the inferential value of relations. To be specific, relation entropy is calculated based on the logical relation information table and information entropy. The experiments of RIMIE were conducted on three benchmark datasets, FB15K-237, NELL-995, and YAGO3-10, to evaluate the effectiveness of RIMIE. The experimental results indicate that RIMIE can effectively identify high-inference value relations in knowledge graphs, which are crucial for logical reasoning tasks. Additionally, the connectivity and the occurrence frequency of relations within knowledge graphs have been analyzed to further verify the effectiveness of RIMIE.

Certainly, we are also aware that this work is far from perfect and has certain limitations, mainly due to the following: The implementation of the RIMIE algorithm relies on the logical path sample in the knowledge graph, and the existing knowledge graph is stored in the form of triples, and the logical path exists implicitly in the graph, which needs to be marked according to the relevant background and relevant knowledge, which requires a certain labor cost. In the future, we will design the automatic construction method for a logical relation information table in the knowledge graph.

In addition, there are still some applications of relation entropy worth exploring, calculating relation entropy on specific path-inference tasks and datasets, paying more attention to relations with higher relation entropy to help model learning and training, and exploring the application of relation entropy to various tasks related to knowledge graphs.

Author Contributions

Conceptualization, X.Z. and H.Y.; Funding acquisition, L.D.; Methodology, X.Z.; Supervision, L.D.; Writing—original draft, X.Z. and L.D.; Writing—review and editing, X.K., H.Y. and L.D. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The datasets used in research are publicly available.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The knowledge graph of the UEFA European Championship. In the reasoning task of finding the top scorer of the 2016 European Championship, starting from the entity “UEFA”, there are three different choices upon reaching “European Championship”. These three choices lead to three different outcomes. If the relation “final date” is chosen, a time entity will be obtained, which is far from the correct answer. Choosing the relation “responsible person” yields a person entity, but this is also incorrect. Only by choosing the relation “top scorer” can the correct answer be found. Different relations lead to different results, and logical reasoning needs to consider different semantics of relations.

Figure 2. High-inference value relations. In a burglary case, there are clues pointing to two suspects (A and B), in which the surveillance camera caught A entering the store one hour before the burglary, and B entering the store the day before. Meanwhile, the neighbor saw A entering the store, and frequently appeared around the store in the previous days, and found A’s fingerprint in the store; here, surveillance, witness and fingerprint are all important relations with high inferential value. Changing the relation value of “2024-6-22 13:14:00” will reduce the suspicion of A and may lead to different results.

Figure 3. The inference chain in a knowledge graph. In a case, according to the clues, the police locks up three suspects A, B and C; there is a conflict between them and the victim S; the severity of the conflict between A and B and S is 1, and the severity of the conflict between C and S is 2; there are three reasoning chains in the figure, that is, the part covered by the shadows.

Figure 4. RIMIE calculation framework. Circular nodes represent entities and short lines represent relations. The five inference chains are divided by the number of occurrences of [Forumla omitted. See PDF.], and are divided into three equivalence classes according to the number of occurrences of [Forumla omitted. See PDF.] 0, 1, 2. Then, the relation entropy is calculated according to the probability related to the equivalence classes.

Figure 5. PR and ROC curves on datasets FB15K-237, NLL-995, YAGO3-10 by TransE. “Raw” represents the experimental result on the original training set with no relation removed. “Random” represents the experimental result on the training set formed after randomly removing a group of relations from the training set. “Max” represents the experimental result on the training set formed after removing the top@k most frequently occurring relations from the training set. “REntropy” represents the experimental result on the training set formed after removing the Top@k relation with the highest relation entropy from the training set.

Figure 6. PR and ROC curves on datasets FB15K-237, NLL-995, YAGO3-10 by ComplEx. The meanings of “Raw”, “Random”, “Max” and “REntropy” are the same as described above.

Figure 7. PR and ROC curves on datasets FB15K-237, NLL-995, YAGO3-10 by VLP. The meanings of “Raw”, “Random”, “Max” and “REntropy” are the same as described above.

Figure 8. Bar chart of the number of triples in the training set obtained after deleting four relations on FB15K-237, NELL-995, YAGO3-10.

Table 1

An instance of LRIT.

U	${\mathit{r}}_1$	${\mathit{r}}_2$	${\mathit{r}}_3$	${\mathit{r}}_4$
$x_1$	2	1	2	1
$x_2$	2	2	1	1
$x_3$	0	3	0	2
$x_4$	1	1	1	3

Table 2

The dataset statistics, in which the numbers of entities and relations and the volumes of training, validation, and test sets, are listed, respectively.

Dataset	Entity	Relation	Train	Valid	Test
FB15K-237	14,541	237	272,115	17,535	20,466
NELL-995	75,492	200	149,678	543	3992
YAGO3-10	123,182	37	1,079,040	5000	5000

Table 3

Number of triples on different training sets in FB15K-237, NLL-995, YAGO3-10.

	Raw	Random	Max	REntropy
FB15K-237	272,115	260,384	173,324	219,407
NELL-995	149,678	118,658	82,329	94,227
YAGO3-10	1,079,040	862,142	149,067	764,671

Table 4

Area ratios (%) under ROC and PR Curve area (auc) on FB15K-237, NLL-995, YAGO3-10 by TransE. The bold part is the lowest value on each dataset of PR Area ratios and ROC Area ratios.

Dataset	PR				ROC
	Raw	Random	Max	REntropy	Raw	Random	Max	REntropy
FB15K-237	97.92	95.70	90.66	88.83	99.45	97.88	94.00	93.99
NELL-995	98.01	99.99	82.38	83.47	98.67	91.24	78.35	80.24
YAGO3-10	94.90	75.25	85.45	63.66	96.84	70.71	84.91	64.97

Table 5

Area ratios (%) under ROC and PR Curve area (auc) on FB15K-237, NLL-995, YAGO3-10 by ComplEx. The bold part is the lowest value on each dataset of PR Area ratios and ROC Area ratios.

Dataset	PR				ROC
	Raw	Random	Max	REntropy	Raw	Random	Max	REntropy
FB15K-237	92.01	87.12	82.56	79.42	92.92	85.04	78.15	69.47
NELL-995	93.41	92.74	88.50	90.05	90.18	89.44	86.67	88.57
YAGO3-10	96.84	94.35	95.93	53.03	97.27	94.00	95.58	63.04

Table 6

Area ratios (%) under ROC and PR Curve area (auc) on FB15K-237, NLL-995, YAGO3-10 by VLP. The bold part is the lowest value on each dataset of PR Area ratios and ROC Area ratios.

Dataset	PR				ROC
	Raw	Random	Max	REntropy	Raw	Random	Max	REntropy
FB15K-237	98.07	98.16	89.53	86.73	97.89	98.06	84.11	80.51
NELL-995	92.48	90.93	64.15	68.02	88.92	86.83	74.06	77.98
YAGO3-10	88.94	88.19	89.95	53.66	90.40	87.74	84.95	51.70

Table 7

Relation Connectivity on FB15K-237, NLL-995, and YAGO3-10.

	Random	Max	REntropy
FB15K-237	850	368	927
NELL-995	268	395	496
YAGO3-10	66	11,847	906

Appendix A. Knowledge Graph Embedding Models

Appendix A.1. TransE

Entities and relations in a knowledge graph need to be embedded before the triplet classification experiment. In this work, the TransE [40] model is adopted as the knowledge graph embedding model of this work. TransE is the most classical translation distance model, which regards relation as the translation from the beginning to the end entity. Through training, the distance between the head entity vector h plus the relation vector r and the tail entity vector t is minimum, i.e., $h+r\approx t$. The score function of the TransE model can be defined as follows:(A1)$$d_r=\left|\right|h+r-t\left|\right|.$$

Negative sampling techniques have been proven to be highly effective for both knowledge graph embeddings and word embeddings. Therefore, this work adopts a log-sigmoid loss function with negative samples, defined as follows:(A2)$$L=-log(\gamma -d_r(h,t))-\sum _{i=1}^{n}p(h_i^1,r,t_i^1)log\sigma (d_r(h,t)-\gamma ),$$ where $\gamma$ is the margin and $\sigma$ is the sigmoid function. In the testing phase, the impact of different relation combinations on the integrity of the knowledge graph and the reasoning ability of the model is evaluated through triple classification after embedding.

Appendix A.2. ComplEx

The ComplEx [41] model is a bilinear model that uses a different scoring function than TransE and introduces negative numbers to model asymmetric relations. Each entity and relation is represented by a complex-valued vector, and the score function for each triple is defined as follows:(A3)$$\begin{array}{cc}\hfill \varphi (h,r,t;\Theta )=& Re\left(<w_r,e_s,{\overline{e}}_o>\right)\hfill \\ \hfill =& Re\left(\sum _{k=1}^K{w}_{rk}{e}_{sk}{\overline{e}}_{ok}\right)\hfill \\ \hfill =& ⟨Re\left(w_r\right),Re\left(e_s\right),Re\left(e_o\right)⟩\hfill \\ & +⟨Re\left(w_r\right),Im\left(e_s\right),Im\left(e_o\right)⟩\hfill \\ & +⟨Im\left(w_r\right),Re\left(e_s\right),Im\left(e_o\right)⟩\hfill \\ & -⟨Im\left(w_r\right),Im\left(e_s\right),Re\left(e_o\right)⟩\hfill \end{array},$$ where ${\omega }_r$∈${\mathbb{C}}^k$ is a complex vector, $Re\left(x\right)$ denotes the real part of x, and $Im\left(x\right)$ denotes the imaginary part of x. The score-calculation process for the triple (s, r, o) involves the product of the representation vectors of the relation r, the subject s, and the conjugate vector of the object o, retaining the real part of the final result. The probability that the triple (s, r, o) is true is obtained through the following formula:(A4)$$P(r,s,o)=\sigma \left(\varphi \right(r,s,o;\Theta ).$$

Appendix A.3. VLP

The VLP [42] model explicitly copies target information from relevant fact triples, providing additional clues to enhance the model’s generalization ability. When calculating the triple score, it uses queries and tail entities with the same relation as the predicted triple within the knowledge graph, incorporating information from triples with the same relation into the learned entity and relation embeddings. This results in more generalized embedding representations. The VLP also employs a different method for scoring triples, detailed as follows:(A5)$$q=W_{r,1}h+b_r$$ and (A6)$$k=W_{r,2}t.$$ where q represents the query embedding composed of the head entity and the relation. Unlike direct computation, $VLP$ employs a matrix operation approach, where k is the answer embedding of the entity after matrix operations.

During model training, each candidate answer utilizes not only its own embedding but also the query and tail entities with the same relation, (A7)$$t^{\prime }=\sigma \left(W_{agg}\left[t_N,q\right]\right)$$ and (A8)$$t_n={\displaystyle \frac{1}{N}}\sum _{i=1}^n(W_{node}{k}_i+W_{edge}{S}_{q,q_i}),$$ where $t_N$ represents the N tail entities selected based on the graph structure that have higher similarity to the correct answer t. These entities are also connected by the relation r and are referred to as clues, which assist in predicting the correct answer. $W_{ndoe}$ and $W_{edge}$ are trained parameter matrices, and $S_{q,q_i}$ is the similarity embedding between query embeddings, calculated as follows:(A9)$$S_{q,q_i}=q-q_i,$$

Finally, the score for each predicted triple is given by:(A10)$$f(h,r,t)=f_c(h,r,t)+\lambda f_g(h,r,t),$$ where $f_c(h,r,t)$ is the score in $VLP$, and $f_g(h,r,t)$ is the operation in TransE, (A11)$$f_c(h,r,t)={\displaystyle \frac{t^T{t}^{\prime }}{\Vert t\Vert \Vert t^{\prime }\Vert }}.$$

References

1. Ji, S.; Pan, S.; Cambria, E.; Marttinen, P.; Philip, S.Y. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans. Neural Netw. Learn. Syst.; 2021; 33, pp. 494-514. [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3070843] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33900922]

2. Xiao, F.; Wen, J.; Pedrycz, W. Generalized divergence-based decision making method with an application to pattern classification. IEEE Trans. Knowl. Data Eng.; 2022; [DOI: https://dx.doi.org/10.1109/TKDE.2022.3177896]

3. Ullah, A.; Wang, B.; Sheng, J.; Long, J.; Khan, N. Identification of influential nodes via effective distance-based centrality mechanism in complex networks. Complexity; 2021; 2021, pp. 1-16. [DOI: https://dx.doi.org/10.1155/2021/8403738]

4. Rossi, A.; Barbosa, D.; Firmani, D.; Matinata, A.; Merialdo, P. Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans. Knowl. Discov. Data (TKDD); 2021; 15, pp. 1-49. [DOI: https://dx.doi.org/10.1145/3424672]

5. Chen, D.; Lü, L.; Shang, M.S.; Zhang, Y.C.; Zhou, T. Identifying influential nodes in complex networks. Phys. A Stat. Mech. Its Appl.; 2012; 391, pp. 1777-1787. [DOI: https://dx.doi.org/10.1016/j.physa.2011.09.017]

6. Zhang, H.; Zhong, S.; Deng, Y.; Cheong, K.H. LFIC: Identifying influential nodes in complex networks by local fuzzy information centrality. IEEE Trans. Fuzzy Syst.; 2021; 30, pp. 3284-3296. [DOI: https://dx.doi.org/10.1109/TFUZZ.2021.3112226]

7. Lei, M.; Cheong, K.H. Node influence ranking in complex networks: A local structure entropy approach. Chaos Solitons Fractals; 2022; 160, 112136. [DOI: https://dx.doi.org/10.1016/j.chaos.2022.112136]

8. Scabini, L.F.; Ribas, L.C.; Neiva, M.B.; Junior, A.G.; Farfan, A.J.; Bruno, O.M. Social interaction layers in complex networks for the dynamical epidemic modeling of COVID-19 in Brazil. Phys. A Stat. Mech. Its Appl.; 2021; 564, 125498. [DOI: https://dx.doi.org/10.1016/j.physa.2020.125498]

9. Sun, N.; Ding, M.; Jiang, J.; Xu, W.; Mo, X.; Tai, Y.; Zhang, J. Cyber threat intelligence mining for proactive cybersecurity defense: A survey and new perspectives. IEEE Commun. Surv. Tutor.; 2023; 25, pp. 1748-1774. [DOI: https://dx.doi.org/10.1109/COMST.2023.3273282]

10. Cohen, A.A.; Ferrucci, L.; Fülöp, T.; Gravel, D.; Hao, N.; Kriete, A.; Levine, M.E.; Lipsitz, L.A.; Olde Rikkert, M.G.; Rutenberg, A. et al. A complex systems approach to aging biology. Nat. Aging; 2022; 2, pp. 580-591. [DOI: https://dx.doi.org/10.1038/s43587-022-00252-6]

11. Kumar, S.; Mallik, A.; Khetarpal, A.; Panda, B.S. Influence maximization in social networks using graph embedding and graph neural network. Inf. Sci.; 2022; 607, pp. 1617-1636. [DOI: https://dx.doi.org/10.1016/j.ins.2022.06.075]

12. He, Q.; Sun, L.; Wang, X.; Wang, Z.; Huang, M.; Yi, B.; Wang, Y.; Ma, L. Positive opinion maximization in signed social networks. Inf. Sci.; 2021; 558, pp. 34-49. [DOI: https://dx.doi.org/10.1016/j.ins.2020.12.091]

13. Yu, E.Y.; Chen, D.B.; Zhao, J.Y. Identifying critical edges in complex networks. Sci. Rep.; 2018; 8, 14469. [DOI: https://dx.doi.org/10.1038/s41598-018-32631-8] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30262804]

14. Dong, L.; Yao, H.; Li, D.; Wang, Y.; Li, S.; Liang, Q. Improving graph neural network via complex-network-based anchor structure. Knowl.-Based Syst.; 2021; 233, 107528. [DOI: https://dx.doi.org/10.1016/j.knosys.2021.107528]

15. Zareie, A.; Sheikhahmadi, A.; Jalili, M.; Fasaei, M.S.K. Finding influential nodes in social networks based on neighborhood correlation coefficient. Knowl.-Based Syst.; 2020; 194, 105580. [DOI: https://dx.doi.org/10.1016/j.knosys.2020.105580]

16. Ai, D.; Liu, X.L.; Kang, W.Z.; Li, L.N.; Lü, S.Q.; Liu, Y. SLGC: Identifying influential nodes in complex networks from the perspectives of self-centrality, local centrality, and global centrality. Chin. Phys. B; 2023; 32, 118902. [DOI: https://dx.doi.org/10.1088/1674-1056/aceee8]

17. Zhang, J.; Luo, Y. Degree centrality, betweenness centrality, and closeness centrality in social network. Proceedings of the 2017 2nd International Conference on Modelling, Simulation and Applied Mathematics (MSAM2017); Bangkok, Thailand, 26–27 March 2017; Atlantis Press: Amsterdam, The Netherlands, 2017; pp. 300-303.

18. Sun, P.G.; Miao, Q.; Staab, S. Community-based k-shell decomposition for identifying influential spreaders. Pattern Recognit.; 2021; 120, 108130. [DOI: https://dx.doi.org/10.1016/j.patcog.2021.108130]

19. Xu, G.Q.; Meng, L.; Tu, D.Q.; Yang, P.L. LCH: A local clustering H-index centrality measure for identifying and ranking influential nodes in complex networks. Chin. Phys. B; 2021; 30, 088901. [DOI: https://dx.doi.org/10.1088/1674-1056/abea86]

20. Ullah, A.; Sheng, J.; Wang, B.; Din, S.U.; Khan, N. Leveraging neighborhood and path information for influential spreaders recognition in complex networks. J. Intell. Inf. Syst.; 2024; 62, pp. 377-401. [DOI: https://dx.doi.org/10.1007/s10844-023-00822-z]

21. Barman, S.C. Closeness centrality of friendship and lollipop graphs. Innov. Multidiscip. Res. Present Future Time; 2023; 4, pp. 30-35.

22. Barthelemy, M. Betweenness centrality in large complex networks. Eur. Phys. J. B; 2004; 38, pp. 163-168. [DOI: https://dx.doi.org/10.1140/epjb/e2004-00111-4]

23. Deng, R.; Li, M.; Zhang, Q. Structural analysis and the sum of nodes’ betweenness centrality in complex networks. arXiv; 2024; arXiv: 2402.13507

24. Xu, Q.; Sun, L.; Bu, C. The two-steps eigenvector centrality in complex networks. Chaos Solitons Fractals; 2023; 173, 113753. [DOI: https://dx.doi.org/10.1016/j.chaos.2023.113753]

25. Yu, E.; Fu, Y.; Tang, Q.; Zhao, J.Y.; Chen, D.B. A re-ranking algorithm for identifying influential nodes in complex networks. IEEE Access; 2020; 8, pp. 211281-211290. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.3038791]

26. Li, Z.; Ren, T.; Ma, X.; Liu, S.; Zhang, Y.; Zhou, T. Identifying influential spreaders by gravity model. Sci. Rep.; 2019; 9, 8387. [DOI: https://dx.doi.org/10.1038/s41598-019-44930-9] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31182773]

27. Li, H.; Shang, Q.; Deng, Y. A generalized gravity model for influential spreaders identification in complex networks. Chaos Solitons Fractals; 2021; 143, 110456. [DOI: https://dx.doi.org/10.1016/j.chaos.2020.110456]

28. Li, A.; Pan, Y. Structural information and dynamical complexity of networks. IEEE Trans. Inf. Theory; 2016; 62, pp. 3290-3339. [DOI: https://dx.doi.org/10.1109/TIT.2016.2555904]

29. Park, N.; Kan, A.; Dong, X.L.; Zhao, T.; Faloutsos, C. Estimating node importance in knowledge graphs using graph neural networks. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; Anchorage, AK, USA, 4–8 August 2019; pp. 596-606.

30. Park, N.; Kan, A.; Dong, X.L.; Zhao, T.; Faloutsos, C. Multiimport: Inferring node importance in a knowledge graph from multiple input signals. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; Virtual, 6–10 July 2020; pp. 503-512.

31. Fensel, D.; Şimşek, U.; Angele, K.; Huaman, E.; Kärle, E.; Panasiuk, O.; Toma, I.; Umbrich, J.; Wahler, A.; Fensel, D. et al. Introduction: What is a knowledge graph?. Knowledge Graphs: Methodology, Tools and Selected Use Cases; Springer: Berlin/Heidelberg, Germany, 2020; pp. 1-10.

32. Maji, G.; Sen, S. Ranking influential nodes in complex network using edge weight degree based shell decomposition. J. Comput. Sci.; 2023; 74, 102179. [DOI: https://dx.doi.org/10.1016/j.jocs.2023.102179]

33. Zhang, Q.; Xie, Q.; Wang, G. A survey on rough set theory and its applications. CAAI Trans. Intell. Technol.; 2016; 1, pp. 323-333. [DOI: https://dx.doi.org/10.1016/j.trit.2016.11.001]

34. Zhang, X.; Yao, Y. Tri-level attribute reduction in rough set theory. Expert Syst. Appl.; 2022; 190, 116187. [DOI: https://dx.doi.org/10.1016/j.eswa.2021.116187]

35. Chen, Y.; Liu, K.; Song, J.; Fujita, H.; Yang, X.; Qian, Y. Attribute group for attribute reduction. Inf. Sci.; 2020; 535, pp. 64-80. [DOI: https://dx.doi.org/10.1016/j.ins.2020.05.010]

36. Bao, H.; Wu, W.Z.; Zheng, J.W.; Li, T.J. Entropy based optimal scale combination selection for generalized multi-scale information tables. Int. J. Mach. Learn. Cybern.; 2021; 12, pp. 1427-1437. [DOI: https://dx.doi.org/10.1007/s13042-020-01243-y]

37. Li, Z.; Zhang, G.; Wu, W.Z.; Xie, N. Measures of uncertainty for knowledge bases. Knowl. Inf. Syst.; 2020; 62, pp. 611-637. [DOI: https://dx.doi.org/10.1007/s10115-019-01363-0]

38. Fang, T.; Chen, Z.; Song, Y.; Bosselut, A. Complex reasoning over logical queries on commonsense knowledge graphs. arXiv; 2024; arXiv: 2403.07398

39. Han, X.; Cao, S.; Xin, L.; Lin, Y.; Liu, Z.; Sun, M.; Li, J. OpenKE: An Open Toolkit for Knowledge Embedding. Proceedings of the EMNLP; Brussels, Belgium, 31 October–4 November 2018.

40. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst.; 2013; 26, pp. 2787-2795.

41. Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; Bouchard, G. Complex embeddings for simple link prediction. Proceedings of the International Conference on Machine Learning (PMLR); New York, NY, USA, 19–24 June 2016; pp. 2071-2080.

42. Li, R.; Chen, X.; Li, C.; Shen, Y.; Zhao, J.; Wang, Y.; Han, W.; Sun, H.; Deng, W.; Zhang, Q. et al. To copy rather than memorize: A vertical learning paradigm for knowledge graph completion. arXiv; 2023; arXiv: 2305.14126