Content area
Purpose
To support digital humanities research more effectively and efficiently, this study develops a novel Knowledge Graph Analysis Tool of People and Organizations (KGAT-PO) for the Digital Humanities Research Platform for Biographies of Chinese Malaysian Personalities (DHRP-BCMP) based on artificial intelligence (AI) technology that would not only allow humanities scholars to look at the relationships between people but also has the potential for aiding digital humanities research by identifying latent relationships between people via relationships between people and organizations.
Design/methodology/approach
To verify the effectiveness of KGAT-PO, a counterbalanced design was applied to compare research participants in two groups using DHRP-BCMP with and without KGAT-PO, respectively, to perform people relationship inquiry and to see if there were significant differences in the effectiveness and efficiency of exploring relationships between people, and the use of technology acceptance between the two groups. Interviews and Lag Sequential Analysis were also used to observe research participants’ perceptions and behaviors.
Findings
The results show that the DHRP-BCMP with KGAT-PO could help research participants improve the effectiveness of exploring relationships between people, and the research participants showed high technology acceptance towards using DHRP-BCMP with KGAT-PO. Moreover, the research participants who used DHRP-BCMP with KGAT-PO could identify helpful textual patterns to explore people’s relationships more quickly than DHRP-BCMP without KGAT-PO. The interviews revealed that most research participants agreed that the KGAT-PO is a good starting point for exploring relationships between people and improves the effectiveness and efficiency of exploring people’s relationship networks.
Research limitations/implications
The research’s limitations encompass challenges related to data quality, complex people relationships, and privacy and ethics concerns. Currently, the KGAT-PO is limited to recognizing eight types of person-to-person relationships, including couple, sibling, parent-child, friend, teacher-student, relative, work, and others. These factors should be carefully considered to ensure the tool’s accuracy, usability, and ethical application in enhancing digital humanities research.
Practical implications
The study’s practical implications encompass enhanced research efficiency, aiding humanities scholars in uncovering latent interpersonal relationships within historical texts with high technology acceptance. Additionally, the tool’s applications can extend to social sciences, business and marketing, educational settings, and innovative research directions, ultimately contributing to data-driven insights in the field of digital humanities.
Originality/value
The research’s originality lies in creating a Knowledge Graph Analysis Tool of People and Organizations (KGAT-PO) using AI, bridging the gap between digital humanities research and AI technology. Its value is evident in its potential to efficiently uncover hidden people relationships, aiding digital humanities scholars in gaining new insights and perspectives, ultimately enhancing the depth and effectiveness of their research.
1. Introduction
Humanities research has often relied chiefly on traditional text-based reading approaches to explore textual content. However, when faced with the information explosion of the current information society, research methods that solely rely on manpower to draw conclusions based on manual reading have already been considered inefficient. It is hard to conduct exploratory research on such big data that far exceeds the capacity of manual reading tasks. Hence, scholars in the humanities have initiated considerations regarding the integration of information technologies to enhance their research, aiming to identify phenomena that might have been challenging to discern through traditional means or to conduct research that has been hard to imagine in the pre-digital era (Hsiang and Tu, 2011). In recent years, humanities research supported by digital databases has gradually become an important research method for humanities scholars (Hockey, 2004). Additionally, the development of digital humanities tools based on exploring the contents of digital databases has also played an essential role in adding momentum to the development of digital humanities. The development of digital humanities tools can currently be divided into several areas: text analysis tools, network analysis tools, Geographic Information System (GIS)/visualization tools, and integrated system platforms (Yuting et al., 2023; Liu and Wang, 2020). An important research topic in digital humanities based on text analysis tools is exploring the relationship between named entities within a given text, such as people, places, organization names, etc. Current text analysis tools for humanities research often lack the capability to comprehensively explore both person-to-person and person-to-organization entity relationships within textual data (Chen et al., 2021). Additionally, the user interfaces of these tools are not as user-friendly or comprehensive as desired (Du et al., 2021). These limitations hinder scholars’ ability to analyze historical documents, social movements, and cultural interactions in a holistic manner. This study aims to address these gaps by developing a more advanced tool that leverages recent advancements in natural language processing (NLP) to enable a more comprehensive analysis of these relationships.
Knowledge graph (KG) that allows for a structured and visual representation of knowledge has slowly evolved from semantic networks, knowledge representation, and natural language processing technologies (Al-Khatib et al., 2020). It is a semantic data structure that facilitates meaningful knowledge representation in digital information systems and supports users in exploring digital resources more effectively, and its knowledge representation and visualization properties are conducive to eliciting and absorbing knowledge (Haslhofer et al., 2018). The advancement of knowledge graphs holds immense importance and significance within the linguistics and social sciences. A knowledge graph consists of nodes and edges, where nodes are entities identified from texts, such as people, places, and organizations’ names, and edges represent the attributes of entities or their relationships to each other, such as friendship, marriage, work, etc. In other words, knowledge graphs can compile complex information into a single diagram concisely and effectively. They provide users with a macro and distant reading perspective of the relationship between entities, thus helping interpret relationship contexts. However, research on applying knowledge reasoning and data visualization techniques to digital humanities studies, such as genealogy study (Wang et al., 2023) or knowledge discovery from ancient Chinese scientific and technological documents (Zheng et al., 2023a), etc. is still limited. In general, the methods for knowledge reasoning in automatically generating knowledge graphs include logic rules-based, distributed representation-based, and neural network-based approaches (Wang et al., 2023).
Based on Chinese-named entity recognition (CNER) technology (Liu et al., 2022), a neural network-based machine learning scheme, and the data visualization method, this study developed a Knowledge Graph Analysis Tool of People and Organizations (KGAT-PO) for digital humanities research that can express the relationship between people, as well as people and organizations as knowledge graphs, to assist humanities researchers in interpreting the relationships in biographical texts. Since biographical texts enable readers to understand the life trajectories of the people they portray, they are significant for cultural exploration and intellectual inspiration (Rustin, 2000). It is important to note that in biographical texts, in addition to describing the relationship between person-to-person, the relationship between person-to-organization is also inextricably linked (Cárdenas et al., 2014). This research uses The Malaysia Henghua Personalities (Ling and Teck, 2019) as the target biographical texts for the developed KGAT-PO. This book is dedicated to the biographies of outstanding Malaysians descended from the Chinese Henghua region. It records 85 biographies of Henghua Malaysians in politics, business, culture and education, and arts, as well as the Chinese Malaysian organizations they were closely connected to, such as the Malaysia Heng Ann Association, the Malaysian Chinese Community, etc. Some organizations are for-profit organizations, while others are social clubs formed by fellow Chinese Malaysians. Regardless of the purpose of their establishment, they all played a crucial role in the continuation and evolution of Chinese Henghua culture in Malaysia.
Based on the above, this study developed a KGAT-PO based on the need for interpreting the person-to-person and person-to-organization relationships recorded in the biographical texts of The Malaysia Henghua Personalities. Through automatic name entity and relationship recognition technology enabled by machine learning, it can represent person-to-person and person-to-organization relationships in the distant-reading perspective of a knowledge graph. It can assist humanities scholars in interpreting the contexts of relationships between people and organizations in biographical texts. It also provides an interface for reading texts to achieve cross-referencing between distant and close reading. This allows humanities scholars to explore person-to-person and person-to-organization relationships, and to use an organization-based approach to identify latent links that would be impossible to fully grasp by looking at individual biographies on their own.
To verify the effectiveness of this tool in supporting digital humanities research, this study evaluated three aspects: the effectiveness and efficiency of exploring textual contexts in person-to-person and person-to-organization relationships, the behavioral processes of users, and their acceptance of the technology. This study also used semi-structured in-depth interviews to examine the effectiveness, efficiency, experiences, and further suggestions from humanities scholars who used the developed KGAT-PO to aid their interpretation of relationship contexts for people and organizations. Based on experimental results, this study proposed several directions for the continued improvement of the tool in the future.
2. Literature review
2.1 Development of digital humanities
Digital humanities is a discipline that integrates digital technology and humanities research and aims to explore the benefits and implications of using digital databases and digital tools to assist research in the humanities. Traditional humanities research is conducted by reading large amounts of paper-based materials and manually summarizing and analyzing them. Examples include manual labeling of source texts or simple annotations alongside specific passages that assist in analyzing research topics hidden in texts. However, this process is time-consuming and labor-intensive for humanities researchers. With the development of digital databases and digital humanities tools, humanities scholars have gradually adjusted their research methods in exploring and analyzing texts to explore research topics that were impossible to investigate with manual reading and analysis in the past. Schreibman (2008) pointed out that using digital tools to support digital humanities research has already become an important research method and trend in the humanities domain (Kirschenbaum, 2012).
In digital humanities research, close and distant reading are the two main aspects of text exploration. Close reading is the process of self-annotating and summarizing a text after reading it (Fisher and Frey, 2012), whereas distant reading does not rely on manual reading to interpret the text and instead uses digital tools to summarize or extract abstract concepts from the text, and to deconstruct and reorganize the text accompanied by information visualization technology (Jänicke et al., 2015). The development of digital humanities tools has long been of great concern to the humanities community (Bradley, 2003), and many digital collections have been compiled in Taiwan since 2002, when the National Digital Archives Program was launched. However, the problem facing digital humanities research is by no means the lack of data but how to analyze and use enormous amounts of data to give them meaning and value in a valuable and efficient way (Rosenzweig, 2003). Therefore, the development of digital humanities research tools or platforms has blossomed in recent years. For example, Chen et al. (2023) developed a hierarchical topic analysis tool (HTAT), which can assist humanities scholars on distant reading with analysis of hierarchical text topics, through classifying time stamped texts into multiple historical eras, conducting hierarchical topic modeling (HTM) according to the texts from different eras and presenting through visualization, based on hierarchical Latent Dirichelet allocation (hLDA) to support digital humanities research that is associated with the need of topic exploration on the Digital Humanities Platform for Mr. Lo Chia-Lun’s Writings (DHP-LCLW). Bingenheimer et al. (2011) used GIS technology to construct a complete spatiotemporal platform and provide humanities researchers with different perspectives to interpret the contents of the Biographies of Eminent Buddhist Monks. Chen et al. (2021) developed a character social network relationship map tool (CSNRMT) based on the China Biographical Database (CBDB) (https://projects.iq.harvard.edu/cbdb/home) to assist humanities scholars in interpreting the social networks of personage relationships. In addition, numerous digital humanities research platforms have been developed, such as the “Universal Type Digital Humanities Research Platform on Chinese Ancient Books” (Chen and Chang, 2019) jointly developed by the National Central Library and the National Chengchi University Library in Taiwan, the “Taiwan History Digital Library” (Chen et al., 2007) developed by the National Taiwan University Library, DocuSky (Tu et al., 2020) developed by the Research Center for Digital Humanities at National Taiwan University, and the “CBETA Digital Research Platform” (Tu et al., 2012) developed by Dharma Drum Institute of Liberal Arts in Taiwan.
In summary, in recent years, humanities research has slowly evolved to use digital humanities tools to assist humanities scholars’ research. However, the role of digital humanities tools should be to help humanities scholars rather than replace them (Moretti, 2016). The KGAT-PO developed by this research is a digital humanities tool that aims to assist humanities scholars in interpreting the relationships between person-to-person and person-to-organization in biographical texts.
2.2 Knowledge graph for digital humanities research
With the development of text mining technologies, attention has gradually shifted toward the need for a knowledge representation that can turn textual content into relationships between entities. Knowledge graphs are regarded as a bridge for visualizing the relationships between entities and concepts and have been widely applied in various fields in recent years. Instances of knowledge graph application can be found in areas including semantic search, intelligent question-answering, natural language, and even the Internet of Things (Pujara et al., 2013). Knowledge graphs were developed from the semantic web, knowledge representation, ontology, and natural language processing technologies and are essential applications in artificial intelligence. More specifically, a knowledge graph is a graphical model composed of nodes and edges, where nodes are entities, such as names of people, places, and organizations, and edges can be relationships or attributes between entities, such as spouses, social peers, etc. These nodes and edges are combined to form the knowledge representation method of triples, which is then used to describe the linking relationships of knowledge concepts or named entities (Singhal, 2012). Berners-Lee et al. (2001) proposed constructing a linked information system in 1989. They believed that compared to tree-based information organization, the presentation of information in a knowledge graph with nodes and their links would be more suitable for the open information system on the Internet. In recent years, knowledge graphs have become increasingly essential in digital humanities. In one particular instance, the 2020 International Conference of Digital Archives and Digital Humanities (DADH) has included knowledge graphs as a research focus, advocating how to use the semantic web and artificial intelligence technologies to develop digital humanities research tools more suitable for assisting humanities scholars’ research. Knowledge graphs show the characteristics of linking information, and their structured knowledge representations can make information more accessible.
However, building knowledge graphs is labor-intensive, so predicting unknown relationships or attributes between entities has already become a more effective way to build knowledge graphs (Shen et al., 2020). He et al. (2019) proposed combining knowledge graphs and deep learning to explore the relationship between people by using DNN and BiGRU neural networks to identify named entities and extract relationships between people. The triples of people’s relationships are then stored in Neo4j’s graphical database to help users explore the overall contexts of relationships between people. Wang et al. (2023) presented a five-layer framework for transforming physical genealogical information into digital, semantic, and interactive formats by knowledge reasoning methods to automatically generate knowledge graphs that can reveal the complex relationships between people, places, and time entities to provide practical guidelines for users to conduct genealogy research. Furthermore, Zheng et al. (2023a) designed a knowledge mining and graph visualization framework to extract knowledge of ancient Chinese scientific and technological documents bibliographic summaries (STDBS) and provide the knowledge graph (KG) comprehensively and systematically that can present the relationships among content, discipline, and author to support digital humanities research.
In summary, the application of knowledge graphs to the study of exploring relationships between people is already quite prevalent, but it is still rare to search for potential people relationships by looking at the relationships between people and organizations. Therefore, the KGAT-PO developed in this study constructs the relationship between people and builds a knowledge graph using the relationship between people and organizations. This study hopes to help humanities scholars interpret people’s relationships in biographical texts through the aspects of both people and organizations and conduct textual context inquiry from different perspectives, thereby deepening humanities scholars’ understanding of biographical texts.
3. System design
3.1 System architecture and operation procedures of KGAT-PO
To assist humanities scholars in exploring relationships between person-to-person and person-to-organization, this study developed the KGAT-PO for the Digital Humanities Research Platform for Biographies of Chinese Malaysian Personalities (DHRP-BCMP), which was developed by the Research Center for Chinese Cultural Metaverse in Taiwan at National Chengchi University based on the texts of The Malaysia Henghua Personalities. The Malaysia Henghua Personalities (Ling and Teck, 2019) results from research cooperation between the Malaysian Chinese Research Center at the University of Malaya and the Federation of Heng Ann Malaysia. The book is a biography of 85 outstanding Malaysian-Chinese personalities of Heng Ann descent. They stem from various fields, such as finance, economy, military, as well as culture and education, reflecting the historical development of Malaysian Chinese in various periods of contemporary history. Each personality is given a separate entry that uses the Dublin Core metadata standard to describe their attributes in the database.
The KGAT-PO has two main functions. The first function is extracting the entities hidden in the text through Chinese-named entity recognition (CNER) technologies (Liu et al., 2022) and automatically identifying relationships between people based on a machine learning classifier. Referring to the CBDB, eight relationships can be identified: spouse, sibling, parent-child, friend, teacher-student, relative, work, and others. The second function can extract the names of organizations related to people in biographies and list those who belong to the same organization with their contributions and associated events. It also links to the original text for close reading. The main objective in building this function is to assist humanities scholars in getting a grasp on understanding potential people relationships through the organizational aspects hidden in the texts, which cannot be found by looking at individual biographies alone. The KGAT-PO developed in this study integrates the above two functions and presents them in a convenient visual knowledge graph to facilitate the interpretation of the relationship contexts of biographical texts for humanities scholars. The system architecture and operation procedures of the KGAT-PO are shown in Figure 1 and explained as follows.
3.2 Technologies used to implement person-to-person and person-to-organization relationship identification in KGAT-PO
The technologies used to develop the KGAT-PO are explained in this section. Chinese-named entity recognition (CNER) is a foundational pre-processing procedure in the developed KGAT-PO. CNER generally refers to extracting relevant words from texts, such as people, places, official titles, organization names, etc. Figure 2 shows the procedures for implementing person-to-person and person-to-organization relationship identification. In the procedures of person-to-person relationship identification, the text is converted into semantic vectors by BERT (Bidirectional Encoder Representations from Transformers) in the first step (Sun et al., 2019). BERT revolutionized the NLP landscape by introducing a powerful encoder-only architecture that leverages bidirectional context for a wide range of tasks. The primary innovation of BERT lies in its ability to generate context-specific word embeddings, enabling it to capture complex language patterns and dependencies. This feature allows BERT to extract machine-readable data representations directly from text, simplifying subsequent tasks such as classification or regression. In contrast, GPT (Generative Pre-trained Transformer) utilizes a decoder-only transformer architecture and focuses on predicting the next word in a sequence (Yenduri et al., 2023). While BERT learns context-specific embeddings through masked language modeling, GPT excels in text generation tasks. Both BERT and GPT are influential NLP technologies with distinct strengths and use cases. BERT’s bidirectional context understanding and fine-tuning capability make it versatile for various NLP tasks, while GPT’s text generation prowess and vast knowledge base offer unique advantages. Therefore, in this study, BERT was employed as a feature extraction method, utilizing context-specific word embeddings to identify person-to-person relationships in text.
After that, CNER is performed to identify people’s names, and bidirectional long short-term memory (BiLSTM)- conditional random field (CRF), which is a recurrent neural network obtained from the combination of a BiLSTM and a CRF, is used as the people relationship classifier for person-to-person relationship identification (Huang et al., 2015). The BiLSTM-CRF model imports the named entities extracted by CNER in pairs into a pre-trained people relationship classifier to perform people relationship identification. The data for training the people relationship classifier were obtained through a web crawler, which crawled sentences describing the relationship of people entities in Wikipedia and Baidu Baike as its training texts. This study uses CBDB to train the person-to-person relationship classifier to differentiate two persons’ relationships into couple, parent-child, sibling, teacher-student, friend, relative, work, and others. Finally, all the person-to-person relationships with corresponding types of relationship names recognized by the people relationship classifier are displayed in a knowledge graph.
In the procedures of person-to-organization relationship identification, CNER is performed to identify people and organizations’ names in the first step. After that, the text between a person and an organization name is performed part-of-speech analysis by the Chinese word segmentation system Jieba to identify verbs for finding possible relationships between the person and organization. After extracting all the verbs from text passages appearing between a person and an organization name, the person-to-organization relationships associated with those verbs are displayed in the knowledge graph. Using organizations associated with target people, and their contributions and associated events related to the organization, KGAT-PO can provide humanities scholars with a more helpful knowledge graph to interpret biographical texts than only through person-to-person relationships. The objective of interpreting biographical texts in this way is to allow humanities scholars to explore hidden potential connections between people’s name entities at a deeper level. The named entity and relationship identification technologies used in this study can automatically label the entities after extraction. For example, people’s names are labeled with PER, locations with LOC, and organizations with ORG. Through these labels, we can obtain those organizations that may be related to the target people. In addition, the function also performs text filtering by an organization’s name through formal expressions so that the passage that mentions the organization’s name is extracted and can be used by humanities scholars to determine the exact contributions of a particular person to the related organization.
3.3 The user interface and functions of KGAT-PO
This section describes the overall user interface and functions of the KGAT-PO developed in this study. The overall user interface, including three main components, is shown in Figure 3. Among the three main components, the knowledge graph page’s user interface is Area 2, the search filter function’s user interface is Area 1, and the relationship information page’s user interface is Area 3. These components are used to assist humanities scholars in interpreting contextual relationships between people, as well as people and organizations hidden in texts, and are intended to achieve a cross-referencing function that combines distant and close reading. The functions are described as follows.
3.3.1 The user interface of the knowledge graph page
The knowledge graph page has two functions: identifying person-to-person and person-to-organization relationships. The system constructs a visual knowledge graph through these two functions, allowing humanities scholars to explore different aspects of person-to-person or person-to-organization relationships. The system identifies the relationships of the people and organizations’ names extracted by the CNER technology and presents them on the connecting edges of the knowledge graph. Humanities scholars can know about the relationships between person-to-person and person-to-organization hidden in the biographical texts by quickly browsing the knowledge graph. In addition, person-to-organization relationships are connected through edges that display the names of the people and organizations associated with the corresponding verbs. In the user interface of the knowledge graph, people’s names nodes are orange, and organizations’ names nodes are blue, as shown in Figure 4. The system also provides a filter function to avoid displaying excessive information in the knowledge graph that might impede the user’s reading experience.
3.3.2 The user interface of the search filter function
As shown in Figure 5, the user interface of the search filter function provides humanities scholars with different search criteria, including “person search,” “organization search,” and “relationship search,” which allow users to generate knowledge graphs using these three aspects as entry points. In addition, this user interface also includes a time search function. The function was designed considering that if humanities scholars divide the time interval too small, the amount of information filtered out will be too small. Therefore, KGAT-PO provides a decade-specific time search. The time search function allows humanities scholars to filter the knowledge graph for the time they want to view.
3.3.3 The user interface of the relationship information page
The user interface of the relationship information page gives further supplementary information about the nodes or edges selected by the user, as shown in Figure 6. The supplementary information displayed includes entity name, entity relationship, year of relationship happened, relationship source chapter, and relationship paragraph. Entity name and relationship are information presented on the knowledge graph. The year of the relationship happened is the year when the relationship between two entities happened. The relationship source chapter provides the source’s name and links back to that person’s chapter for immediate reading. Lastly, the source textual paragraph used to recognize the relationship between two people is the corresponding textual content involving both people’s names. The supplementary information presented on this page provides users with a convenient way of interpreting the relationship of biographical texts by combining distant and close reading.
4. Research methodology
4.1 Research design
To verify whether the KGAT-PO developed in this study can effectively assist humanities scholars’ research in using biographical texts to explore person-to-person and person-to-organization relationships, this research conducted a counterbalanced design to compare the differences between the research participants of an experimental group using the DHRP-BCMP with the KGAT-PO and the research participants of a control group using the DHRP-BCMP without the KGAT-PO in their effectiveness, efficiency, technology acceptance, and behavioral transfer patterns of exploring person-to-person and person-to-organization relationships. While exploring person-to-person and person-to-organization relationships with the help of the DHRP-BCMP with or without the KGAT-PO, both groups’ effectiveness and efficiency in exploring person-to-person and person-to-organization relationships were assessed. At the end of the experiment, the research participants of experimental and control groups were invited to complete a technology acceptance questionnaire to investigate whether there were significant differences between the two groups of research participants in terms of their overall technology acceptance, perceived usefulness, and perceived ease of use to both the systems. At the same time, semi-structured in-depth interviews were conducted to understand the research participants’ perceptions, experiences, and suggestions regarding using two different systems to assist in interpreting person-to-person and person-to-organization relationships in biographical texts, hoping these interviews would complement the quantitative data analysis. The interviews included the following questions: “Did you encounter any difficulties in using the KGAT-PO to assist in interpreting the relationships between person-to-person and people-to-organization in biographical texts?” “Did you understand how to use the user interfaces and functions of the KGAT-PO effectively?” “Do you have any suggestions for improving the KGAT-PO?”, and “Would the KGAT-PO help you interpret the person-to-person and people-to-organization relationships?.”
4.2 Experimental procedure
This study used The Biographies of Malaysian Henghua Personalities, including 85 eminent Malaysian Chinese of Henghua ancestry in different fields such as politics, business, culture, education, arts, etc. as the target biographical text to explore person-to-person and person-to-organization relationships in the experimental design. The experiment was conducted with the help of the DHRP-BCMP with or without the KGAT-PO, and the overall process of the experiment is shown in Figure 7. The experiment was divided into four stages; the total experiment time was 120 min. The experimental explanation stage lasted 10 min, aiming to explain the experimental process to the research participants and ask them to complete the experimental consent form. To avoid the experimental results being affected by the order of using two different systems, this study adopted a counterbalanced design that allowed each user to use two systems to help interpret the relationships between person-to-person and person-to-organization in biographical texts, one after the other. Therefore, the research participants were divided into A and B groups, as shown in Figure 7. In experimental stage 1, which lasted 10 min, both the experimental systems, including the design purpose of each function and its operation method, were explained to each research participant according to their assigned group. Afterward, the research participants in group A used the DHRP-BCMP with the KGAT-PO to explore economic-oriented topics. During the exploration process, they completed a personage relationship inquiry form, which focuses on potential relationships between person-to-person and person-to-organization hidden in the text related to economic-oriented topics. The research participants in group B used the DHRP-BCMP without the KGAT-PO to explore the relationships between person-to-person and person-to-organization, focusing on exploring cultural and educational topics. This step lasted 30 min. After completing the task, the two experimental groups were invited to fill out a technology acceptance questionnaire for 5 min.
Then, the experiment would proceed to experimental stage 2, where the systems used and the task orientation (economic/cultural and educational) explored by groups A and B will be the opposite of the experiment implementation stage 1. The research participants would interpret person-to-person and person-to-organization relationships and complete the personage relationship inquiry form according to their assigned task orientation for 30 min. After completing the task, the two groups of research participants were invited to fill out a technology acceptance questionnaire which took 5 min. Google Analytics recorded the two research participants’ system operating behaviors. After completing the task, the two groups of research participants were invited to conduct semi-structured interviews, and each respondent was interviewed for 20 min. We hope that analyzing the qualitative data of the interviews can supplement any shortcomings of the quantitative analysis.
4.3 Research participants
This study’s research participants are young scholars interested in conducting digital humanities research and have a certain degree of understanding of the text of The Biographies of Malaysian Henghua Personalities. Considering factors such as the cost and time of this study, this research invited 12 Malaysian Chinese with some knowledge of The Biographies of Malaysian Henghua Personalities or Malaysian local history to participate in the experiment. Among them, four were master’s students, and eight were university students; eight were male, and four were female. The age distribution of all research participants ranged from 21 to 42 years old.
4.4 Research tools
4.4.1 Knowledge graph analysis tool of people and organizations (KGAT-PO)
The KGAT-PO developed by this study used CNER technology to pre-process entity name recognition for people and organizations and then used the machine learning technology of relationship recognition to automatically determine relationships between people, as well as used part-of-speech technology based on the Chinese word segmentation system to determine relationships between people and organizations. In addition to offering a person-to-person oriented view of the relationship context, this is supplemented by providing users with a person-to-organization oriented view of the relationship context to explore the text of The Biographies of Malaysian Henghua Personalities. The visualization extracts people’s and organizations’ entity names and corresponding relationships to form a knowledge graph. It provides users with a distant reading analysis perspective and effectively allows for cross-referencing between distant and close reading through the text reading interface.
4.4.2 Personage relationship inquiry form
To record the relationships between people and organizations explored by the research participants during the experiment, this study designed two tasks for exploring the relationships between people and organizations, namely, the “personage relationship inquiry form from the cultural and educational perspective” and the “personage relationship inquiry form from the economic perspective.” The research participants in both groups were asked to complete these two forms while using the assigned tool to record information such as “entities of interest,” “descriptions of context relationships,” “key events,” and “further collation of key data” for each specific task. This study also invited two Malaysian humanities scholars familiar with The Biographies of Malaysian Henghua Personalities to jointly develop the scoring mechanism for the personage relationship inquiry form, as shown in Table 1.
4.4.3 Technology acceptance questionnaire
Referring to the technology acceptance questionnaire developed by Hwang et al. (2013), content phrases in our questionnaire were slightly adjusted according to the experimental needs of this study for investigating the technology acceptance for the use of the DHRP-BCMP with and without the KGAT-PO. The questionnaire was based on a six-point Likert scale, consisting of two major components: eight questions on “perceived usefulness” and three on “perceived ease of use,” totaling 11 questions. Regarding the reliability of the questionnaire, Cronbach’s α-value for the “perceived usefulness” component is 0.95, and Cronbach’s α-value for the “perceived ease of use” component is 0.94. Both components achieved good reliability. This study asked research participants to fill out this questionnaire after the experiment to determine the level of technology acceptance in using the DHRP-BCMP with and without the KGAT-PO to assist in interpreting person-to-person and person-to-organization relationships.
4.4.4 Google analytics
Google Analytics is a free web analytics service provided by Google that helps owners of websites track users’ operational behavior when using websites. This study used Google Analytics to record behaviors such as clicking on system functions, typing queries, and moving the mouse exhibited by research participants while using the DHRP-BCMP with or without the KGAT-PO for interpreting person-to-person and person-to-organization relationships. The recorded behavioral processes were then subjected to a Lag Sequential Analysis to explore the behavioral pattern transfer of the research participants who underwent to interpret person-to-person and person-to-organization relationships.
5. Experimental results
5.1 Evaluation of the quality of knowledge graph
While the KGAT-PO primarily comprises two functions — person-to-person relationship identification and person-to-organization relationship identification — this study focused solely on evaluating the effectiveness of the people relationship classifier for person-to-person relationship identification. The accuracy of the people relationship classifier is pivotal to the KGAT-PO because it will significantly impact the tool’s ability to assist digital humanists in interpreting people relationships based on personality biographies. The assessment was conducted using 3,881 testing data instances. Table 2 shows the performance of the people relationship classifier for recognizing person-to-person relationships, with an average precision rate of 0.81, average recall rate of 0.77, and average F1 measure of 0.79, indicating favorable performance.
5.2 Analysis of the difference in personage relationship inquiry effectiveness between the two systems
Table 3 shows the descriptive statistics of the effectiveness of personage relationship inquiry for research participants using different systems. Table 4 shows the results of the Mann-Whitney U test analysis on the effectiveness of personage relationship exploration for research participants using two systems. The results show a significant difference in the scores (U = 33.50, p = 0.026 <0.05) for the effectiveness of personage relationship exploration between research participants using two different systems. In addition, the research participants who used the DHRP-BCMP with KGAT-PO significantly outperformed those who used the DHRP-BCMP without KGAT-PO.
5.3 Analysis of the difference in personage relationship inquiry time between the two systems
This study conducted a Mann-Whitney U test to examine whether there were significant differences in the time taken to perform personage relationship inquiry among research participants using two different systems. Based on the descriptive statistics results of the average time, it took for the research participants to find personage relationships, as shown in Table 5, a Mann-Whitney U test was conducted, as shown in Table 6. The results showed that there was no significant difference in the average time taken by research participants using two different systems to perform personage relationship inquiry (U = 51.50, p = 0.23 >0.05). However, from a descriptive statistical point of view, the average time taken to perform personage relationship inquiry using the DHRP-BCMP with KGAT-PO was 293.81 s, while the average time taken using the DHRP-BCMP without KGAT-PO was 403.69 s. This indicates that using the DHRP-BCMP with KGAT-PO is more time-efficient for performing personage relationship inquiry than using the DHRP-BCMP without KGAT-PO.
5.4 Analysis of the difference in technology acceptance of personage relationship inquiry between the two systems
Table 7 shows the descriptive statistics results of technology acceptance for the research participants of both groups who used two different systems to conduct a personage relationship inquiry. The results of the Mann-Whitney U test are shown in Table 8. The results show no significant difference in the overall technology acceptance, perceived usefulness, and perceived ease of use between the research participants of the two groups using two different systems to conduct a personage relationship inquiry. This means that after using the two different systems in turn, the research participants subjectively felt that the overall technology acceptance, perceived usefulness, and perceived ease of use of the two systems were comparable. In addition, the results of the descriptive statistics show that the research participants showed a high degree of positive attitude towards the two systems in terms of overall technology acceptance, perceived usefulness, and perceived ease of use.
5.5 Behavioral differences between users operating the two systems for personage contextual inquiry
This study used Google Analytics to record the research participants’ behaviors in operating the two systems for personage relationship exploration. Lag Sequential Analysis was used to analyze sequences with significant behavioral transfers to understand effective behavioral patterns of research participants in personage relationship inquiry. The system operation behavior codes are shown in Table 9, where P represents the system operation behavior codes common to both systems, and K is the system operation behavior code specific to the KGAT-PO. Figures 8–12 show the locations of the operation behavior codes on the system user interface of DHRP-BCMP.
The sequence transfer patterns of the system operation behavior of the research participants using DHRP-BCMP with/without KGAT-PO are shown in Figures 13 and 14, respectively. Nodes in the figure represent the user’s operation behaviors, and arrows show how behaviors are transferred. The value on the arrow indicates the Z value of the behavior transfer; if Z > 1.96, it means the behavior transfer reaches a significant level. As Figure 13 shows, the research participants using DHRP-BCMP with KGAT-PO showed significant behavioral transfers, including “Browse collections (P4)” to “Read an article (P2)” (Z = 9.644 > 1.96), “Read an article (P2)” to “Click a node in a knowledge graph (K1)” (Z = 2.906 > 1.96), “Search on a knowledge graph (K4)” to “Browse collections (P4)” (Z = 5.701 > 1.96). The results show that the KGAT-PO can effectively support the research participants in performing personage relationship inquiry using knowledge graphs. As Figure 14 shows, the behavioral transform of using DHRP-BCMP without KGAT-PO was more fragmented, with significant behavioral transform from “Browse people of various professional fields (P3)” to “Read an article (P2)” (Z = 6.349 > 1.96) and from “Read an article (P2)” to “Browse people of various professional fields (P3)” (Z = 2.104 > 1.96). This study found that the research participants without the KGAT-PO support could only perform personage relationship inquiry by comparing the contents.
5.6 Analysis of interview data
To understand the research participants’ perceptions, experiences, and suggestions on personage relationship inquiry with the two systems, the 12 research participants were invited to give semi-structured in-depth interviews at the end of the experiment. Through these interviews, we obtained the research participants’ use experience and suggestions for system improvements to complement the insufficiency of quantitative data analysis.
First of all, most of the interviewees thought that, if they were unfamiliar with a biography text, the KGAT-PO was a good entry point for personage relationship exploration compared to simply reading the text and using search tools for personage relationship inquiry and that it could help to speed up both personage relationship inquiry and understanding of the overall meaning of the text. However, some interviewees also mentioned difficulties in the operation of the KGAT-PO, such as an abundance of information presented and the need for improvement in the accuracy of determining the relationships between people and the relationships between people and organizations. Further, the interviewees stated that the “search filter” function allowed them to filter for the areas they wanted to explore, that the search was intuitive, and that the content presented was apparent at a glance. However, it was also mentioned that the “search filter” provided too many categories for those who were not yet familiar with the text. In addition, regarding the “filter function,” some interviewees mentioned that it would be beneficial to filter for people or organizations after retrieval since the amount of information provided by the KGAT-PO may be too extensive when retrieving multiple people or organizations simultaneously. Filtering for people or organizations would reduce the information load and help them focus on the content of interest. However, some interviewees also mentioned that it would be helpful if more filtering options were added to the filter, such as structural stratification of organizations (e.g. Youth League Association, Malacca Youth League, Women’s League, etc.). Finally, some interviewees thought that even though the KGAT-PO provided enough visualization of the relationship between people and organizations, adding visualization of other entities, such as place names, official names or titles, etc., would make the exploration process more diverse and prosperous.
6. Discussion
To review previous research related to the automatic generation of knowledge graphs applied in digital humanities studies (Wang et al., 2023; Zheng et al., 2023a; Koho et al., 2022), this study found no research that has assessed the benefits of the developed knowledge graph tools to digital humanities studies through the use of experimental research methods with statistical analysis. This study fills the gap. This study used the Mann-Whitney U Test to analyze the effectiveness and efficiency of personage relationship inquiry, technology acceptance, and behavioral patterns with and without the help of the KGAT-PO for digital humanities study. This study found that the effectiveness of personage relationship inquiry was significantly higher with the KGAT-PO than without, indicating that the KGAT-PO could help research participants clarify personage relationships in the biography text and provide them with faster and more comprehensive information. Regarding the efficiency of the personage relationship inquiry, the results showed no significant improvement in the time efficiency of research participants who used the KGAT-PO in completing the personage relationship inquiry compared to those without. However, the average retrieval time of the research participants who used the KGAT-PO was much lower than those who did not. This indicates that using the KGAT-PO can let users grasp the overall personage relationship information more quickly and give the research participants a clearer process for personage relationship inquiry. However, some interviewees stated the need for improvement in the accuracy of determining the relationship between people. Suissa et al.’s study (2021) indicated that using deep neural networks (DNNs) for analyzing text resources in digital humanities research presents two main challenges: unavailability of training data and a need for domain adaptation. The data used for training the people relationship classifier by the BiLSTM-CRF model were obtained through a web crawler, which crawled sentences describing the relationship of people entities in Wikipedia and Baidu Baike as its training text. The BiLSTM-CRF model also has an unavailability of training data problems, thus affecting its accuracy in recognizing the relationships between people.
This study found no statistically significant differences in overall technology acceptance, perceived usefulness, and perceived ease of use among the experimental subjects with and without the KGAT-PO. The research finding is inconsistent with Chen et al.’s study (2021), indicating that the experimental subjects with the support of a digital humanities platform with a character social network relationship map tool (CSNRMT) for digital humanities study presents remarkably higher technology acceptance than the control subjects with the support of a digital humanities platform without the CSNRMT. But encouragingly, this result shows that adding a knowledge graph function did not cause users to be uncomfortable or reduce user-friendliness in the system’s operation. Moreover, the results show that the averages for overall technology acceptance, perceived usefulness, and perceived ease of use of the KGAT-PO were much higher than the median of the six-point Likert scale. This indicates that the research participants have a relatively high affirmative attitude towards their use in personage relationship inquiry.
The results of the Lag Sequential Analysis showed that the research participants using the KGAT-PO showed significant transfers in behavior from “Browse collections (P4)” to “Read an article (P2),” “Read an article (P2)” to “Click a node in a knowledge graph (K1),” “Search on a knowledge graph (K4)” to “Browse collections (P4).” In contrast, the behavior not using the KGAT-PO showed significant behavioral transfers from “Browse people of various professional fields (P3)” to “Read an article (P2)” and from “Read an article (P2)” to “Browse people of various professional fields (P3).” From the interviews, it can be surmised that the KGAT-PO can enable the research participants to accurately identify relationships between people and between people and organizations and complete a more in-depth personage relationship inquiry by searching the organization aspect. The research participants who did not use the KGAT-PO had no accurate information to help them quickly examine whether the people they were looking at met the personage relationship inquiry requirements after clicking on “People and organizations.” Therefore, they had to rely solely on reading the text, which required them to spend more time reading. However, after reading the text, they often found that they did not have the required personage relationship content, forcing them to repeatedly switch between “Browse people of various professional fields (P3)” to “Read an article (P2).” In addition, the research participants with the KGAT-PO showed a significant behavioral transfer from “Browse collections (P4)” to “Read an article (P2),” while those without did not. The interviews found that operating the KGAT-PO gave the research participants a more efficient way to complete a personage relationship inquiry process once and repeat it. However, there is no similar function without the KGAT-PO, which resulted in a fragmented user behavior of the research participants on the system, since they lacked an efficient way of personage relationship inquiry. From the above analysis, it was found that the development of a distant reading visualization tool with a knowledge graph, coupled with a close reading function that links back to the textual content, can effectively help users analyze texts in more depth, which echoes the findings of Fisher and Frey (2012). Furthermore, this study also reinforces the findings of Chen et al. (2019) and Chen et al. (2021), suggesting that conducting a user study to interpret user behaviors in a digital humanities research tool/platform based on Lag Sequential Analysis provides valuable information for understanding the problems that users encounter during the usage process.
In addition to digital humanities, it is certain that KGAT-PO can be applied to other fields that involve texts containing person-to-person and person-to-organization relationships, such as social sciences, business and marketing, and educational settings. Researchers in social science fields such as sociology, anthropology, and psychology could utilize the tool to analyze social networks and understand how individuals and organizations interact within communities or societies (Koranteng et al., 2023). Additionally, the tool can be employed in the business and marketing fields for customer relationship management (CRM) to identify connections between individuals and organizations, helping businesses tailor their marketing strategies and improve customer engagement (Gaurav, 2016). Moreover, educational institutions could use the tool to analyze interactions between students, teachers, and administrative staff, thereby improving collaboration and communication within academic communities (Zheng et al., 2023a, b).
7. Conclusion and future work
This study aimed to develop a KGAT-PO to assist humanities scholars in exploring relationships between people and between people and organizations in biographical texts. In particular, the organizational approach allows humanities scholars to explore personage relationships through distant reading and potential personage relationships hidden in these texts through personage and organizational approaches. Based on this, humanities scholars can further interpret the personage and organizational relationships in biographical texts. The analysis results show that using the KGAT-PO helps users increase the effectiveness of personage relationship inquiry, and helps them to more accurately grasp relationships between people, and relationships between people and organizations hidden in the text. In addition, research participants were generally positive about the suitability of both systems in terms of overall technology acceptance, perceived usefulness, and perceived ease of use. According to the Lag Sequential Analysis and interview data, most of the research participants felt they could accurately explore the relationships between people and between people and organizations using the knowledge graphs provided by the KGAT-PO. Further, they could complete the experimental task of personage relationship inquiry by a set of systematic combinations of behaviors. However, it took more work to find the desired content for research participants not using the KGAT-PO, which resulted in them spending more time searching for relevant personage relationship information.
Based on the above research conclusions, we propose future research directions. Firstly, currently, the entity information presented in the knowledge graph based on CNER technology includes people and organizations. In the future, we suggest giving other essential entities, such as time and place names, into the knowledge graph so that it can be more helpful for personage relationship inquiry. Secondly, the knowledge graph should be integrated with the organizational structure local to Malaysia. Several research participants from Malaysia indicated that a tightly structured organizational system characterizes the Malaysian Chinese community and that by incorporating this system into the presentation of the knowledge graph of people-organization relationships, it may be possible to develop a personage relationship inquiry tool that is better suited to the local context. Finally, we could collect more personage records or biographies of famous people from Southeast Asia on the platform to form a more extensive biography database in Southeast Asia. This will further demonstrate how knowledge graphs can assist humanities scholars in personage relationship inquiry.
The authors would like to thank the Research Center for Chinese Cultural Metaverse in Taiwan for financially supporting this research under Contract No. 113H21.
Figure 1
System architecture and operation procedures of KGAT-PO
[Figure omitted. See PDF]
Figure 2
The procedures of person-to-person and person-to-organization relationship identification
[Figure omitted. See PDF]
Figure 3
The overall user interface of the KGAT-PO
[Figure omitted. See PDF]
Figure 4
The user interface of the knowledge graph page
[Figure omitted. See PDF]
Figure 5
The user interface of the search filter function
[Figure omitted. See PDF]
Figure 6
The user interface of the relationship information page
[Figure omitted. See PDF]
Figure 7
The experimental procedures of the study
[Figure omitted. See PDF]
Figure 8
Operation behavior codes of DHRP-BCMP on the home page
[Figure omitted. See PDF]
Figure 9
Operation behavior codes of DHRP-BCMP on the full-text search page
[Figure omitted. See PDF]
Figure 10
Operation behavior codes of DHRP-BCMP on the personages of various professional fields
[Figure omitted. See PDF]
Figure 11
Operation behavior codes of DHRP-BCMP on an article page
[Figure omitted. See PDF]
Figure 12
Operation behavior codes of DHRP-BCMP on a knowledge graph page
[Figure omitted. See PDF]
Figure 13
The sequence transfer patterns of the system operation behavior of the research participants using DHRP-BCMP with KGAT-PO
[Figure omitted. See PDF]
Figure 14
The sequence transfer patterns of the system operation behavior of the research participants using DHRP-BCMP without KGAT-PO
[Figure omitted. See PDF]
Table 1
Personage relationship inquiry scoring criteria
| Scoring fields | Points | Explanation | Using the “cultural and educational perspective” of The Malaysia Henghua Personalities as an example |
|---|---|---|---|
| Number of personage relationships | 1 | Points will be awarded according to the number of personage relationships found, and one point will be awarded for each context | |
| Task-related facets of relationship descriptions | Facet 1 | ||
| 2 | The “key event” fits outside the task-related facets, and its relationship description is found by following the previous task relationship entity to a different description |
| |
| 1 | The “key event” fits inside the task-related facets |
| |
| 0 | The “key event” does not fit the task-related facets | ||
| Further collation of critical data | Facet 1 | ||
| 1 | Determination of fit for relevance for task-oriented facet | All of these figures formed a political party, the Malaysian People’s Movement Party, and then joined the National Front to improve the Chinese education system through political power. Ong Tin Kim was a lawyer by profession and then joined different political organizations to make improvements to Chinese education. (Match, therefore, is 1 point) | |
| Facet 2 | |||
| 1 | Calculation of the number of entities mentioned (limited to the entities and relationships mentioned in Stage 1) | All of these figures formed a political party, the Malaysian People’s Movement Party, and then joined the National Front to improve the Chinese education system through political power. Ong Tin Kim was a professional lawyer and then joined different political organizations to improve Chinese education. (Three entities match, therefore 3 points) | |
Source(s): Table by authors
Table 2
Performance of the people relationship classifier used for recognizing person-to-person relationships
| Relationship type | Average precision rate | Average recall rate | Average F1 measure |
|---|---|---|---|
| Couple | 0.82 | 0.79 | 0.81 |
| Parent-child | 0.84 | 0.91 | 0.87 |
| Sibling | 0.81 | 0.84 | 0.82 |
| Teacher-student | 0.71 | 0.83 | 0.76 |
| Friend | 0.82 | 0.67 | 0.73 |
| Relative | 0.83 | 0.68 | 0.75 |
| Work | 0.83 | 0.77 | 0.8 |
| Undefined | 0.72 | 0.77 | 0.75 |
| Average performance | 0.81 | 0.77 | 0.79 |
Source(s): Table by authors
Table 3
The descriptive statistics results of personage relationship inquiry evaluation score using both systems
| Item | Group | |||
|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | |||
| Mean | Standard deviation | Mean | Standard deviation | |
| Personage relationship inquiry evaluation score | 19.17 | 7.49 | 12.50 | 5.96 |
Source(s): Table by authors
Table 4
Mann-Whitney U test result of personage relationship inquiry evaluation using both systems
| Group | ||||||
|---|---|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | U | p (two-tailed) | |||
| Item | Grade mean | Grade sum | Grade mean | Grade sum | ||
| Personage relationship inquiry evaluation score | 15.71 | 188.50 | 9.29 | 111.50 | 33.50* | 0.026 |
Note(s): *p < 0.05
Source(s): Table by authors
Table 5
The descriptive statistics of personage relationship inquiry time using both systems
| Item | Group | |||
|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | |||
| Mean | Standard deviation | Mean | Standard deviation | |
| Personage contextual inquiry time | 293.81(s) | 124.45(s) | 403.69(s) | 242.70(s) |
Source(s): Table by authors
Table 6
Mann-Whitney U Test result of personage relationship inquiry time using both systems
| Item | Group | |||||
|---|---|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | U | p (two-tailed) | |||
| Grade mean | Grade sum | Grade mean | Grade sum | |||
| Personage relationship inquiry time | 10.79 | 129.50 | 14.21 | 170.50 | 51.50 | 0.23 |
Source(s): Table by authors
Table 7
The descriptive statistics of technology acceptance for the research participants of both groups who used two different systems to conduct a personage relationship inquiry
| Item | Group | |||
|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | |||
| Mean | Standard deviation | Mean | Standard deviation | |
| Overall technology acceptance | 5.30 | 0.63 | 4.82 | 0.98 |
| Perceived usefulness | 5.40 | 0.64 | 4.69 | 1.24 |
| Perceived ease of use | 5.06 | 0.66 | 5.17 | 1.00 |
Source(s): Table by authors
Table 8
Mann-Whitney U test result of technology acceptance for the research participants of both groups who used two different systems to conduct a personage relationship inquiry
| Item | Group | |||||
|---|---|---|---|---|---|---|
| DHRP-BCMP with KGAT-PO (n = 12) | DHRP-BCMP without KGAT-PO (n = 12) | U | Significance (two-tailed) | |||
| Grade mean | Grade sum | Grade mean | Grade sum | |||
| Overall technology acceptance | 14.00 | 168.00 | 11.00 | 132.00 | 54.00 | 0.297 |
| Perceived usefulness | 14.42 | 173.00 | 10.58 | 127.00 | 49.00 | 0.178 |
| Perceived ease of use | 11.17 | 134.00 | 13.83 | 166.00 | 56.50 | 0.345 |
Source(s): Table by authors
Table 9
Operation behavior code for DHRP-BCMP with/without KGAT-PO
| Code | Event description | Code | Event description |
|---|---|---|---|
| P1 | Home page | P11 | Click the keyword of the metadata field |
| P2 | Read an article | P12 | Click the developer information page |
| P3 | Browse people from various professional fields | K1 | Click the node in the knowledge graph |
| P4 | Browse collections | K2 | Click the edge in the knowledge graph |
| P5 | Use the GIS function | K3 | Click the knowledge graph page |
| P6 | Use the post-categorization function | K4 | Search on the knowledge graph |
| P7 | Full-text search | K5 | Click the search full-text button on the knowledge graph of the item page |
| P8 | Click the preface page | ||
| P9 | Click the author information page | ||
| P10 | Click the keyword on an article page | ||
Source(s): Table by authors
© Emerald Publishing Limited.