Enhancing Topic Interpretability with ChatGPT: A Dual Evaluation of Keyword and Context-Based Labeling

Abstract

Accurate topic labeling is essential for structuring and interpreting large-scale textual data across various domains. Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA), effectively extract topic-related keywords but lack the capability to generate semantically meaningful and contextually appropriate labels. This study investigates the integration of a large language model (LLM), specifically ChatGPT, as an automatic topic label generator. A dual evaluation frame-work was employed, combining keyword-based and context-based assessments. In the keyword-based evaluation, domain experts reviewed ChatGPT-generated labels for semantic relevance using LDA-derived keywords. In the context-based evaluation, experts rated the alignment between ChatGPT-assigned topic labels and actual content from representative sample posts. The findings demonstrate strong agreement between AI-generated labels and human judgments in both dimensions, with high inter-rater reliability and consistent contextual relevance for several topics. These results underscore the potential of LLMs to enhance both the coherence and interpretability of topic modeling outputs. The study highlights the value of incorporating context in evaluating automated topic labeling and affirms ChatGPT’s viability as a scalable, efficient alternative to manual topic interpretation in research, business intelligence, and content management systems.