The “LLM World of Words” English free association norms generated by large language models

Background & Summary

How is conceptual knowledge organized in the mind? Such a question has long been the focus of linguists and cognitive psychologists who aim to better understand the human language capacity^1,2. Recently, this question has become increasingly relevant in the field of artificial intelligence, particularly regarding large language models (LLMs). Human semantic memory – the repository of conceptual knowledge that encompasses how words get their meaning³ – forms the foundation of human language and thought, and thus, its structure and properties influence how we reason, form beliefs and make decisions^4,5, ultimately shaping our social and political systems. Similarly, the semantic representations that comprise the knowledge encoded in LLMs are the underlying source behind the outputs they produce, and as LLMs become more integrated into our everyday lives, these outputs have an increasing impact on society^6,7. Thus, the study of the structure and properties of semantic memory is central to understanding not only our own thinking and reasoning, but also the “thinking” and “reasoning” of LLMs, which carries important societal implications.

Studying semantic memory involves creating representations of word meanings (semantic representations), often in terms of how words relate to other words. In humans, one common way to do this is using free associations^1,8,9, which are usually accessed by prompting participants with a cue word and asking them to come up with (typically three) associated responses. Since the task is context neutral, responses represent the associative knowledge of words that we possess at an implicit level. Free associations have been extensively used in cognitive psychology and linguistics for studying lexical retrieval^1,4, semantic organization⁸, and similarity judgments^2,10,11. They have also been used for studying differences in cognitive processing between concrete and abstract words, i.e. concreteness effects¹². Given that free associations have been shown to correlate with stable implicit attitudes¹³, they have also been used for studying affective biases⁹. Investigations of conceptual knowledge using free associations are often conducted within network models of semantic memory built from free associations by connecting cue words to their responses. This results in a complex network structure of human conceptual knowledge in which words get their meanings through relationships...