Using Large Language Models in the Experimental Analysis of Persuasion and Bias

Abstract

The United States of America currently faces high levels of political polarization, which has resulted in political violence including the events of January 6^th, 2021. Traditional cable news networks such as FOX and MSNBC are in part responsible for this worsening polarization among American citizens. As America faces rising political tension in the 2024 election cycle, Artificial Intelligence-powered products such as Large Language Models may have the potential to pour fuel on the fire. This study uses the Relational Frame Theory of language and cognition to conceptualize how media content may contribute to polarization, and what the role of AI may be in the future of polarizing content. This study consisted of two experiments. Experiment 1 scraped transcripts from ABC, FOX, and MSNBC along with asking ChatGPT (an LLM developed by OpenAI) to produce monologues mimicking the hosts of popular shows on this network. These Real or AI transcripts were then compared using the Linguistic Inquiry and Word Count (LIWC) tool and the Inflexitext program to visually compare and contrast the linguistic elements used across these sources of media content. Experiment 2 consisted of a two by two ANOVA and ANCOVA to assess the main and interaction effects of media outlet (FOX or MSNBC) and monologue type (Real or AI) on a sample of 250 participants. The results of this experiment indicated that MSNBC monologues are perceived as more persuasive (p < .001) and credible (p < .001) than FOX monologues, and FOX monologues were viewed as more polarizing (p < .001). Additionally, MSNBC AI monologues were significantly more persuasive than MSNBC Real monologues (p < .05), indicating that LLMs may be capable of providing text that is more persuasive than real writers in some contexts. Implications for polarization in America are discussed.