Why Some Are Better than Others at Detecting Social Bots: Comparing Baseline Performance to Performance with Aids and Training

Abstract

Social bots have infiltrated many social media platforms, sowing misinformation and disinformation. The harm caused by social bots depends on their ability to avoid detection by credibly impersonating human users. These three studies use a signal detection task to compare human detection of Twitter social bot personas with that of machine learning assessments. Across these studies, we find that sensitivity was (1a) minimal without training or aid, (1b) people were hesitant to respond ‘bot,’ and (1c) people were prone to “myside bias,” judging personas less critically when they shared political views. We also observed (1d) sensitivity improved when a bot detection aid was provided and (1e) when users received training focused on the objectives of social bot creators: to amplify narratives to an extensive social network. When participants labeled a persona a social bot, (2) the probability of their willingness to share its content dropped dramatically. We investigated the relationships between users’ attributes and social bot detection performance and found, (3a) social media experience did not improve detection and at times impaired it; (3b) myside bias affected the sensitivity and criterion used by liberals and conservatives differently; and (3c) analytical reasoning did not improve social bot detection, nor did it mitigate observed myside bias effects, but increased them slightly. We found that (4) people were more concerned about social bots influencing others’ online behaviors than being influenced themselves. Additionally, users’ willingness to pay for a social bot detection aid increased (5a) the more they were concerned about social bots, (5b) the greater their social media experience, (5c) the greater their sensitivity, and (5d) the higher their threshold for responding ‘bot.’ These findings demonstrate the threat posed by social bots and two interventions that may reduce them.