Reward Sharing Relational Networks: Cultivating Socially Aligned Agents Through Multi-Agent Reinforcement Learning

Abstract

This study introduces a novel framework for guiding the behaviors of agents in Multi-Agent Reinforcement Learning (MARL) systems towards socially preferred outcomes. A big challenge in human-AI integration is to ensure AI systems can interact within the complex web of human social structures, leading to more harmonious, ethical, and effective integration of AI in everyday life by fostering a socially-aware attitude that aligns with human values and societal structures. A crucial step toward achieving this objective involves enabling AI agents to infer and incorporate social preferences pertaining to their interactions with neighboring individuals into their decision-making processes and policies. Specifically, we introduce the notion of the Reward Sharing Relational Network (RSRN), which embeds social preferences of agents directly into their learning process by means of a directed graph structure. Within this framework, the graph encodes the relational structures among agents, which in turn dictate how they value each other’s rewards and ultimately how they collaborate or compete within the environment. Additionally, we propose a systematic relational rewarding mechanism, by scalarizing neighborhood rewards, which can incorporate agents’ social preferences into the learning framework for AI agents. We examine these novel concepts in a well-known multi-agent reinforcement learning (MARL) problem, called the simple-spread navigation scenario, where a given set of agents are required to learn policies that enable them to reach pre-specified landmarks in the least amount of time. Our investigation focuses on identifying and implementing efficient mechanisms for scalarizing neighborhood rewards, which involves transforming rewards acquired from neighboring agents into a shared relational reward used for the training of each individual agent. The combination of neighborhood reward scalarization and the reward-sharing relational network constitutes what we refer to as the concept of a Relationally-Networked Decentralized Markov Decision Process (RN-Dec-MDP) within the framework of MARL.

We demonstrate the effectiveness of the Weighted Product Model (WPM) neighborhood scalarization function in comparison to two other commonly-used scalarization methods (viz., Weighted Sum Model (WSM) and MiniMax) in a 2-agent environment where agents try to capture the closest landmark and have different levels of capabilities in performing this task. Our results indicate that using the Weighted Sum Model to integrate a shared reward is ineffective in scenarios where agents have different characteristics whereas other alternative methods such as Weighted Product Model (WPM) exhibit promising performance in scenarios where some basic characteristics of the agents differ from each other. Further, we also evaluate RN-Dec-MDP framework and the WPM across 6 different RSRN structures for a 3-agent environment and analyze the resulting emergent behavior associated with the deployed social networks. The results demonstrate that the novel combination of the WPM neighborhood scalarization function and the RSRN network structures enable AI agents to learn policies that are intuitively similar to commonly observed social behaviors.