Computational Foundations of Multi-Agent Learning in Cyber-Physical-Human Networks Under Amorphous Information Attributes

Abstract

Multi-agent learning (MAL) has emerged as a promising artificial intelligence (AI) and machine learning (ML) paradigm for creating agent-based technologies to develop, operate, and secure cyber-physical-human networks (CPHNs). Studying the adaptive behaviors of intelligent agents in the presence of other agents and environmental uncertainties, MAL offers a natural set of tools and frameworks for analyzing and designing decentralized and resilient autonomous operation, leading to networked intelligence responsive to uncertainties, anomalies, and disruptions within CPHNs. However, the complex nature of CPHNs and the call for secure and resilient operation pose two primary challenges that prevent a straightforward and effective application of existing MAL methodologies.

The first challenge concerns amorphous information structures and attributes. In plain words, an information structure is a set of observable variables in MAL that specifies who can observe what and when. Early-stage MAL research focuses on structured multi-agent network systems as a jumping-off point, where agents share the same observation (symmetric information), have perfect recall of past decision-making (perfect information), and enjoy full observation of the network environment (complete information) that evolves as a stationary process (stationary information). Unlike these well-behaved testbeds, real-world CPHNs operate in the presence of complex information structures with less-structured attributes, where agents must coordinate with each other from within and defend against adversarial agents from without under a variety of information constraints imposed by network topologies, agent capabilities, and ethical standards. The paradigm shift from structured information structures to those with amorphous attributes, including asymmetric, imperfect, incomplete, and nonstationary information, calls for novel MAL approaches.

The other challenge regards the real-time learning and computation under complex information structures. Aiming to operationalize MAL in CPHNs, one must be aware of the gap between the conventional focus on equilibrium analysis of MAL in the long run and practical desideratum of online adaptation and real-time learning with certified robustness and resilience. Intelligent agents must respond agilely, adapting themselves to unexpected incidents and completing the cycle of observation, orientation, decision, and action within a short window subject to amorphous information structures.

This dissertation embarks on developing computational foundations of MAL under amorphous information structures and attributes, leveraging game theory, control and optimization theory, statistical machine learning theory, and probability theory. This dissertation is organized into seven parts in accordance with its contributions to game-theoretic modeling, algorithmic design, and theoretical analysis. Part I develops a unified multi-agent decision-making and learning framework that includes well-received game and decision-making models as special cases, leading to refined characterizations of information structures, which generalize the existing notions in game and control theory. Part II, as a preparatory step, introduces multi-scale reinforcement learning algorithms inspired by multiresolution analysis and multi-timescale stochastic approximation, which constitute the major theoretical tools for algorithm design and analysis in later parts.

Part III tackles the challenge of MAL under incomplete information, utilizing meta-learning techniques and stochastic composite optimization, offering a gradient-based online adaptation in equilibrium-seeking that is more computationally efficient than the conventional Bayesian-posterior approach. Part IV presents a novel MAL paradigm, conjectural reinforcement learning, under asymmetric information. By endogenizing the exogenous epistemic uncertainty, agents learn the optimal policy under informationally consistent subjective perceptions of the uncertainty by trial and error. Part V addresses learning under imperfect and nonstationary information, where we develop a novel equilibrium concept for generic learning dynamics, nonequilibrium, characterizing the learning agents' transient behaviors based on the finite-time sample analysis.

Part VI shifts the focus to the inverse problem of which information structure induces the desired outcomes. Leveraging bilinear programming and Lagrangian multiplier methods, we develop a computational information design framework under the constraint of transparent information disclosure, often seen in the public domain, to incentivize agents to act accordingly. Finally, Part VII concludes the dissertation by pointing out the promising development of MAL research and implementation in the age of AI.

This dissertation bridges science and technology to create theoretically certified and practically implementable solutions for secure and resilient CPHN automation. In addition to the theoretical advancements presented above, this dissertation showcases encouraging success in engineering applications based on the developed MAL computational foundations. These applications include computer and communication network security in cyberspace (Chapter 5, 6, and 7), intelligent transportation systems in physical domains (Chapter 8, 9, and 11), and misinformation security over social networks (Chapter 13 and 14). As an embodiment of system science and engineering, this dissertation modernizes the principled system thinking---rooted in the concepts of feedback and equilibrium---by integrating the latest generative AI technologies, such as transformers and large language models (Chapter 9 and 10), to achieve human-centered autonomy and emerging agency for a wide range of engineering applications in CPHNs.