Chemotactic navigation in robotic swimmers via reset-free hierarchical reinforcement learning

Introduction

Throughout biological evolution, organisms ranging from single cells to multicellular species have evolved diverse strategies in response to environmental stimuli, collectively termed taxis¹. These responses include directed movement toward or away from cues such as light (phototaxis)², temperature (thermotaxis)³, chemical gradients (chemotaxis)^4,5, and many others. Of these, chemotaxis emerges as one of the most phylogenetically widespread and functionally critical strategies, underpinning essential biological processes, for instance: (1) immune response—neutrophils migrate toward infection sites by sensing chemoattractants like interleukin-8 (IL-8) released by damaged tissues or pathogens⁶; (2) microbial survival—Escherichia coli execute a run-and-tumble movement to climb nutrient gradients for optimal foraging; and 3) reproductive success—mammalian sperms navigate the female reproductive tract to the egg by tracing the gradients of its secreta⁷.

Chemotaxis is a feedback-regulated tactic employed by organisms. They sense chemical gradients through receptors and dynamically adjust their actions via feedback mechanisms, thereby enabling fitness-enhancing directed movement. This strategy of feedback regulation is not unique to chemotaxis or other tactic behaviors, but reflects a universal principle of natural intelligence shaped by evolution and natural selection⁸. In parallel, this feedback-regulated principle is equally widespread in engineered systems, its logic foreshadowing the rise of artificial intelligence (AI) born of human ingenuity.

Despite longstanding recognition of chemotaxis, synthetic systems that exhibit chemotactic intelligence remain rare^9,10. Beyond the technical hurdles of engineering such systems, especially at scales comparable to those of chemotactic organisms, a lingering question persists: Can the evolutionary wisdom underlying biological taxis be meaningfully translated into engineering platforms? Modern AI, with its capacity to decipher and emulate biological feedback architectures, may hold the key to bridging this gap¹¹. However, it remains elusive how AI might endow synthetic agents with such evolution-informed natural intelligence.

To explore these questions, we leverage reinforcement learning (RL)—an AI framework—to achieve chemotaxis in bio-inspired model swimmers within viscous fluids. Mimicking the goal-oriented, trial-and-error process fundamental to biological learning, RL enables an artificial agent to iteratively refine its behavior through environmental interactions¹²: sensing its surroundings, executing feedback-regulated actions, and continually optimizing strategies to maximize a cumulative goal-encoding reward. RL-based design and manipulation of robotic swimmers is an emerging field^{13, 14, 15,...}