Re-Emergence of Gameplay Strategy in Adversarial RL Agents

Abstract

In a study by OpenAI, agents were placed in a 3D physics environment filled with simple geometrical objects and trained using reinforcement learning to play hide-and-seek. Hiders and seekers were rewarded based on whether or not they could observe each other without occlusion by other objects in the environment. Through competitive self-play, both sides developed strategies to use the objects in increasingly complex and even unforeseen ways. This suggests high potential for self-supervision in learning, both in the complexity of behaviors as well as the generality of applications. This paper attempts to explore the capabilities of self-play approach to learning. Maintaining the simplicity of the environment, agents are placed into a similar environment with new rules and rewards reflecting the game of capture-the-flag. First agents pre-trained to play hide-and-seek, then newly initialized policies are evaluated and trained in the new environment. The dynamic shifts such that agents have the same starting state and end goal. However, the agents’ initial policies differ due to the nature of their previous training. In this study, the emergent complexification and convergence of strate- gies will be examined. The generality of multi-agent co-adaptation will be shown, particularly with respect to the permissivity of the environment and the variation of game dynamics. The symptoms of transfer learning will also be examined via the familiarization of agents to new rewards and environ- ments, and heuristics to transfer policies efficiently will derive from attempts to minimize these effects.