Skip navigation
Please use this identifier to cite or link to this item:
Title: Message-Passing Structures for Improved Policy Finding in Decentralized Multi-Agent Q-Learning
Authors: Dillender, Sarah
Advisors: Leonard, Naomi
Madhushani, Udari
Department: Mechanical and Aerospace Engineering
Certificate Program: Robotics & Intelligent Systems Program
Class Year: 2021
Abstract: Q-learning is a widely-applicable policy-finding algorithm for Markov decision processes. For single-agent problems, the application of Q-learning is straightforward: we iteratively update our Q-function according to the agent’s actions, rewards, and transition probabilities. The Qfunction is then guaranteed to converge to some optimal policy after sufficient iterations. However, multi-agent problems are more complex, as agents often lack full information about the system and the actions selected by each other agent. This presents a challenge for Q-learning, as the Q-learning update rule depends on full system information. One way of addressing this issue is to share knowledge about agents’ policies for action selection. However, this requires either the communication of policy parameters, which can be quite complex, or for all agents to share the same policy. We propose that a more realistic approach to Q-learning for multi-agent problems is for agents to communicate about their local actions directly through messages, which can be passed along chains of connected agents. In this way, agents have (time-delayed) information about the local actions of other agents, without needing to know any information about their policies. Significantly, such a communication policy can be implemented and will yield benefits even when agents are heterogeneous (i.e. agents can have different goals and/or policies). This work explores and simulates the efficacy of message-passing in a variety of agent organizational and communication structures, to show that these messages can indeed be used to address the problem of partial knowledge in multi-agent Q-learning.
Type of Material: Princeton University Senior Theses
Language: en
Appears in Collections:Mechanical and Aerospace Engineering, 1924-2021

Files in This Item:
File Description SizeFormat 
DILLENDER-SARAH-THESIS.pdf881.22 kBAdobe PDF    Request a copy

Items in Dataspace are protected by copyright, with all rights reserved, unless otherwise indicated.