Proximal Policy Optimization for Optimal Trade Execution with Price Impact

Abstract

Price impact is a consequential phenomenon in financial markets that affects traders who execute large order quantities. Due to the causality challenge, price impact is difficult to analyze with historical market data; thus, we apply reinforcement learning techniques to the trade execution problem to find optimal strategies that minimize trading costs. We describe a temporary price impact model and specify a trading environment; then, we apply a policy gradient method called proximal policy optimization (PPO). We estimate an optimal predetermined strategy based on a supplied alpha signal and designate this as the benchmark trading strategy; we then train two PPO agents under two different reward schemes. After successfully training these two agents, we find that both PPO agents outperform our benchmark and interpret results behind the reinforcement learning decisions.