RGB-Thermal Fusion for Improved Object Detection

Abstract

Object detection based on RGB images alone often suffers due to a lack of illumination or other environmental conditions, while thermal infrared cameras generally succeed in those exact challenging scenarios. In the realm of RGB-thermal sensor fusion, previous research has been conducted using a variety of network architectures and fusion techniques. Using a YOLOv7 architecture, this work uses pixel-level early fusion and ensemble late fusion to compare against single-sensor models, trained and evaluated on the Teledyne FLIR dataset. Results indicate that pixel-level fusion significantly outperforms RGB and thermal models while maintaining latency in the 8 ms range on a Tesla V100 with an overall mAP of 92.8%, while the ensemble approach performed below baseline and more than doubled latency.