Seeing with Sound: Long-Range Acoustic Beamforming for Automotive Imaging and Scene Understanding

Abstract

Autonomous vehicles leverage diverse sensing modalities, including camera, lidar, and radar, that enable perception stacks to drive safety-critical decision making. As such, existing sensors primarily leverage electromagnetic waves which are undisturbed in good environmental conditions but can suffer in adverse scenarios, including low-light, inclement weather, and objects with low reflectance. Moreover, only objects in direct line-of-sight are typically detected by these existing methods. Acoustic pressure waves emanating from road users do not share these fundamental limitations. Such signals have so far been ignored in automotive perception because they suffer from low spatial resolution and lack directional information.

In this work, we depart from traditional optical sensor approaches and instead use long-range acoustic beamforming of pressure waves from noise directly produced by automotive vehicles in-the-wild for detection and localization of objects in large-scale dynamic environments. We introduce the first long-range acoustic beamforming dataset and we demonstrate acoustic beamforming’s utility for critical automotive tasks, specifically scene domain transfer, detection, and ultra-fast vision at rates up to 46kHz.