Panoramas from Photons

Scene reconstruction in the presence of high-speed motion and low illumination is important in many applications such as drone navigation, autonomous robotics, and augmented and virtual reality. Traditional motion estimation techniques fail in such conditions, suffering from too much blur in the presence of high-speed motion and strong noise in low-light conditions. Due to their high speed and extreme sensitivity, single-photon cameras have recently emerged as a promising technology capable of capturing hundreds of thousands of photon frames per second. Unfortunately, traditional computer vision techniques are not well suited for dealing with the binary-valued photon data captured by these cameras because these are corrupted by extreme Poisson noise. Here we present a method capable of estimating extreme scene motion under challenging conditions, such as low light or high dynamic range, from a sequence of high-speed image frames such as those captured by a single-photon camera. Our method relies on iteratively improving a motion estimate by grouping and aggregating frames after-the-fact, in a stratified manner. We demonstrate the creation of high-quality panoramas under fast motion and extremely low light, and super-resolution results using a custom single-photon camera prototype.