Resolving Motion with Single-Photon Cameras

Motion estimation is important not only for human vision systems, but also for machine vision. While conventional burst imaging has achieved considerable success in resolving this trade-off, extreme conditions including high speed and low light scenarios still remain challenging. This dissertation proposes using single-photon cameras to estimate motion and resolve such trade-off. Single-photon cameras are a novel class of imaging sensors that are sensitive to the arrival of individual photons. Their negligible read noise and high temporal resolution make them the ideal sensor for burst imaging in challenging scenario. In this dissertation, three techniques are proposed. First, I introduce a burst imaging framework called quanta burst photography, which computes and compensates for the motion between binary images captured by single-photon cameras, and merge them into a single, high-quality intensity image.  Second, I extend the quanta burst photography to color imaging through a novel color filter array design with pseudorandom RGBW color filters, and adopting the burst imaging pipeline to deal with RGBW mosaicked quanta images. Third, I explore the possibility of using single-photon cameras for subsequent computer vision tasks, proposing quanta burst vision as the canonical processing paradigm for single-photon cameras. The dissertation is concluded with limitations and future outlook on single-photon cameras and burst processing schemes.


Resolving Motion with Single-Photon Cameras

Sizhuo Ma

PhD Thesis

Outstanding Graduate Student Research Award

Seeing Photons in Color

Sizhuo Ma, Varun Sundar, Paul Mos, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2023 (ACM Trans. on Graphics)

Burst Vision Using Single-Photon Cameras

Sizhuo Ma, Paul Mos, E Charbon, Mohit Gupta

Proc. WACV 2023

Quanta Burst Photography

Sizhuo Ma, S Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2020 (ACM Trans. on Graphics)

SIGGRAPH technical papers highlights

University news coverage

Inertial Safety from Structured Light

Sizhuo Ma, Mohit Gupta

Proc. ECCV 2020


Differential Scene Flow from Light-Field Gradients

Sizhuo Ma, Brandon M. Smith, Mohit Gupta

International Journal of Computer Vision

Special Issue on ‘Best Papers of ECCV’ (Invited Paper)

3D Scene Flow from 4D Light Field Gradients

Sizhuo Ma, Brandon Smith, Mohit Gupta

Proc. ECCV 2018

oral presentation

selected for IJCV Special Issue on `Best of ECCV’

Passive Single-Photon Imaging

Conventional image sensors need to collect 100-1000 photons in order to generate a clear image. Single-photon image sensors are novel sensors that are sensitive to the arrivals of single photon arrivals. They capture binary frames which record if a photon arrives at each pixel during the exposure time. If we take a continuous burst of binary frames, the average number of detected photons depends nonlinearly on the incoming light intensity, and a linear intensity image can be recovered by inverting this nonlinear response curve. However, this does not take into account possible scene/camera motion between binary frames, which results in motion blur.

Quanta Burst Photography

We propose quanta burst photography, a computational photography technique that computationally re-aligns the photons along motion trajectories, for achieving high-quality images in challenging scenarios, including low-light and high-speed motion. We develop algorithms that align the binary frames, thus creating a high-bit-depth, high-dynamic-range, super-resolved image, while minimizing noise and motion blur.

Color Photography

The proposed burst imaging technique can be extended to color photography. For acquisition, we propose a blue-noise random RGBW color filter array that is optimized for low-light burst imaging. For processing, we design a joint align-merge pipeline for mosaicked quanta images. We also propose a noise-aware visualization technique that decreases the contribution of chrominance channels on low-light images for best perceptual quality.

Quanta Burst Vision

Naive reconstruction of quanta images suffers from the blur-noise trade-off and degrades performance on downstream tasks. We propose using quanta burst photography techniques to address this problem. To evaluate the capabilities of single-photon cameras for general computer vision problems, we capture binary sequences for a wide range of tasks, consisting of over 50 million binary images in total. Tested tasks and algorithms including: QR decoding, scene text detection, object detection, SLAM, face detection, human pose estimation, action recognition, background subtraction, object tracking.


Share This Article