Resolving Motion with Single-Photon Cameras

Single-photon cameras can measure motion robustly in low-light, high-speed scenarios, which can be used to resolve the blur-noise trade-off and reconstruct high-quality images not only for human viewing, but also for subsequent computer vision tasks.

Motion estimation is important not only for human vision systems, but also for machine vision. While conventional burst imaging has achieved considerable success in resolving this trade-off, extreme conditions including high speed and low light scenarios still remain challenging. This dissertation proposes using single-photon cameras to estimate motion and resolve such trade-off. Single-photon cameras are a novel class of imaging sensors that are sensitive to the arrival of individual photons. Their negligible read noise and high temporal resolution make them the ideal sensor for burst imaging in challenging scenario. In this dissertation, three techniques are proposed. First, I introduce a burst imaging framework called quanta burst photography, which computes and compensates for the motion between binary images captured by single-photon cameras, and merge them into a single, high-quality intensity image.  Second, I extend the quanta burst photography to color imaging through a novel color filter array design with pseudorandom RGBW color filters, and adopting the burst imaging pipeline to deal with RGBW mosaicked quanta images. Third, I explore the possibility of using single-photon cameras for subsequent computer vision tasks, proposing quanta burst vision as the canonical processing paradigm for single-photon cameras. The dissertation is concluded with limitations and future outlook on single-photon cameras and burst processing schemes.

Publications

Resolving Motion with Single-Photon Cameras

Sizhuo Ma

PhD Thesis

Outstanding Graduate Student Research Award

  • pdf

  • @phdthesis{10.5555/AAI28964917,
    author = {Ma, Sizhuo},
    title = {Resolving Motion with Single-Photon Cameras},
    year = {2022},
    isbn = {9798780614869},
    publisher = {The University of Wisconsin – Madison},
    }

Seeing Photons in Color

Sizhuo Ma, Varun Sundar, Paul Mos, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2023 (ACM Trans. on Graphics)

  • pdf

  • @article{Ma_SIGGRAPH23,
    title = “Seeing Photons in Color”,
    author = “Ma, Sizhuo and Sundar, Varun and Mos, Paul and Brushini, Claudio and Charbon, Edoardo and Gupta, Mohit”,
    journal = “ACM Transactions on Graphics (TOG)”,
    year = “2023”,
    publisher = “ACM”
    }

Burst Vision Using Single-Photon Cameras

Sizhuo Ma, Paul Mos, E Charbon, Mohit Gupta

Proc. WACV 2023

  • pdf

  • @inproceedings{Ma_WACV_23,
    title={Burst Vision Using Single-Photon Cameras},
    author={Ma, Sizhuo and Mos, Paul and Charbon, E and Gupta, Mohit},
    booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
    pages={},
    year={2023}
    }

Quanta Burst Photography

Sizhuo Ma, S Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2020 (ACM Trans. on Graphics)

SIGGRAPH technical papers highlights

  • pdf

  • @article{Ma_SIGGRAPH20,
    title = “Quanta Burst Photography”,
    author = “Ma, Sizhuo and Gupta, Shantanu and Ulku, Arin C. and Brushini, Claudio and Charbon, Edoardo and Gupta, Mohit”,
    journal = “ACM Transactions on Graphics (TOG)”,
    doi = “10.1145/3386569.3392470”,
    volume = “39”,
    number = “4”,
    year = “2020”,
    month = “7”
    publisher = “ACM”
    }

Inertial Safety from Structured Light

Sizhuo Ma, Mohit Gupta

Proc. ECCV 2020

 

  • pdf

  • @inproceedings{Ma_ECCV_20,
    title={Inertial Safety from Structured Light},
    author={Ma, Sizhuo and Gupta, Mohit},
    booktitle={Proceedings of the European Conference on Computer Vision},
    pages={744-761},
    year={2020}
    }

Differential Scene Flow from Light-Field Gradients

Sizhuo Ma, Brandon M. Smith, Mohit Gupta

International Journal of Computer Vision

Special Issue on ‘Best Papers of ECCV’ (Invited Paper)

  • pdf

  • @article{Ma_IJCV2019,
    author={Sizhuo Ma and Brandon M. Smith and Mohit Gupta},
    booktitle={International Journal on Computer Vision},
    title={Differential Scene Flow from Light-Field Gradients},
    volume={128},
    number={3},
    pages={679-697},
    year={2020},
    }

3D Scene Flow from 4D Light Field Gradients

Sizhuo Ma, Brandon Smith, Mohit Gupta

Proc. ECCV 2018

oral presentation

selected for IJCV Special Issue on `Best of ECCV’

  • pdf

  • @INPROCEEDINGS{Ma_ECCV2018,
    author={Sizhuo Ma and Brandon Smith and Mohit Gupta},
    booktitle={Proc. ECCV},
    title={3D Scene Flow from 4D Light Field Gradients},
    pages={666–681},
    year={2018},
    month={September},}

Passive Single-Photon Imaging

Conventional image sensors need to collect 100-1000 photons in order to generate a clear image. Single-photon image sensors are novel sensors that are sensitive to the arrivals of single photon arrivals. They capture binary frames which record if a photon arrives at each pixel during the exposure time. If we take a continuous burst of binary frames, the average number of detected photons depends nonlinearly on the incoming light intensity, and a linear intensity image can be recovered by inverting this nonlinear response curve. However, this does not take into account possible scene/camera motion between binary frames, which results in motion blur.

Quanta Burst Photography

We propose quanta burst photography, a computational photography technique that computationally re-aligns the photons along motion trajectories, for achieving high-quality images in challenging scenarios, including low-light and high-speed motion. We develop algorithms that align the binary frames, thus creating a high-bit-depth, high-dynamic-range, super-resolved image, while minimizing noise and motion blur.

Color Photography

The proposed burst imaging technique can be extended to color photography. For acquisition, we propose a blue-noise random RGBW color filter array that is optimized for low-light burst imaging. For processing, we design a joint align-merge pipeline for mosaicked quanta images. We also propose a noise-aware visualization technique that decreases the contribution of chrominance channels on low-light images for best perceptual quality.

Quanta Burst Vision

Naive reconstruction of quanta images suffers from the blur-noise trade-off and degrades performance on downstream tasks. We propose using quanta burst photography techniques to address this problem. To evaluate the capabilities of single-photon cameras for general computer vision problems, we capture binary sequences for a wide range of tasks, consisting of over 50 million binary images in total. Tested tasks and algorithms including: QR decoding, scene text detection, object detection, SLAM, face detection, human pose estimation, action recognition, background subtraction, object tracking.

Comments

Share This Article