Inter-photon timing measurements captured by a passive single-photon sensitive camera enable unprecedented dynamic range
Digital camera pixels capture images by converting incident light energy into an analog electrical current, and then digitizing it into a binary representation. This direct measurement method, while conceptually simple, suffers from limited dynamic range: electronic noise dominates under low illumination, and the pixel’s finite full-well capacity causes saturation under bright illumination. Here we show that inter-photon timing information captured by a dead-time-limited single-photon detector can be used to estimate scene intensity over a much higher range of brightness levels. We experimentally demonstrate imaging scenes with a dynamic range of over ten million to one. The proposed techniques, aided by the emergence of single-photon sensors such as single-photon avalanche diodes (SPADs) with picosecond timing resolution, will have implications for a wide range of imaging applications: robotics, consumer photography, astronomy, microscopy and biomedical imaging.
Proc. CVPR 2021
oral presentation
@inproceedings{Ingle_CVPR_21,
title={Passive Inter-Photon Imaging},
author={Ingle, Atul and Seets, Trevor and Buttafava, M and Gupta, Shantanu and Tosi, Alberto and Velten, Andreas and Gupta, Mohit},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={},
year={2021}
}
Proc. CVPR 2019
oral presentation
@INPROCEEDINGS{Ingle_CVPR2019,
author={Atul Ingle and Andreas Velten and Mohit Gupta},
booktitle={Proc. CVPR},
title={High Flux Passive Imaging with Single Photon Sensors},
year={2019},
month={June},}
Proc. SIGGRAPH 2020
SIGGRAPH technical papers highlights
@article{Ma_SIGGRAPH20,
title = “Quanta Burst Photography”,
author = “Ma, Sizhuo and Gupta, Shantanu and Ulku, Arin C. and Brushini, Claudio and Charbon, Edoardo and Gupta, Mohit”,
journal = “ACM Transactions on Graphics (TOG)”,
doi = “10.1145/3386569.3392470”,
volume = “39”,
number = “4”,
year = “2020”,
month = “7”
publisher = “ACM”
}
Proc. ICCV 2019
Marr Prize (best paper) honorable mention
oral presentation
@INPROCEEDINGS{Gupta_ICCV2019,
author={Anant Gupta and Atul Ingle and Mohit Gupta},
booktitle={Proc. ICCV},
title={Asynchronous Single-Photon 3D Imaging},
year={2019},
month={October},}
Proc. CVPR 2019
oral presentation
@INPROCEEDINGS{Gupta_CVPR2019,
author={Anant Gupta and Atul Ingle and Andreas Velten and Mohit Gupta},
booktitle={Proc. CVPR},
title={Photon-Flooded Single-Photon 3D Cameras},
year={2019},
month={June},}
There is a non-linear relationship between the number of incident and detected photons for a dead-time limited SPAD pixel.
If we rely solely on discrete photon counts, at extremely high flux, we suffer from a quantization artifact: a large range of incident flux values may map to the same photon count.
But, if we measure the inter-photon timing directly, we can distinguish between two similar flux values. Higher flux values result in smaller average inter-photon timing even if the resulting photon counts are the same.
This tabletop scene has over a million-to-one dynamic range. A conventional monochrome CMOS camera cannot reliably capture this in a single exposure. Our inter-photon timing method (IP-SPAD) captures this in a single shot, including fine details on the bright halogen lamp and text in the dark tunnel.
If we only use photon counts (PF-SPAD) we still obtain higher dynamic range than a conventional CMOS camera, but notice that the extremely bright pixels around the halogen lamp suffer from "soft saturation" artifacts. The photon flux at these pixels is so high that it results in the same (high) photon count. However, using the small variations in average inter-photon timing the IP-SPAD method can discern different brightness levels in this extremely high flux regime. Observe the fine details around the halogen lamp.
Photon timing information can enable image reconstruction with extremely low photon counts, as low as 1 and 10 photons per pixel. Two denoising methods are shown here: an off-the-shelf BM3D denoiser and a kernel-prediction-network (KPN) based neural network denoiser. Quite remarkably, with as few as 10 photons per pixel, details such as facial features and text on the fire truck are discernible.
A conventional camera pixel is like a “light bucket”. It collects photons for a fixed exposure time. The bucket fills up proportionally to the scene brightness. The pixel digitizes the charge level in the bucket which is read-out in a digital (e.g. 8-bit or 16-bit) format after analog-to-digital conversion.
Conventional cameras measure scene brightness by capturing photon counts over a fixed exposure time. In contrast, we use a single-photon sensitive image sensor called single-photon avalanche diode (SPAD) which captures precise inter-photon timing information with sub-nano-second resolution. Intuitively, the higher the brightness, the smaller the measured inter-photon time.
Unlike conventional image sensor pixels that produce a signal that is linearly proportional to the scene brightness, a SPAD pixel has a non-linear response curve. The scene pixel brightness is inversely proportional to the average inter-photon times measured by the SPAD pixel. At extremely brightness levels, this response curve flattens out, approaching an asymptotic saturation limit.
Brightness estimated from just photon counts over a fixed exposure time is indeed a good approximation for the true inter-photon timing in low-to-moderate flux values. However, the key difference is at high flux. This is due to the following reason:
If we rely just on discrete photon counts, at extremely high flux, we suffer from a quantization artifact. This is because there is a non-linear relationship between the number of incident and detected photons for a dead-time limited SPAD. Therefore, a large range of incident flux values may map to the same photon count.
On the other hand, if we can measure the inter-photon timing directly, we can distinguish between two similar flux values, even if they give the same photon count: the higher flux results in smaller average inter-photon timing and vice versa.
Our previous work on passive free-running SPAD (PF-SPAD) imaging proposes a photon flux estimator that only uses photon counts. Due to its non-linear response curve, the counts-based PF-SPAD brightness estimator suffers from a quantization effect in extremely high brightness settings. The new IP-SPAD method overcomes this limitation by using precise photon timing information which, unlike photon counts, is inherently continuous-valued. This extends the dynamic range of the IP-SPAD well beyond the quantization limit of a PF-SPAD.
Although our IP-SPAD method can give extremely high dynamic range, it is not infinite because in reality the photon timestamps cannot be measured with infinite precision. Due to internal sources of noise and jitter, a SPAD can only measure photon timing with ~100’s of picosecond resolution. As a result, we observe quantization artifacts if the true brightness is so high that the inter-photon times vary on time-scales smaller than this resolution. Our method is also quite sensitive to variations in the pixel’s dead-time and sensitivity. Moreover, our experiment proof-of-concept is limited to just one camera pixel. Scaling this to large arrays will introduce additional design challenges and noise sources.
The goal of this work is to build video cameras whose spatial and temporal resolutions…
Multi-camera interference (MCI) is an important challenge faced by continuous-wave time-of-flight (C-ToF) cameras. In the…
Structured light 3D scanning systems are fundamentally constrained by limited sensor bandwidth and light source…
Comments
Share This Article