Photon Inhibition for Energy-Efficient Single-Photon Imaging

Reducing avalanche energy consumption when imaging passively with SPADs

Single-photon cameras (SPCs) are emerging as sensors of choice for various challenging imaging applications. One class of SPCs based on the single-photon avalanche diode (SPAD) detects individual photons using an avalanche process; the raw photon data can then be processed to extract scene information under extremely low light, high dynamic range, and rapid motion. Yet, single-photon sensitivity in SPADs comes at a cost — each photon detection consumes more energy than that of a CMOS camera. This avalanche power significantly limits sensor resolution and could restrict widespread adoption of SPAD-based SPCs. We propose a computational-imaging approach called photon inhibition to address this challenge. Photon inhibition strategically allocates detections in space and time based on downstream inference task goals and resource constraints. We develop lightweight, on-sensor computational inhibition policies that use past photon data to disable SPAD pixels in real-time, to select the most informative future photons. As case studies, we design policies tailored for image reconstruction and edge detection, and demonstrate, both via simulations and real SPC captured data, considerable reduction in photon detections (over 90% of photons) while maintaining task performance metrics. Our work raises the question of “which photons should be detected?”, and paves the way for future energy-efficient single-photon imaging.

GitHub repository: https://github.com/lucask07/spad_photon_inhibition

Publications

Photon Inhibition for Energy-Efficient Single-Photon Imaging

Lucas J. Koerner, Shantanu Gupta, Atul Ingle, Mohit Gupta

Proc. ECCV 2024

oral presentation

pdf
@inproceedings{Koerner:2024:Inhibition,
author = {Koerner, Lucas J. and Gupta, Shantanu and Ingle, Atul and Gupta, Mohit},
year = {2024},
title = {Photon {Inhibition} for {Energy}-{Efficient} {Single}-{Photon} {Imaging}},
booktitle = {Proceedings of the {European} {Conference} on {Computer} {Vision} ({ECCV})},
}

Supplementary Video

Flux-dependent power consumption in SPAD sensors

The total energy consumption in SPAD cameras is dominated by the avalanche process employed to detect photons (see Fig. 17 of Takatsuka et al (2024) for an example). As a result, it increases substantially with scene brightness, unlike conventional CMOS sensors (CIS). This potentially limits the applicability of high-resolution SPAD cameras in resource-constrained applications.

Computationally controlled pixel states to reduce avalanche events

We propose to limit bandwidth and power consumption by incorporating “inhibition” that electronically enables or disables individual pixels, represented here via the pixel state M. Our method relies on lightweight mathematical operations called “inhibition policies” that update M based on the history of photon detections F. Inhibition policies can be optimized for image SNR or for downstream vision tasks.

Computationally imposed dead-time

The diagram illustrates one possible family of inhibition policies, where a score S is calculated based on data from a pixel (and possibly its spatial neighbors), and compared to a threshold η. If crossed, future measurements are disabled for a hold-off period τ_H. 'X' denotes a photon that arrives but for which no avalanche event is generated at the SPAD.

Power-efficient static imaging

Comparison between a simulated SPAD image with clocked-recharge exposure (a baseline mechanism for controlling avalanche power consumption,) and one with computational dead-time applied on top. The total number of photon detections (and therefore avalanche energy consumption) are kept constant. Images are simulated according to an exposure bracketing sequence with average exposure of 0.1, 1.0, 10.0 photons per pixel (ppp), and then combined using SNR_H² weighting. See Fig. 5 in the paper for more details about this experiment.

Saturation look-ahead inhibition for exposure brackets

Exposure bracketing itself can be further optimized at a higher level using our proposed saturation look-ahead inhibition policy, which ensures that the longer exposures of a bracket are not "wasted" on very bright pixels (in these regimes the SPAD nearly saturates but not much information is gained). It is implemented by thresholding the number of photons detected during short exposures, disabling the longer exposures when the threshold is crossed.

Saturation look-ahead inhibition is employed in conjunction with burst reconstruction algorithms to recover images even after reducing photon detections by over 90%. See Teaser/Supplementary video and Fig. 7 of the paper for those results and further details, respectively.