CASPI: Collaborative Photon Processing for Active Single-Photon Imaging

Technology- and application-agnostic, training-free photon processing for high-resolution single-photon cameras

Image sensors capable of capturing individual photons have made tremendous technological progress in recent years. Single-photon cameras, which once used to be a niche technology with very few pixels, have now achieved kilo-to-megapixel resolution and have found their way into commercial devices such as smartphones. However, this technology faces a major limitation—because they capture scene information at the individual photon level, the raw data is extremely sparse and noisy. Unlike conventional CMOS image sensors that have mature image processing pipelines implemented on-chip, single-photon cameras currently do not have any standard well-agreed-upon processing pipelines. Here we propose CASPI: Collaborative Photon Processing for Active Single-Photon Imaging. CASPI is a technology-agnostic, application-agnostic, training-free, and blind photon processing pipeline for emerging high-resolution single-photon cameras. CASPI exploits the abundant spatio-temporal correlations that are present in the raw photon data cubes captured by a single-photon camera. Our key observation is the if both local and non-local correlations in the cubes are collaboratively exploited, we can estimate scene properties reliably even under extreme illumination conditions. CASPI is versatile and can be integrated into a wide range of imaging applications that could benefit from the high-speed and high-resolution single-photon camera data, including fluorescence microscopy, machine vision, and long-range 3D imaging. We experimentally demonstrate LiDAR imaging over a wide range of photon flux levels, from a sub-photon regime (<1 average signal photon per pixel) to extremely high ambient sunlight regime (>20,000 lux) and live-cell autofluorescence FLIM in extremely low photon count regimes (10 photons per pixel) where state-of-the-art methods fail to provide reliable lifetime estimates. We envision CASPI as a basic building block of general-purpose photon processing units (analogous to conventional image processing units) that will be implemented on-chip in future single-photon camera sensor hardware.

Publications

CASPI: Collaborative Photon Processing for Single-Photon Imaging

Jongho Lee, Atul Ingle, J V Chacko, K W Eliceiri, Mohit Gupta

Nature Communications 2023

Active imaging can operate reliably over a limited range of photon flux levels

a. In active imaging, a camera operates in synchronization with a controllable light source (e.g., a laser) to probe various scene properties such as fluorescence lifetimes and 3D depths.

b. A single-photon camera-based active imaging system can operate reliably over a limited range of photon flux levels. In low signal conditions, it suffers from strong noise due to poor signal-to-noise ratio, whereas in high illumination conditions, it suffers from severely distorted measurements, resulting in large errors in estimated depths and fluorescence lifetimes.

CASPI is a versatile photon data processing technique for active imaging applications

We propose CASPI, a versatile photon processing method that enables reliable scene property estimation in such challenging lighting conditions for a wide range of applications including (a) 3D scene recovery and (b) fluorescence lifetime microscopy.

Spatio-temporal photon correlations in photon transient cubes

When stronger correlations are available in the photon data, we can suppress noise effectively relative to the signal in the Fourier domain, leading to better signal and noise separation by accurate noise estimation.

Hierarchical blind photon processing

It is challenging to use non-local correlations directly in challenging illumination conditions due to severe noise. To address this problem, we propose a hierarchical approach: We recover photon fluxes using only local correlations first, and after finding similar cubelets, final photon fluxes are recovered by exploiting local and non-local correlations collaboratively.

CASPI for single-photon LiDAR

a. LiDAR imaging via CASPI is compared with matched filtering (MF), a statistical approach, and a learning-based approach in various illumination conditions using data from the Middlebury dataset and CARLA autonomous-driving simulator. The three numbers underneath each depth map show the percent fraction of inlier pixels that lie within 0.2%, 0.5%, and 1% of the true depths. As shown in the third row, CASPI can enhance the performance of existing methods.

b. CASPI recovers the latent photon fluxes over various flux regimes and enables reliable depth estimation even in challenging conditions.

Comparisons with volume data filtering approaches

State-of-the-art volumetric data denoising approaches fail to recover true photon fluxes in challenging flux regimes, which leads to unreliable depth estimates.

CASPI for single-photon LiDAR experiments

Our approach succeeds in recovering 3D geometry both in (a) high background flux and (b) sub-photon regimes where the conventional approaches fail. CASPI is robust to non-idealities in real-world experimental datasets (e.g., non-Gaussian bi-modal laser pulse shapes shown in the last row). This demonstrates the practical versatility of our approach across a wide range of operating conditions.

Recovering multipath transients

The kitchen scene was simulated using a photo-realistic graphics rendering engine which emulates active single-photon imaging, and includes multipath effects that cause the ground-truth photon transients to deviate significantly from an ideal Gaussian pulse shape. CASPI recovers the entire temporal profiles of light transport including indirect multipath reflections as well as direct reflections. (a) The top row shows the measured photon transient cube with as few as average 10 signal photons/pixel and the photon fluxes recovered by CASPI. (b) The second and third rows show accurate recovery of a variety of transients that include multiple peaks due to multipath effects.

CASPI for FLIM

CASPI enables reliable lifetime estimates with as few as 10 photons per pixel and achieves 5 times better performance in root-mean-square error (RMSE) compared to spatial binning of the photon transient cubes followed by BM3D applied to the lifetime estimates. The sample imaged here contains fixed BPAE endothelial cells with fluorescent labels. DAPI stained nuclei and mitotracker stained mitochondrial structures are separable using CASPI even with 50 times fewer photons than the ground-truth.

Autofluorescence FLIM

CASPI recovers the underlying transients from the autofluorescence emission (last rows for a and b) from the low photon count datasets of autofluorescence of biological samples. When combined with existing fitting methods (a) state-of-the-art maximum likelihood estimation (MLE), or (b) naive linear-fit on log-transformed histograms), CASPI enables to recover fine structures and details even for moving living cells (see also, Supplementary Video).

Comparisons with global analysis for bi-exponential decay in FLIM

a. The ground-truth used for simulating photon transient cubes consists of two invariant lifetimes t1 = 3ns and t2 = 1.5ns with relative contributions beta1=0.1, 0.5, 0.9 and beta2 = 1 - beta1 that vary over the field-of-view.

b. When estimating the parameters of a multi-exponential decay model, pixel-wise fitting is often unreliable if fewer than 1,000 photons are available per pixel.

c. Global analysis provides better estimation accuracy than pixel-wise fitting by considering all photon transients simultaneously assuming that the lifetimes are spatially invariant.

d. CASPI can reliably estimate the parameters of a bi-exponential decay model in FLIM even with as few as 200 photons per pixel without the spatial-invariance assumption that global analysis relies on. After applying CASPI, even a pixel-wise fitting provides better estimates than global analysis by 5 times in RMSE.