CASPI: Collaborative Photon Processing for Active Single-Photon Imaging

Image sensors capable of capturing individual photons have made tremendous technological progress in recent years. Single-photon cameras, which once used to be a niche technology with few pixels, have now achieved kilo-to-megapixel resolution and have found their way into commercial devices such as smartphones. However, this technology faces a major limitation: because they capture scene information at the individual photon level, the raw data can be extremely sparse and noisy. Unlike conventional CMOS image sensors that have mature image processing pipelines implemented on-chip, single-photon cameras currently do not have any standard well-agreed-upon processing pipelines. Here we propose CASPI: Collaborative Photon Processing for Active Single-Photon Imaging. CASPI is a technology-agnostic, application-agnostic, tuning-free and training-free photon processing pipeline for emerging high-resolution single-photon cameras. The photon transient cubes captured by high resolution single-photon cameras include abundant spatio-temporal correlations. Our key observation is if both local and non-local correlations in the cubes are collaboratively exploited, we can estimate scene properties reliably even under extreme lighting conditions. CASPI is versatile and can be integrated into a wide range of imaging applications that could benefit from the high-speed and high-resolution single-photon camera data, including fluorescence microscopy, machine vision, and long-range 3D imaging. We experimentally demonstrate LiDAR imaging over a wide range of photon flux levels, from a sub-photon regime (<1 average signal photon per pixel) to extremely high ambient sunlight regime (>20,000 lux) and live-cell autofluorescence FLIM in extremely low photon count regimes (10 photons per pixel) where state-of-the-art methods fail to provide reliable lifetime estimates. We envision CASPI as a basic building block of general-purpose photon processing units (analogous to conventional image processing units) that will be implemented on-chip in future single-photon camera sensor hardware.


CASPI: Collaborative Photon Processing for Active Single-Photon Imaging

Jongho Lee, Atul Ingle, JV Chacko, KW Eliceiri, Mohit Gupta

Under Review


Share This Article