Seeing Photons in Color

Megapixel single-photon avalanche diode (SPAD) arrays have been developed recently, opening up the possibility of deploying SPADs as general-purpose passive cameras for photography and computer vision. However, most previous work on SPADs has been limited to monochrome imaging.
In this work, we propose computational imaging techniques for capturing and reconstructing color images from SPAD arrays with an RGBW color filter array, especially for imaging dark, dynamic scenes.

For low light imaging, there exists a trade-off between noise and blur, which can be alleviated by burst photography approaches. The key challenge is that brightness constancy does not hold. It is difficult, if not impossible, to estimate the motion between mosaicked binary frame directly because a scene point may ``move'', for example, from a green filter to a red filter across the sequence.
To address this challenge, we develop the first burst photography algorithm for color SPADs. First, we divide the whole sequence into blocks, compute the sum of each block, and convert each such mosaicked block into a grayscale image by either downsampling or interpolating the W channel. The motion is computed between blocks and then interpolated to each frame.

The challenge for the merging problem because the raw quanta frames are heavily quantized and mosaicked, which necessitates solving the merging and demosaicking problems jointly.
We first use the estimated block-wise motion to warp the grayscale block-sum images to obtain a reference image, which is then used to guide the joint demosaicking and merging of mosaicked quanta images into full-resolution R, G, B, and W channels respectively. The four channels are then combined into an RGB image, where the luminance comes from the W channel and the chromaticity comes from the R, G and B channels. Since the W channel has higher SNR and spatial resolution, the merged image has improved spatial details and reduced noise as compared to the raw RGB channels.

In addition algorithmic approaches to reconstruct images, we also consider the problem of raw quanta frame acquisition itself: What is the right color filter array (CFA) for color single-photon imaging?
One important observation for burst photography is that, when the inter-frame motion is correctly measured and compensated, the color samples measured by each color filter are dispersed along the motion trajectories. Therefore, the color information could be sampled at a higher spatial frequency than in the original CFA. In other words, when there is motion, the color aliasing artifacts are alleviated and we can afford to have sparser color filters (with more W pixels).

Periodic CFAs such as Bayer patterns consist of blocks of fixed permutation of color filters (e.g. RGGB) that are repeated spatially, while pseudorandom CFAs contain pseudorandomly generated pixels without repetitions. The benefit of using pseudorandom patterns is that aliasing artifacts appear as incoherent noise, which is less perceptible than coherent moiré patterns generated by regular patterns. However, a completely random pattern makes demosaicking more challenging, leading to severe color artifacts. Inspired by existing work on pseudorandom RGB pattern, we design a novel blue-noise RGBW pattern, with a large fraction of W pixels (75%) that takes advantage of camera motion for burst photography.

We simulate the imaging process of single-photon sensors by using an open-source path tracer (Blender Cycles) to synthesize photorealistic scenes from high-quality 3D scene models, and then render mosaicked binary frames.
For a daytime scene with sufficient lighting, both conventional camera and SPAD are able to resolve the camera motion and reconstruct a blur-free, high-SNR image. For a night scene with both unlit regions and strong direct light sources, conventional burst photography cannot recover the entire dynamic range, while the proposed method can reconstruct both dark and bright regions. RGBW pattern recovers fine structure in the dark better than RGB pattern because of the higher transmission of W pixels.

We temporally interpolate a 1000FPS video dataset (X4K1000FPS) by a factor of 16x using RIFE and then sample binary images. We compare with five baselines. Naive average simply takes the average of either the entire sequence (long) or a single block used by the proposed method (short). Burst photography algorithms that take raw images as input cannot be directly applied to our RGBW CFA. Instead, we first apply a universal demosaic algorithm on block-sum images and then run classic (VBM4D) and pretrained learning-based (MFIR and BIPNet) burst denoising algorithms. The proposed method is able to compensate for the motion while reconstructing a clean image, which achieves best PSRN, SSIM and LPIPS.

In addition to simulation results, we also build a hardware prototype and validate our assumptions on real color SPAD data. We compare the methods on the same scene with either abundant lighting or a very dim projector. Baseline methods cannot fully remove the noise, especially in the dark scene (brightened 8.5X for visualization). In contrast, the proposed method reconstructs a clean, blur-free image.

Here we show a few scenes with challenging objects, including high-frequency fence structures, complex occlusion between plant branches, specular reflection on fake fruits, and thin fence at a different depth than the rest of the scene. The proposed method outperforms naive average and the baseline methods.

The proposed method is able to reconstruct scenes with high dynamic range. (Top) Both the sharp text in the dark and detailed shape of the cloud in the sky are reconstructed. (Bottom) Both the texture in the backdrop and the detailed reflection within the lights are reconstructed.

In addition to rigid camera motion, the proposed method is also robust to complex scene motion. (Top) Naive average of a rotating color wheel blends the colors. The proposed method reconstructs a clear image with no color blending. (Bottom) Naive average of a waving feather creates motion blur. The proposed method is robust to this nonrigid, spatially-varying motion, and generates an image with significantly reduced motion blur. The blur on the feather tip cannot be perfectly removed due to faster motion.