What are single-photon cameras?
Single-photon cameras are novel cameras that are extremely sensitive to light and can capture single photon arrivals. Currently, there are two main enabling technologies for large single-photon camera arrays: SPADs and jots. SPADs achieve single photon sensitivity by amplifying the weak signal from each incident photon via avalanche multiplication, which enables zero read noise and extremely high frame rate (~100kfps). Jots, on the other hand, amplify the single-photon signal by using an active pixel with high conversion gain (low capacitance). By avoiding avalanche, jots achieve smaller pixel pitch, higher quantum efficiency and lower dark current, but have lower temporal resolution.
Although the techniques proposed in this project are applicable to both SPADs and jots, we primarily focus on SPADs because of their capability to resolve fast motion due to high temporal resolution.
What is quanta burst photography (QBP)?
Quanta burst photography (QBP) is a computational photography technique that measures the motion between binary frames captured by single-photon cameras and merge them into a high-quality intensity image with low motion and blur and high dynamic range.
How is quanta burst photography different from burst mode on my smartphone camera?
Many commercial smartphone cameras such as Apple iPhone and Google Pixel support “burst mode” for capturing fast moving scenes. They capture a sequence of images in rapid succession when the user clicks a photo and rapidly merge the image sequence to produce a high quality image. Our work is inspired by ideas from burst photography. We take it to an extreme limit of individual photons per pixel in each burst frame. The extreme sensitivity and high speed of a quanta burst camera enables capturing binary frames at extremely high frame rates with sensitivity down to individual photons. This means quanta burst photography can capture fast moving objects in extremely low light which is beyond the capabilities of current smartphone cameras.
What about still scenes?
In this work we focus on scenes that contain significant motion. If we know a priori that the scene is perfectly still, then quanta burst photography will perform as well as a conventional camera assuming sufficient light in the scene. In case of extremely low illumination conditions, below the noise floor of conventional camera pixels, quanta burst photography can still provide improved performance due to its extreme sensitivity and low noise.
Is this method useful for scenarios other than low-light photography?
We have recently shown that, surprisingly, sensing individual photons can provide extended dynamic range under extremely high illumination conditions. By relying on the non-linear film-like response of these single-photon sensors, we show that it is possible to capture much higher brightness levels, orders of magnitude beyond the saturation limit of conventional image sensors. For more details please visit this project page: From One Photon to a Billion: High Flux Passive Imaging with Single-Photon Sensors.
Should I use SPADs or jots for quanta burst photography?
We compare the performance of SPADs and jots-based quanta burst photography in simulated sequences. Under fast motion, the merged image from jots contains motion blur, while SPADs are able to register the binary images and merge them into a sharp image. On the other hand, when the motion is slow, jots are able to generate a sharper image due to their high spatial resolution. Therefore, we envision these two technologies to complement each other: SPADs achieve higher performance in high-speed scenarios, while jots with projected high resolution will achieve better image quality for scenes with relatively slow motion and high-frequency texture details. Comparison between the two technologies on real data remains for future work.
Is it difficult to align the binary frames?
It is indeed difficult to match the binary frames directly due to their low dynamic range (1-bit) and highly randomness. Therefore, we divide the entire sequence into temporal blocks (e.g. every consecutive 100 frames), compute the sum image of each block and align the sum images. Then the motion field between blocks is further interpolated to get the intra-frame motion. Furthermore, a robust frequency-space merging algorithm is used to reduce the visual artifacts introduced by alignment errors.
What kinds of motion artifacts can QBP correct?
The proposed algorithm assumes certain level of spatio-temporal smoothness on the image-space motion field (patchwise 2D translation and constant velocities within blocks). When this assumption does not hold, the discrepancies between the true deformation of the patch and the translation approximation can be mitigated by the robust merging algorithm. The proposed algorithm has shown robust performance for global 6DoF handheld camera motion and moderate amount of local, non-rigid scene motion (e.g. guitar player example). An interesting future research direction is to design optical flow algorithms for aligning images for challenging scenes including fast changing global motion (e.g. drone) and highly non-rigid scene motion (e.g. cloth, air turbulence).
How fast is the algorithm?
Currently, our algorithm is not optimized for real-time implementation. Our MATLAB code takes about 30 minutes to process a sequence with 10,000 binary frames. One important future direction is to implement quanta burst photography in a fast and also energy-efficient way, which will be critical for consumer photography applications.
How much data does this method require? Is bandwidth a challenge?
Our quanta burst camera prototype captures about 100,000 frames per second. The high dynamic range and temporal resolution of SPADs comes at the cost of large bandwidth requirement. Currently, the captured binary images are stored on a local memory on the camera board, and later transferred to a PC via USB 3.0 for offline processing. The bandwidth requirement can be relaxed to some extent by capturing multi-bit images and sacrificing some temporal resolution. The bandwidth in future SPAD sensors can also be improved by using faster interfaces such as PCI Express or CameraLink.
Can quanta burst photography be used for 3D Imaging?
In this work we focus on imaging in passive illumination conditions where the camera does not have its own light source. Indeed, when used in conjunction with an active light source such as a pulsed laser, the high sensitivity and timing resolution of single-photon sensors can be used to capture high resolution 3D information about the scene. For more details about our work in 3D imaging with single-photon cameras, please visit our project page: http://www.SinglePhoton3DImaging.com
Can quanta burst cameras capture only black & white photos?
The hardware demonstration in our paper was limited to black & white photos because our prototype camera only captures monochrome images. This is not a fundamental limitation of our technique. Our technique can be easily extended to color imaging using standard color capture techniques widely used in conventional cameras (e.g., an RGB Bayer filter pattern on the image sensor).
Comments
Share This Article