Light-in-Flight for a World-in-Motion

I-ToF cameras capture a set of correlation images of the scenes to estimate their depth and intensity images. Although I-ToF cameras provide correct depth and intensity information for (a) static scenes, they suffer from motion artifacts for (b) dynamic scenes due to misalignment between the correlation images.

All correlation images in each set have different pixel values (depicted as distinct colors) along the true XY-motion (∆X, ∆Y ), posing a challenge for motion estimation. However, under the small and linear motion, the spatial gradient of the intensity image obtained from each correlation image set maintains its pixel values (represented by the same color) along the motion, facilitating XY-motion estimation. Two depth values (one from each set) can be obtained along the estimated XY-motion, and the Z-motion (∆Z) can be simply derived from their difference.

Velocity standard deviations by Doppler ToF (σ_v∆f ) and by depth difference (σ_v∆Z ) are compared. Doppler ToF shows about 40 times higher standard deviation in the given practical conditions, and its estimation becomes very unreliable at certain modulation frequencies and depth values.

Although multi-frequency coding achieves lower depth errors than single-frequency coding, it fails to decode the correct depths under extremely low SNR conditions. The performance of multi-frequency coding can be improved when combined with burst denoising, which reduces the depth noise in a complementary way. The three numbers inside each error map show the percent fraction of inlier pixels that lie within 0.5, 1, and 2 % of the true depths.

Depth estimates of the dynamic scene obtained via conventional I-ToF imaging suffer from noise or motion artifacts. In contrast, our approach recovers high-quality 3D geometry without such artifacts. The three numbers underneath each depth map show the percent fraction of inlier pixels that lie within 0.5, 1, and 2 % of the true depths.

Our approach can estimate dense and high-quality 3D motions for various dynamic scenes. The exact motion scenarios can be identified correctly from our XY- and Z-motion estimates.

Doppler ToF cannot estimate small Z-motions (∼ 4 m/s) accurately since the corresponding Doppler frequency shifts (< 1 Hz) are negligibly small compared to the modulation frequency (MHz). In contrast, our approach can resolve even these small Z-motions reliably with lower source power.

3D geometry and intensity estimates by conventional I-ToF imaging suffer from low SNR and motion artifacts in both (a) indoor and (b) outdoor dynamic scenes. In contrast, our approach can recover high-quality and motion artifact-free estimates.

Our approach can recover 3D motions reliably for (a) indoor and (b) outdoor dynamic scenes. Our approach can recover local and global motions under challenging conditions such as a low scene albedo and a thin object.

Comparisons between conventional and proposed I-ToF imaging for dynamic scenes through simulations. While conventional I-ToF imaging suffers from motion artifacts or low SNR depending on the integration time, our approach can recover not only high-quality 3D geometry and intensity but also 3D motion using a single I-ToF camera.

Comparisons between conventional and proposed I-ToF imaging for dynamic scenes through real experiments. While conventional I-ToF imaging suffers from motion artifacts or low SNR depending on the integration time, our approach can recover not only high-quality 3D geometry and intensity but also 3D motion using a single I-ToF camera.

3D motion estimation results and corresponding motion histograms when a hand moves along the X-, Y-, and Z-directions. Each hand gesture is represented distinctively by motion histograms, allowing us to classify the hand gestures correctly based on the 3D motion estimation results from our approach.