3D Micro-Motion Analysis Using Speckle Imaging

Novel low-cost speckle imaging techniques that can measure micro motions of multiple, independently moving objects

We present CoLux, a novel system for measuring micro 3D motion of multiple independently moving objects at macroscopic standoff distances. CoLux is based on speckle imaging, where the scene is illuminated with a coherent light source and imaged with a camera. Coherent light, on interacting with optically rough surfaces, creates a high-frequency speckle pattern in the captured images. The motion of objects results in movement of speckle, which can be measured to estimate the object motion. Speckle imaging is widely used for micro-motion estimation in several applications, including industrial inspection, scientific imaging, and user interfaces (e.g., optical mice). However, current speckle imaging methods are largely limited to measuring 2D motion (parallel to the sensor image plane) of a single rigid object. We develop a novel theoretical model for speckle movement due to multi-object motion, and present a simple technique based on global scale-space speckle motion analysis for measuring small (5-50 microns) compound motion of multiple objects, along all three axes. Using these tools, we develop a method for measuring 3D micro-motion histograms of multiple independently moving objects, without tracking the individual motion trajectories. In order to demonstrate the capabilities of CoLux, we develop a hardware prototype and a proof-of-concept subtle hand gesture recognition system with a broad range of potential applications in user interfaces and interactive computer graphics.

Publications

CoLux: 3D Micro-Motion Analysis Using Speckle Imaging

Brandon Smith, P Desai, V Agarwal, Mohit Gupta

Proc. SIGGRAPH 2017

Finalist, WARF Innovation Awards

pdf
@article{Smith:SIGGRAPH2017,
author = {Brandon M. Smith and Pratham Desai and Vishal Agarwal and Mohit Gupta},
title = {CoLux: Multi-Object 3D Micro-Motion Analysis Using Speckle Imaging},
journal = {ACM Trans. Graph.},
issue_date = {July 2017},
volume = {36},
number = {4},
month = jul,
year = {2017},
articleno = {34},
numpages = {12},
url = {http://doi.acm.org/10.1145/3072959.3073607},
doi = {10.1145/3072959.3073607},
publisher = {ACM},
address = {New York, NY, USA},
}

Tracking Objects Outside the Line of Sight using Speckle Imaging

Brandon Smith, Matthew O'Toole, Mohit Gupta

Proc. CVPR 2018

spotlight oral

pdf
@INPROCEEDINGS{SmithOtooleGuptaCVPR2018,
author={Brandon M. Smith and Matthew O’Toole and Mohit Gupta},
booktitle={2018 CVF and IEEE Conference on Computer Vision and Pattern Recognition},
title={Tracking Multiple Objects Outside the Line of Sight using Speckle Imaging},
year={2018},
month={June},}

SpeDo: 6 DOF Ego-Motion Sensor Using Speckle Defocus Imaging

Kensei Jo, Mohit Gupta, Shree Nayar

Proc. ICCV 2015

Video Overview

Speckle Formation and Motion Models

(a) Speckle formation model. A bare sensor images a surface illuminated by a coherent light source located at L, which is approximately co-located with the sensor. (b) Speckle motion model for rigid object motion. As the object moves in 3D, the speckle pattern recorded by the sensor also moves. (c) Qualitative depiction of speckle motion. Lateral object motion results in speckle pattern shift. Axial motion results in speckle image contraction or expansion.

Speckle-Based Axial Motion Measurement

I^scale_χ (orange speckles) is a scaled version of I (blue speckles), which means I^scale_χ is most like a delta function when χ equals the correct scale factor hat{χ} such that I^scale_hat{χ} = I.

Multi-Object Motion

(a) microscopic movement of two objects. (b) The observed speckle image I_tot is the summation of the speckle images I_1 and I_2 due to Object 1 and Object 2, respectively. (c) Observed speckle image I'_tot after object motion. The microscopic movements of Object 1 and Object 2 result in macroscopically shifted speckle images I'_1 and I'_2. The individual speckle images are artificially colored here for illustration purposes; in reality, their color is uniform and reflects the wavelength of the light source. (d) The 2D cross-correlation result of I_1 and I'_1 due to Object 1 motion. (e) The 2D cross-correlation result of I_2 and I'_2 due to Object 2 motion. (f) The 2D cross-correlation result of I_tot * I'_tot due to both motions. I_1 and I'_1 are uncorrelated with I_2 and I'_2, which is crucial for measuring multiple motions simultaneously.

3D Motion Histogram

(a) The scene: three small objects moving along x, y, and z axes. (b) Two views of the 3D motion histogram showing the three different motion modes. (c) 1D motion histograms along each axis for each object. CoLux can recover multi-object motion with extremely high accuracy. Here the bins are set to 25μm x 25μm x 200μm for illustration purposes; average error is <5 μm laterally and <50 μm axially.

Motion Sensitivity

Lateral (x axis) motion sensitivity. Left: lateral motion along the x axis was computed by optical flow using images from a conventional camera. Right: CoLux measures lateral object motion with an order of magnitude more sensitivity than a conventional camera. The lateral motion sensitivity ratio is approximately 60:1.

3D Motion Histograms for Different Gestures

Each cell represents 20 μm in each direction. CoLux correctly measures a single motion mode for the Swipe Down gesture (one finger), two motion modes for Horizontal Slide (two fingers), and multiple motion modes for Inflate (all fingers).

Experimental Equipment Used for Controlled Motion Experiments

Left: A small object (5-mm diameter piece of white chalk) mounted to a linear translation stage is positioned 50 cm from the sensor on an optics table. The sensor on the left of the laser is bare (lensless) except for a 532 nm bandpassfilter, which is matched to the coherent light source. A conventional camera with a 16-mm lens is placed on the other side of the laser. We used the conventional camera only for visualization purposes and to measure the motion sensitivity ratio. A conventional camera is otherwise unnecessary; the CoLux sensor works without it. Right: Front view of our experimental hardware (a conventional camera and the CoLux sensor).