Tracking Multiple Objects Outside the Line of Sight using Speckle Imaging

Techniques for tracking micro-motion (<10 microns) of multiple objects, around the corner, with only low-cost components

This paper presents techniques for tracking non-line-of-sight (NLOS) objects using speckle imaging. We develop a novel speckle formation and motion model where both the sensor and the source view objects only indirectly via a diffuse wall. We show that this NLOS imaging scenario is analogous to direct LOS imaging with the wall acting as a virtual, bare (lens-less) sensor. This enables tracking of a single, rigidly moving NLOS object using existing speckle-based motion estimation techniques. However, when imaging multiple NLOS objects, the speckle components due to different objects are superimposed on the virtual bare sensor image, and cannot be analyzed separately for recovering the motion of individual objects. We develop a novel clustering algorithm based on the statistical and geometrical properties of speckle images, which enables identifying the motion trajectories of multiple, independently moving NLOS objects. We demonstrate, for the first time, tracking individual trajectories of multiple objects around a corner with extreme precision (< 10  microns) using only off-the-shelf imaging components.

Publications

Tracking Objects Outside the Line of Sight using Speckle Imaging

Brandon Smith, Matthew O'Toole, Mohit Gupta

Proc. CVPR 2018

spotlight oral

  • pdf

  • @INPROCEEDINGS{SmithOtooleGuptaCVPR2018,
    author={Brandon M. Smith and Matthew O’Toole and Mohit Gupta},
    booktitle={2018 CVF and IEEE Conference on Computer Vision and Pattern Recognition},
    title={Tracking Multiple Objects Outside the Line of Sight using Speckle Imaging},
    year={2018},
    month={June},}

CoLux: 3D Micro-Motion Analysis Using Speckle Imaging

Brandon Smith, P Desai, V Agarwal, Mohit Gupta

Proc. SIGGRAPH 2017

Finalist, WARF Innovation Awards

  • pdf

  • @article{Smith:SIGGRAPH2017,
    author = {Brandon M. Smith and Pratham Desai and Vishal Agarwal and Mohit Gupta},
    title = {CoLux: Multi-Object 3D Micro-Motion Analysis Using Speckle Imaging},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2017},
    volume = {36},
    number = {4},
    month = jul,
    year = {2017},
    articleno = {34},
    numpages = {12},
    url = {http://doi.acm.org/10.1145/3072959.3073607},
    doi = {10.1145/3072959.3073607},
    publisher = {ACM},
    address = {New York, NY, USA},
    }

SpeDo: 6 DOF Ego-Motion Sensor Using Speckle Defocus Imaging

Kensei Jo, Mohit Gupta, Shree Nayar

Proc. ICCV 2015

Video Overview and Supplementary Results

Overview

Tracking objects around a corner (left) is analogous to direct line of sight imaging, where the objects are illuminated directly by a diffuse source and imaged with a bare sensor (right).

Object motion vs. speckle motion

When an object moves laterally, the recorded speckle pattern shifts (left); when an object moves axially, the speckle pattern expands and contracts (center); and when an object rotates about the camera viewing axis, the speckle pattern rotates (right). This relationship is linear.

Ambiguity in motion histograms

Previous speckle based techniques for multi-object motion estimation compute a motion histogram where each object’s motion is represented as a peak. Since the peaks are not assigned to individual objects, motion histogram is an ambiguous representation. Different scene motions can result in the same motion histogram, which prevents tracking more than one object. In contrast, we show that it is possible to track multiple objects by developing a clustering algorithm based on the statistical properties of speckle, that assigns each peak in the motion histograms to a unique object over time.

Algorithm for labeling shift vectors

We could separate individual speckle patterns explicitly, but we take a clustering based approach based on the statistics of speckle patterns.

Bare sensor direct LOS tracking of wristwatch hands

Speckle images were recorded at one second intervals. Despite the large dynamic range of motion between second, minute, and hour hands, we were able to track all wristwatch hands simultaneously with high precision.

Isolating the primary speckle pattern

The primary speckle image encodes the object motion, and can be isolated by taking the ratio of the raw captured image and the mean image (computed by averaging all the raw captured images as the object moves).

Tracking results around the corner

Single and multiple object trajectories. Despite the macroscopic (cm) scale of the setup, we show that it is possible to recover complex, micron scale trajectories of multiple objects moving simultaneously outside the line of sight.