2023

CASPI: Collaborative Photon Processing for Single-Photon Imaging

Technology- and application-agnostic, training-free photon processing for high-resolution single-photon cameras

Jongho Lee, Atul Ingle, J V Chacko, K W Eliceiri, Mohit Gupta

Nature Communications 2023

Software-defined single-photon imaging

SoDaCam: Software-defined Cameras via Single-Photon Imaging

The data acquired by single-photon sensors can be used to emulate a wide variety of imaging modalities.

Varun Sundar, Andrei Ardelean, Tristan Swedish, C Bruschini, Edoardo Charbon, Mohit Gupta

Proc. ICCV 2023

oral presentation

Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers

Reducing the cost of vision Transformers for video inference, by exploiting temporal redundancy between subsequent inputs
GitHub: PyTorch

Matthew Dutson, Yin Li, Mohit Gupta

Proc. ICCV 2023

Computational 3D Imaging with Position Sensors

Low-bandwidth computational imaging and 3D sensing with a novel class of position sensors

Jeremy Klotz, Mohit Gupta, Aswin Sankaranarayanan

Proc. ICCV 2023

oral presentation

Learned Compressive Representations for Single-Photon 3D Imaging

A framework to compress the spatio-temporal information of photon timestamp data into a compressive histogram, and a method to integrate the compressive histogram with existing SPAD data denoising models based on 3D CNNs.

Project Page

 

Felipe Gutierrez-Barragan, Fangzhou Mu, Andrei Ardelean, Atul Ingle, Claudio Bruschini, Edoardo Charbon, Yin Li, Mohit Gupta, Andreas Velten

Proc. ICCV 2023

Eulerian Single-Photon Vision

Fast low-level vision in low light with single-photon cameras

Shantanu Gupta, Mohit Gupta

Proc. ICCV 2023

Panoramas from Photons

High-quality single photon panoramas under fast motion and low-light

Sacha Jungerman, Atul Ingle, Mohit Gupta

Proc. ICCV 2023

Mitigating AC and DC Interference in Multi-ToF-Camera Environments

Novel approach for mitigating multi-ToF-Camera interference by reducing both AC and DC interference

Jongho Lee, Mohit Gupta

IEEE Trans. on PAMI 2023

Seeing Photons in Color

Color filter and algorithm design for single-photon color imaging in low light

Sizhuo Ma, Varun Sundar, Paul Mos, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2023 (ACM Trans. on Graphics)

Light Codes for Fast Two-Way Human-Centric Visual Communication

A human-centric communication modality that enables fast two-way exchange of visual code-like information among users

Mohit Gupta, Jian Wang, Karl Bayer, Shree Nayar

ACM Trans. on Graphics, presented at SIGGRAPH 2024

Unlocking the Performance of Proximity Sensors by Transient Histograms

Utilizing Ultra Low-Cost Single-Photon Proximity Sensors for Robotics Manipulation Tasks by Utilizing Transient Histograms

Project Page

Carter Sifferman, Yeping Wang, Mohit Gupta, Michael Gleicher

IEEE Robotics and Automation Letters (RAL)

Proc. IEEE ICRA 2024

When Two Cameras Are a Crowd (Understanding and Handling Interference Across Multiple Active Cameras)

Novel approach for mitigating multi-ToF-Camera interference by reducing both AC and DC interference

Jongho Lee, Mohit Gupta, Bhuvana Krishnaswamy, Suman Banerjee

Communications of the ACM 2023

QfaR: Location-Guided Scanning of Visual Codes from Long Distances

Location-Guided Scanning of Visual Codes from Long Distances

Project Page

 

Sizhuo Ma, Jian Wang, Wenzheng Chen, Suman Banerjee, Mohit Gupta, Shree Nayar

Proc. ACM MobiCom 2023

Spike-Based Anytime Perception

Moving beyond frame-based visual recognition, targeting applications with tight latency and power constraints
GitHub: TensorFlow

Matthew Dutson, Yin Li, Mohit Gupta

Proc. WACV 2023

Burst Vision Using Single-Photon Cameras

Exploring the capabilities of SPAD sensors for a wide gamut of real-world computer vision tasks including object detection, pose estimation, SLAM, text recognition and so on.

Sizhuo Ma, Paul Mos, E Charbon, Mohit Gupta

Proc. WACV 2023

2022

Event Neural Networks

Leveraging temporal redundancy for efficient visual recognition on video and other time-series data
GitHub: PyTorch, TensorFlow

Matthew Dutson, Yin Li, Mohit Gupta

Proc. ECCV 2022

Fig. 1. Transient histogram: Measurement and examples. (a) Measuring a transient histogram involves illuminating the scene with a pulsed illumination source such as a pulsed laser which is diffused uniformly over the field of view of a time-resolved image sensor pixel such as a single-photon avalanche diode. (b) Transient histograms were captured using a hardware prototype for some basic 3D shapes. Observe that these have unique features which can help distinguish these shapes.

3D Scene Inference from Transient Histograms

Time-critical and resource-constrained 3D scene inference using transient histograms as a primitive scene representation.

Sacha Jungerman, Atul Ingle, Yin Li, Mohit Gupta

Proc. ECCV 2022

Robust Scene Inference under Noise-Blur Dual Corruptions

Scene Understanding under Low Light and Motion

 

 

Bhavya Goyal, JF Lalonde, Yin Li, Mohit Gupta

Proc. ICCP 2022

Geometric Calibration of Single-Pixel Distance Sensors

Pose estimation for ultra-low cost single-pixel distance sensors for robotics applications

Project Page

Carter Sifferman, Dev Mehrotra, Mohit Gupta, Michael Gleicher

IEEE Robotics and Automation Letters (RAL)

Proc. IEEE IROS 2022

Single-Photon Structured Light

Structured light 3D imaging enabled at extreme speeds and challenging scenarios using single-photon cameras and digital micro-mirror devices.

Varun Sundar, Sizhuo Ma, Aswin Sankaranarayanan, Mohit Gupta

Proc. CVPR 2022

Compressive Single-Photon 3D Cameras

An online compression framework for SPAD-based 3D cameras based on the coded projection of photon timestamp data.

Project Page 

Felipe Gutierrez-Barragan, Atul Ingle, Trevor Seets, Mohit Gupta, Andreas Velten

Proc. CVPR 2022

Single-Photon Camera Guided Extreme Dynamic Range Imaging

Hybrid single-photon and conventional imaging for high-resolution and high dynamic range imaging capabilities

Project Page

 

Yuhao Liu, Felipe Gutierrez-Barragan, Atul Ingle, Mohit Gupta, Andreas Velten

IEEE WACV 2022

Resolving Motion with Single-Photon Cameras

Single-photon cameras can measure motion robustly in low-light, high-speed scenarios, which can be used to resolve the blur-noise trade-off and reconstruct high-quality images not only for human viewing, but also for subsequent computer vision tasks.

Sizhuo Ma

PhD Thesis

Outstanding Graduate Student Research Award

2021

Photon-Starved Scene Inference using Single Photon Cameras

Scene understanding under low-light conditions

[Code] [Arxiv]

Bhavya Goyal, Mohit Gupta

Proc. ICCV 2021

Blocks-World Cameras

Novel approach to recover piece-wise planar scenes accurately at low computational cost without creating 3D point clouds

Jongho Lee, Mohit Gupta

Proc. CVPR 2021

oral presentation

Passive Inter-Photon Imaging

Inter-photon timing measurements captured by a passive single-photon sensitive camera enable unprecedented dynamic range

Atul Ingle, Trevor Seets, Mauro Buttafava, Shantanu Gupta, Alberto Tosi, Mohit Gupta*, Andreas Velten*

Proc. CVPR 2021

oral presentation

Invisible Perturbations: Physical Adversarial Examples Exploiting the Rolling Shutter Effect

Physical adversarial perturbations using the rolling shutter effect of cameras

Athena Sayles, Ashish Hooda, Mohit Gupta, Rahul Chatterjee, Earlence Fernandes

Proc. CVPR 2021

iToF2dToF: A Robust and Flexible Representation for Data-driven Time-of-Flight Imaging

Project Page

Felipe Gutierrez, Huajin Chen, Mohit Gupta, Andreas Velten, Jinwei Gu

IEEE Transactions on Computational Imaging

2020

Quanta Burst Photography

A computational imaging technique with single-photon cameras enables ultra-low light photography

Sizhuo Ma, S Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, Mohit Gupta

Proc. SIGGRAPH 2020 (ACM Trans. on Graphics)

SIGGRAPH technical papers highlights

University news coverage

Inertial Safety from Structured Light

A novel scene representation that enables fast detection of obstacles in scenarios involving camera or scene motion using single-shot structured light.

Sizhuo Ma, Mohit Gupta

Proc. ECCV 2020

 

Smart Time-Multiplexing of Quads Solves the Multicamera Interference Problem

Novel approach for mitigating multi-ToF-Camera interference by reducing both AC and DC interference

T. Pribanic, T. Petkovic, David Bojanic, K. Bartol, Mohit Gupta

Proc. International Conf. on 3D Vision (3DV) 2020

2019

Asynchronous Single-Photon 3D Imaging

Optimal acquisition strategies for single-photon 3D cameras: Towards long-range laser-scan quality 3D imaging

Anant Gupta*, Atul Ingle*, Mohit Gupta

Proc. ICCV 2019

Marr Prize (best paper) honorable mention

oral presentation

Stochastic Exposure Coding for Handling Multi-ToF-Camera Interference

Novel approach for mitigating multi-ToF-Camera interference by reducing both AC and DC interference

Jongho Lee, Mohit Gupta

Proc. ICCV 2019

oral presentation

Micro-Baseline Structured Light

Low-cost compact structured light 3D imaging for small-sized devices

Viswanath Saradagam, J Wang, Mohit Gupta, Shree Nayar

Proc. ICCV 2019

Differential Scene Flow from Light-Field Gradients

Novel techniques that recover high-precision dense scene flow from light fields

Sizhuo Ma, Brandon M. Smith, Mohit Gupta

International Journal of Computer Vision

Special Issue on ‘Best Papers of ECCV’ (Invited Paper)

Coding Optimization for Fast Fluorescence Lifetime Imaging

Novel and high-performance coding schemes for fast fluorescence lifetime imaging that achieve up to an order of magnitude shorter acquisition time as compared to existing approaches

Jongho Lee, JV Chacko, B Dai, SA Reza, AK Sagar, KW Eliceiri, A Velten, Mohit Gupta

ACM TOG, presented at SIGGRAPH 2019

Practical Coding Function Design for Time-of-Flight Imaging

ToF coding theory and novel time-of-flight camera designs that achieve extreme depth resolution at long distances

Felipe Gutierrez, SA Reza, A Velten, Mohit Gupta

Proc. CVPR 2019

PF-SPAD vs. Conventional CCD or CMOS Pixel

High Flux Passive Imaging with Single-Photon Sensors

Photon-counting achieves several orders of magnitude higher dynamic range than conventional CCD or CMOS pixels

Atul Ingle, A Velten, Mohit Gupta

Proc. CVPR 2019

oral presentation

Photon-Flooded Single-Photon 3D Cameras

Optimal acquisition strategies for single-photon 3D cameras: Towards long-range laser-scan quality 3D imaging

Anant Gupta, Atul Ingle, A Velten, Mohit Gupta

Proc. CVPR 2019

oral presentation

2018

What Are Optimal Coding Functions for Time-of-Flight Imaging?

ToF coding theory and novel time-of-flight camera designs that achieve extreme depth resolution at long distances

Mohit Gupta, S Nayar, A Velten, Eric Breitbach

ACM TOG, presented at SIGGRAPH 2018

Finalists in the Most Influential Research category AutoSens Awards, 2019

Micro Resolution Time-of-Flight Imaging

A technique to generate modulation frequencies as high as 1 THz using optical super heterodyne interferometry for high resolution time-of-flight depth imaging

Fengqiang Li, Florian Willomitzer, P Rangarajan, Mohit Gupta, Andreas Velten, Oliver Cossairt

Proc. ICCP 2018

A Geometric Perspective on Structured Light Coding

ToF coding theory and novel time-of-flight camera designs that achieve extreme depth resolution at long distances

Mohit Gupta, Nikhil Nakhate

Proc. ECCV 2018

3D Scene Flow from 4D Light Field Gradients

Novel techniques that recover high-precision dense scene flow from light fields

Sizhuo Ma, Brandon Smith, Mohit Gupta

Proc. ECCV 2018

oral presentation

selected for IJCV Special Issue on `Best of ECCV’

Trapping Light for Time of Flight

A novel imaging method for near-complete, surround, 3D reconstruction of geometrically complex objects, in a single scan

Ruilin Xu, Mohit Gupta, Shree Nayar

Proc. CVPR 2018

oral presentation

Tracking Objects Outside the Line of Sight using Speckle Imaging

Techniques for tracking micro-motion (<10 microns) of multiple objects, around the corner, with only low-cost components

Brandon Smith, Matthew O'Toole, Mohit Gupta

Proc. CVPR 2018

spotlight oral

2017

CoLux: 3D Micro-Motion Analysis Using Speckle Imaging

Novel low-cost speckle imaging techniques that can measure micro motions of multiple, independently moving objects

Brandon Smith, P Desai, V Agarwal, Mohit Gupta

Proc. SIGGRAPH 2017

Finalist, WARF Innovation Awards

High-depth-resolution Range Imaging with Multiple-wavelength Superheterodyne Interferometry using 1550-nm Lasers

A technique to generate modulation frequencies as high as 1 THz using optical super heterodyne interferometry for high resolution time-of-flight depth imaging

Fengqiang Li, Joshua Yablon, Andreas Velten, Mohit Gupta, Oliver Cossairt

Applied Optics 2017

2016

Dual Structured Light 3D using a 1D Sensor

a novel structured light approach that uses only a 1D sensor with simple optics and no moving parts to reconstruct scenes with the same acquisition speed as a traditional 2D sensor

Jian Wang, Aswin Sankaranarayanan, Mohit Gupta, Srinivasa Narasimhan

Proc. ECCV 2016

oral presentation

DisCo: Display-Camera Communication Using Rolling Shutter Sensors

We present DisCo, a novel display-camera communication system. DisCo enables displays and cameras to communicate with each other, while also displaying and capturing images for human consumption.

Kensei Jo, Mohit Gupta, Shree Nayar

ACM TOG, presented at SIGGRAPH 2016

2015

SpeDo: 6 DOF Ego-Motion Sensor Using Speckle Defocus Imaging

Sensors that measure their motion with respect to the surrounding environment (ego-motion sensors) can be broadly classified into two categories. The first is inertial sensors, such as accelerometers.

Kensei Jo, Mohit Gupta, Shree Nayar

Proc. ICCV 2015

LiSens— A Scalable Architecture for Video Compressive Sensing

The measurement rate of cameras that take spatially multiplexed measurements by using spatial light modulators (SLM) is often limited by the switching speed of the SLMs.

Jian Wang, Mohit Gupta, Aswin Sankaranarayanan

Proc. ICCP 2015

MC3D: Motion Contrast 3D Scanning

We present a novel structured light technique called Motion Contrast 3D scanning (MC3D) that maximizes bandwidth and light source power to avoid performance trade-offs.

Nathan Matsuda, Oliver Cossairt, Mohit Gupta

Proc. ICCP 2015

Phasor Imaging: A Generalization of Correlation ToF Imaging

Time-of-flight (ToF) based 3D imaging in scenes with multipath interference (interreflections) and scattering (e.g., smoke and fog). Frequency domain analysis of transient light transport. Applications in autonomous transportation, 3D mapping and robotic automation.

Mohit Gupta, Shree Nayar, Matthias Hullin, Jaime Martin

ACM TOG, presented at SIGGRAPH, 2015

2014

Recovering Scene Geometry under Wavy Fluid via Distortion and Defocus Analysis

undistorting scenes that are immersed in transparent refractive media with a dynamic surface

M Zhang, Xing Lin, Mohit Gupta, Jinli Suo, Q Dai

Proc. ECCV 2014

Digital Refocusing with Incoherent Holography

Scene refocusing using 2D coherent holograms

Oliver Cossairt, Nathan Matsuda, Mohit Gupta

Proc. ICCP 2014

best paper (honorable mention)

Efficient Space-Time Sampling with Pixel-wise Coded Exposure for High Speed Imaging

High speed compressive imaging (up to 1000 fps) using a conventional 30 fps camera, an LCoS (liquid crystal on silicon) light modulator and large scale dictionary learning. Applications in consumer and scientific imaging.

Dengyu Liu, Jinwei Gu, Y Hitomi, Mohit Gupta, Tomoo Mitsunaga, Shree Nayar

IEEE Trans. PAMI

2013

Fibonacci Exposure Bracketing for High Dynamic Range Imaging

High dynamic range imaging for moving scenes and cameras by capturing images with Fibonacci sequence exposures. Applications in consumer imaging.

Mohit Gupta, Daisuke Iso, Shree Nayar

Proc. ICCV 2013

Structured Light in Sunlight

Outdoor 3D scanning in strong ambient illumination with a limited power-budget. Applications in autonomous transportation, outdoor robotics and urban mapping

Mohit Gupta, Qi Yin, Shree Nayar

Proc. ICCV 2013

2012

A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus

The goal of this work is to build an end-to-end system for structured light 3D scanning under a broad range of global illumination effects, such as inter-reflections, diffusion and sub-surface scattering.

Mohit Gupta, Amit Agrawal, Ashok Veeraraghavan, Srinivasa Narasimhan

International Journal of Computer Vision

When Does Computational Imaging Improve Performance?

Performance analysis of computational imaging techniques such as defocus and motion deblurring and light field capture. Practical guidelines for imaging system design.

Oliver Cossairt, Mohit Gupta, Shree Nayar

IEEE Trans. on Image Processing

A Combined Theory of Defocused Illumination and Global Light Transport

The goal is to recover scene properties in the presence of global illumination.

Mohit Gupta, Yuandong Tian, Srinivasa Narasimhan, Li Zhang

International Journal of Computer Vision

Micro Phase Shifting

3D scanning of optically challenging scenes, such as with translucent materials (e.g., marble, human skin) and concave objects. Applications in digital cultural heritage and performance capture, factory automation and indoor mapping

Mohit Gupta, Shree Nayar

Proc. CVPR 2012

oral presentation

Diffuse Structured Light

High resolution 3D scanning for specular materials (e.g., metallic machine parts), intricate geometries (e.g., finger-prints) and objects with sharp depth discontinuities.

Shree Nayar, Mohit Gupta

Proc. ICCP 2012

2011

Multiplexed Illumination for Scene Recovery in the Presence of Global Illumination

Direct-global separation and scene recovery using multiple light sources

J Gu, T Kobayashi, Mohit Gupta, Shree Nayar

Proc. ICCV 2011

oral presentation

Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary

High speed compressive imaging (up to 1000 fps) using a conventional 30 fps camera, an LCoS (liquid crystal on silicon) light modulator and large scale dictionary learning. Applications in consumer and scientific imaging.

Y Hitomi, J Gu, Mohit Gupta, Tomoo Mitsunaga, Shree Nayar

Proc. ICCV 2011

Structured Light 3D Scanning in the Presence of Global Illumination

The goal of this work is to build an end-to-end system for structured light 3D scanning under a broad range of global illumination effects, such as inter-reflections, diffusion and sub-surface scattering.

Mohit Gupta, Amit Agrawal, Ashok Veeraraghavan, Srinivasa Narasimhan

Proc. CVPR 2011

2010

Flexible Voxels for Motion-Aware Videography

Post capture control of spatial resolution and frame-rate of videos, depending on the scene content. Applications in consumer imaging, scientific imaging and microscopy.

Mohit Gupta, Amit Agrawal, Ashok Veeraraghavan, Srinivasa Narasimhan

Proc. ECCV 2010

Optimal Coded Sampling for Temporal Super-Resolution

We propose coded sampling that address these issues: using N cameras it allows N times temporal super-resolution while allowing N/2 times more light compared to an equivalent high speed camera.

Amit Agrawal, Mohit Gupta, Ashok Veeraraghavan, Srinivasa Narasimhan

Proc. CVPR 2010

2009

(De) Focusing on Global Light Transport for Active Scene Recovery

The goal is to recover scene properties in the presence of global illumination.

Mohit Gupta, Y Tian, S Narasimhan, Li Zhang

Proc. CVPR 2009

oral presentation

2008

On Controlling Light Transport in Poor Visibility Environments

Seeing clearer and farther in scattering media using active illumination and polarization. Applications in underwater imaging, autonomous transportation and medical imaging.

Mohit Gupta, S Narasimhan, Yoav Schechner

Proc. CVPR 2008

High Resolution Tracking of Non-Rigid 3D Motion Using Harmonic Maps

High resolution tracking of facial expressions using harmonic maps

Yang Wang, Mohit Gupta, Song Zhang, Sen Wang, X Gu, D Samaras, Peisen Huang

International Journal of Computer Vision

2007

Legendre Fluids: A Unified Framework for Analytic Reduced Space Modeling and Rendering of Participating Media

Reduced space framework for fast simulation and rendering of scattering media such as smoke and fog. Applications in gaming, special effects and virtual reality.

Mohit Gupta, Srinivasa Narasimhan

ACM Symposium on Computer Animation 2007

Legendre polynomials Triple Product Integral and Chebyshev Polynomials

Reduced space framework for fast simulation and rendering of scattering media such as smoke and fog. Applications in gaming, special effects and virtual reality.

Mohit Gupta, Srinivasa Narasimhan

Technical Report, Robotics Institute, CMU

2006

Acquiring Scattering Properties of Participating Media by Dilution

Imaging and illumination setup for measuring scattering properties of liquids. Applications in computer graphics rendering and ocean monitoring.

Srinivasa Narasimhan, Mohit Gupta, Craig Donner, R Ramamoorthi, S Nayar, H Jensen

Proc. SIGGRAPH 2006

2005

High Resolution Tracking of Non-Rigid 3D Motion of Densely Sampled Data Using Harmonic Maps

High resolution tracking of facial expressions using harmonic maps

Yang Wang, Mohit Gupta, Song Zhang, Sen Wang, X Gu, D Samaras, Peisen Huang

Proc. ICCV 2005

oral presentation

2003