Generalized Event Cameras

Frame-based cameras capture images at a fixed cadence or frame rate. Low frame rates (e.g., 30 FPS) can be too slow to capture scene dynamics (such as the slingshot here), while high frame rates can be too data intensive (requires reading off 15 Gbps for a 1-megapixel array).

Event cameras---inspired by biological visual systems---were designed to address these fundamental challenges, with the goal of capturing fast dynamics while maintaining a low sensor readout.
 
Event cameras achieve this by selectively transmitting data, rather than dense transmission.

Unfortunately, the sparse spatiotemporal data captured by an event camera does not retain sufficient information to reconstruct high-quality scene intensities.
 
For example, here are the outputs of a recent intensity-from-events technique. Details of the scene are hard to discern in dynamic regions, while being completely lost in static regions.

The lossy nature of event camera outputs also limits their downstream performance.
 
While it is possible to run certain tasks on events, such as corner detection, for other tasks, such as object detection, events do not provide sufficient contextual information. As an example, in the scene shown here, we fail to detect the tennis racket and the ball.

Our goal is to design a new class of event cameras that can be used in a plug-and-play manner with a gamut of computer vision algorithms, even in very challenging scenarios involving high-speed motion and low light.

For example, in addition to detecting corners; we would like to detect the person, racket, and ball in this scene (object detection); estimate human poses reliably; and segment all portions of the scene---including static regions.

To achieve this, we conceptualize a novel space of generalized event cameras in terms of when events are generated and what flux information is captured.

Our generalized event cameras also produce a sparse spatiotemporal stream---but the key difference is that this stream inherently preserves scene intensity.

We make use of rich flux descriptors that go beyond what existing event cameras capture; these descriptors can be motion-adaptive, span spatial extents, or even utilize unconventional modalities such as coded exposures.

We also utilize change detectors that span larger spatial and temporal supports (i.e., make use of historical information), or those that exploit statistical properties of the integrator to better tell apart scene dynamics from noise.

The crucial detail in generalized events is how we implement them, since their enabling techniques may not be easy to implement in analog.
 
For this reason, we turn to an emerging sensing technology, single-photon sensors, that can provide digital access to photon detections as a sequence of extremely high-speed binary frames.

We design lightweight and efficient algorithms that transform photon detections into generalized event streams.

We image a stress ball that is hurled at a coffee cup, aiming to capture its non-rigid deformations.
 
A high-speed camera can capture the deformation dynamics at 500 FPS. However, each captured frame is SNR-limited in indoor conditions, and features prominent noise and compression artifacts arising from the camera’s ISP. Meanwhile, reconstructions from an event camera lack scene information and do not capture the coffee mug placed on the table.

In comparison, our generalized events provide high-quality reconstructions at 3000 FPS that successfully capture the extensive deformations of the stress ball. The incurred readout is also quite low at 18 bits per second per pixel (or 18 bps).

Next, we consider an urban nighttime traffic scene.

For frame-based cameras, low-light scenes force a choice between short exposures that preserve motion information but are too noisy, and long exposures that mitigate noise, but suffer from motion blur.

Nighttime sequences are also challenging for existing event cameras, leading to significant blur in detecting events. As a result, the outputs of event reconstruction methods are also of low quality.


Our generalized events benefit from their single-photon implementation and offer good low-light performance. Here, we show reconstructions at 3000 FPS and with a readout of 12 bps (or 315 Mbps per megapixel).