Event-based Vision: Image Reconstruction, Super-Resolution, Depth Estimation

Abstract

Event cameras are new imaging devices that can report scene movements as an asynchronous stream of data called the events. Unlike traditional cameras, event cameras have very low latency (microseconds for event cameras and milliseconds for traditional cameras) very high dynamic range (140 dB for event cameras and 60 dB for traditional), and low power consumption as they report changes of pixels and not a complete frame. As they report per pixel events and not the whole intensity frame they are immune to motion blur. First, we investigate the potential of creating intensity images and videos from an adjustable portion of the event data stream via event-based conditional generative adversarial networks (cGANs). Using the proposed framework, we further show the versatility of our method in directly handling similar supervised tasks, such as optical flow and depth prediction. We then demonstrate the unique capability of our approach in generating HDR images under extreme illumination conditions, creating non-blurred images of objects in a rapid motion, and generating very high frame rate videos up to the temporal resolution of event cameras. Second, we propose an end-to-end recurrent neural network to reconstruct high resolution, high dynamic range, and temporally consistent gray-scale or color frames directly from event streams, and extend it to generate temporally consistent videos. We additionally investigate how to incorporate active pixel sensor frames (produced by an event camera) and events together in a complementary setting and reconstruct images iteratively to create an even higher quality and resolution in the images. Third, we use events and intensity images together to estimate dense disparity. The proposed end-to-end design combines events and images in a sequential manner and correlates them to estimate dense depth values. We also evaluate our method in extreme cases of missing the left or right event or stereo pair and also investigate stereo depth estimation with inconsistent dynamic ranges on the left and right pairs. We use both real-world and simulated sequences and verify that they create fine predictions while outperforming previous methods in quantitative quality measures. Finally, we summarize the presented methods and discuss possible future work.