Depth Estimation Using Comparative Variation and CNN to Train Image Patches

Abstract

Stereo matching is one of the most famous computer vision problems. Depth perception is important for a variety of advanced technologies such as self-driving vehicles and unmanned aerial transport systems. Through decades of research, recent algorithms have shown the possibility of using stereo matching in real-world applications. However, there are many problems and difficulties in the process of each step. Thus, we contemplate the stereo matching problems that arise at each step and analyse and experiment how to tackle them. First, we introduce an approach to use convolutional neural network for matching cost computation method. The network architectures include a trainable feature extractor that represents each image patch as a vector of features. The trainable feature extractor uses a siamese-type network architecture. Training is performed in a supervised manner by constructing a binary classification data set as an example of a positive or negative patch pair. The Siamese convolutional neural network consists of Siamese and decision network for computational efficiency. We employed a Siamese network to learn such descriptors. We propagate the two patches through the Siamese network to extract the descriptor (feature vector). Then, the decision network manages to determine whether the features are of a positive image pair or negative pair. Second, we present the edge-preserving suppression method for depth estimation via comparative variation. We formulate a functional energy function based on the relative total intensity and space variation, and we minimize the energy function via iteratively reweighted least squares. Assuming that textural edges most likely correspond to depth discontinuities, we exploit the comparative variations of the color image to produce a more accurate depth map. Lastly, we propose the occlusion detection method for stereo matching and hole filling method to generate 3D information. Exploiting the result from the previous step, we segment the depth map occlusion and error regions into non-occlusion regions. To detect occlusion and error regions, we formulate an energy function with three constraints such as ordering, uniqueness, and color similarity constraints. After labeling the occlusion and error regions, we optimize an energy function based MRF via dynamic programing method. Additionally, we presented a novel weighted median filter with skew normal distribution weight (SWMF). SWMF outperforms the conventional methods, and it reduces noise efficiently while preserving the meaningful structure. The proposed algorithms in this dissertation provide reasonable and stable results. Therefore, our approaches are expected to be utilized in various applications for the next generation.