Methods of Depth Information Estimation for Various Lighting and Multi-spectral Stereo Vision System

Abstract

Depth information performs an important role in the production of three-dimensional (3D) video content. One way to estimate this information is a stereo matching method. The stereo matching method searches for correspondences from a stereo image that has two different viewpoints. Subsequently, it estimates the depth information by calculating a disparity value between two corresponding points. Generally, a relatively accurate result is obtained from the correspondence search when the stereo image is captured under uniform illumination and exposure conditions. However, it is difficult to estimate accurate correspondences if each viewpoint image is captured under varying illumination and exposure conditions. Several methods have been proposed to overcome this problem. However, those methods have high computational complexity and have limited performances when differences of exposure and illumination levels between the two images are large. Therefore, this problem is one of the challenges to be solved in the stereo matching field. Traditional stereo matching methods generally have been targeted at the depth estimation from two RGB cameras. Recently, a stereo camera system has been created by using cameras that take multi spectral images. Such a system can utilize the advantages of each spectral image. However, this system also has a disadvantage in that it is difficult to accurately match the multi spectral images using the conventional matching methods. With the development of deep learning technologies, networks with performance beyond the existing computer vision have been proposed. Recently, deep learning networks with unsupervised learning methods have been proposed to reduce the high dependence on data, which is one of the drawbacks of the deep learning technologies. With regard to this, cross spectral stereo matching methods aimed at unsupervised learning have been proposed. However, the existing methods are not performed in a complete unsupervised learning manner because of using pre-trained segmentation information. Research works related to this have been studied until recently, but this problem remains one of the challenges. In this dissertation, we present methods of depth information acquisition for various lighting and multi--spectral stereo vision system to solve the problems mentioned above. Also, we present a post--processing method for disparity map enhancement at the end of the dissertation. In the case of various lighting condition problem, we approach this problem in two ways. The first approach is to modify the matching equation. In this approach, we analyze the illumination invariant stereo matching in pixel--wise and block--wise matching manners that are robust to lighting condition changes. Afterward, we combine those matching methods for an adaptive pixel--wise and block--wise stereo matching method based on the analysis result. The second approach is to solve the problem by changing the structure of the stereo matching process. In this approach, we use a stereo matching method based on a color correction to enhance the performance of stereo matching under various lighting conditions. The purpose of this method is to make stereo matching perform efficiently by correcting the color difference of input images, rather than complex equation--based stereo matching such as eliminating lighting factors from the color formation model. For the depth estimation from the multi--spectral stereo vision system, we propose an unsupervised cross--spectral stereo matching method using confidence--based smoothness terms. The smoothing terms we propose are included in the loss function to improve the performance of the stereo matching network. At the end of the dissertation, a post--processing method using bilateral and trilateral kernels is also described to improve depth information estimated from stereo images. As a result, our methods show better performance and more efficient stereo matching results for the images that have various lighting and multi--spectral conditions compared with the conventional methods.