A data-driven appearance modeling for virtual exhibition

Abstract

Recently, virtual exhibition based on digital reconstruction of exhibits have been widely attracting attentions. In this application, a realistic representation of exhibits is important to increase immersion and realism of virtual contents. However, unlike homogeneous material or commercial products, exhibits have their own uniqueness in appearance and shape so that it is challenging to faithfully reconstruct them due to a large cost and computation time. In this thesis, a novel method for realistic appearance modeling and an acquisition system that captures the shape and appearance are proposed. The proposed digital reconstruction method is divided into two parts. First, an image-based measurement system, which has a physical gantry consisting of multi-cameras and multi-light sources, acquires the shape and reflectance data of a target object. Second, an appearance modeling method estimates the proper reflectance model from the shape and measured reflectance data under the assumption that the surface is opaque. The proposed measurement system consists of the rotational gantry including multi-cameras and LED light sources, a control unit, and multi-projectors for the shape acquisition. The shape of the target object is densely estimated by a structured-light based 3D reconstruction algorithm. Reflectance data associated with the reconstructed shape is also acquired by uniformly sampling a hemispherical light field centered on the target object. The measurement system is combined with a dark room in order to control material preservation conditions such as temperature, humidity, and light characteristics. In the second part, two efficient methods for appearance modeling are proposed. The first method aims to efficiently capture the BTF (Bidirectional Texture Function) in which non-local surface effects occur frequently. Based on the observation that the most non-linearity of reflectance data is generated by the non-Lambertian effects such as shadows or specular reflections, this method separates diffuse reflection from non-Lambertian reflections and models them using a linear factorization. Specular reflections as a non-linear term are stored using a sparse matrix representation so that this method effectively improves the modeling accuracy as well as the visual quality of the estimated BTF. The second method proposes a Deep Embedding Clustering (DEC) based auto-encoder that estimates the SVBRDF (Spatially-Varying Bidirectional Reflectance Distribution Function) in order to reduce the data dependency of the BTF by excluding the non-local surface effects. The proposed auto-encoder simultaneously estimates basis functions of the SVBRDF and its blending weights in a linear manifold with two customized layers. The joint-optimization of the auto-encoder for basis functions and weights allows a fast computation of the SVBRDF estimation. The thesis contributes in reducing the measurement cost and the computation time for modeling the appearance of an exhibit by using the multi-camera based shape and by developing reflectance measurement system, and computationally-efficient appearance model for the BTF and the SVBRDF, respectively. The proposed reconstruction method can be used in the representation of digital contents in virtual exhibition by providing photo-realistic results.