Development of Human Joints Information Detecting Algorithm by using CNN in 3D Environment

Abstract

In this research, experiment to obtain positional information of human joints in 3 dimensional space is conducted. Especially, this research focuses on obtaining joints information by minimizing invisible area. To minimize invisible area, multiple RGB-D sensors are used. RGB-D sensor contains multiple camera sensors to measure both 2D and 3D information. Since it’s camera based sensor, the sight is limited and there exists invisible area that camera can’t see. To solve this problem, multiple RGB-D sensors are properly located in the space to construct appropriate environment where there is no invisible area. During reconstructing process, data obtained from multiple KINECTs in 3D space are merged to one point cloud data. After reconstruction process, positional information of human joints in 3D space is extracted. This research suggests the recognition algorithm of the human joint information in Pointcloud data. After the human point cloud data in 3D space is projected to multiple 2D plane, 2D positional information of human joints is obtained by Convolutional Neural Network. The extracted 2D coordinates of human joints from multiple 2D planes are reconstructed in 3D space and finally 3D positional coordinates (x,y,z) are extracted. The performance of suggested algorithm is verified by comparing with motion capture system. Since suggested method minimizes invisible area by using multiple RGB-D sensors, its performance is 50% higher than obtaining joints information by using a single sensor and developer tool. Compared with joints information obtained by using merged joints information and developer tool, suggested method shows 30% higher performance.