Deep-Learning-Based Medication Action Recognition Using Information Fusion between 3D Skeleton Coordinate and Object-Classification of Hand-ROI

Abstract

As the elderly population increases in an aging society, the number of patients with chronic disease is also increasing. Therefore, an automated sensor-based medication monitoring system is continuously being developed. In this paper, we propose a robot-based scenario that can guide medication by directly interacting with a patient beyond simply monitoring behavior. Behavior recognition was performed in this robot-based scenario. In previous studies, there was a lack of alternatives to variation in human behavior, and there was a weakness in distinguishing similar behaviors. In addition, the behavioral recognition in the mobile robot scenario has a problem that the existing behavioral recognition and the camera perspective are continuously different. In this regard, we presented an algorithm for behavior recognition by collecting and fusion of time-series information by extracting hand-ROI using behavior segmentation, the addition of object information, and joint information. As a result of comparing the method presented in the dataset including variation with the state-of-the-art algorithm, the behavior recognition rate was improved by 20%, resulting in 97%. Behavior segmentation makes the system much more robust to variation compared to existing algorithms. In addition, Hand-ROI supplements behavioral recognition with information about objects that are difficult to capture in the skeleton and frees them from dependency on the background that varies depending on the viewpoint.