Learning Strategies for Gaussian Process Regression Imitation Learning Using Stochastic Reward-Based Optimization

Abstract

On-line motion planning in dynamically changing environments poses a significant challenge in the design of autonomous robotic systems. Conventional methods often require intricate design choices, while modern deep reinforcement learning (DRL) approaches demand vast amounts of robot motion data. These requirements become even more complex and extensive for systems with higher degrees of freedom (DOFs). Imitation learning addresses these issues by allowing human experts to intuitively provide demonstrations for robotic agents to learn from. Some learning frameworks utilize Gaussian process regression (GPR) models along with stochastic reward-based trajectory optimization algorithms to learn optimal policies with minimal number of demonstration data. These algorithms, however, may result in suboptimal policies such as abrupt changes in trajectories and require substantial amount of training data. In this study, an experiment, detailed in an approach by Ewerton et al. [1], is scrutinized, where the authors’ method yields suboptimal policies. The causes of these suboptimalities are identified and mitigated by introducing problem-specific objective functions such as trajectory smoothness cost to the optimization algorithm. Additionally, a novel continuous-space data sampling method based on Voronoi tessellation is proposed to enhance data efficiency of the learning algorithm. This method leverages the off-line nature of the learning algorithm and the spatial correlation between input features and output of the trajectory prediction model to balance exploration and exploitation. Experimental results demonstrate that the proposed method can improve learned policies with fewer data compared to the previous baseline method.