DIAYN-PPO Diversity based Reinforcement Learning for Procedural Contents Generation

Abstract

Reinforcement learning (RL) has been used to solve real-world problems in various fields, from gameplay to accurate robotic arm control. Recently, Game AI researchers try to use RL for Procedural content generation (PCG), which generates contents algorithmically, such as textures, sounds, maps, etc. However, most of the existing papers used Proximal policy optimization algorithms (PPO) to learn generative models. Then they attempted to solve the problem without change the algorithm. Because first, the PCG environment is different from the one used in RL. Second, The environment of PCG is different for each other papers. Third, RL in PCG research is in the early stage. Hence, it needs more effort to propose and prove a new method using RL. In this paper, we propose DIAYN-PPO, a diversity driven RL method for PCGRL. To evaluate it, the experimental method and comparative metric for PCG using RL and confirm through experiments whether the RL algorithm for solving a specific problem can be used in PCG. Through these experiments, we show that a single model DIAYN-PPO outperforms the single baseline PPO model in diversity. Furthermore, we show that using RL methods to PCG, it is possible to use various algorithms suitable for the previously developed purpose. It was confirmed that the RL algorithm for a specific purpose, such as RL2, outperform the existing baseline PPO algorithm.