Evaluation of StarCraft artificial intelligence competition bots by experienced human players

Abstract

StarCraft is one of the most successful real-time strategy (RTS) games and is also actively being researched by artificial intelligence (AI) communities. Since 2010, game AI researchers have hosted annual AI competition events to develop human-level RTS AIs using StarCraft. It ranks the AI bots by their winning ratio from thousands of AI vs. AI matches without human involvement. It is questionable whether successful AI bots are also competitive and preferable to human players. In this study, we invited 20 experienced players with varying expertise to evaluate skill levels, overall performance and human likeness of AI bots. Results show that human's ranking of AI bots are not identical to the current one from AI competitions. It suggests the need for developing new AI competitions that consider human factors (