Action matching network: open-set action recognition using spatio-temporal representation matching

Abstract

In this paper, we address an open-set action recognition problem. While the closed-set action recognition classifies test samples into the same classes of actions used for model training, the problem of the open-set action recognition is more challenging because there is a possibility that the trained model has to recognize actions which do not appear in the training set. To address this issue, we propose an action matching network (AMN) that can identify and classify both actions in the training dataset and the actions not included in the set. AMN extracts spatio-temporal representations from the given video clips and constructs an action dictionary using the given samples. Then, AMN classifies an action by computing the similarity based on Euclidean distance or generates a new action class in the constructed dictionary if it is necessary. Experimental results on UCF101 dataset and a large human motion dataset (a.k.a., HMDB dataset) demonstrate the benefits of AMN over the state-of-the-art approaches to open-set action recognition problems. © 2019, Springer-Verlag GmbH Germany, part of Springer Nature.