Architecture Search and Unsupervised Learning for Binary Networks

Abstract

Recently, deployment of AI models to resource constrained edge devices has gained increasing attention. As one of the leading methods in the field for edge device deployment, binary networks in turn have become more and more important. However, the current research on binary networks are lacking in two key aspects. First, the architecture of binary networks is mostly derived from floating point networks instead of designing architectures specifically for binary networks. Second, binary networks are mostly tested in a restricted learning scenario of supervised classification. Considering the recent advancements using floating point networks in neural architecture search and unsupervised representation learning for diverse downstream tasks, we aim to mitigate the aforementioned shortcomings of binary networks.

In detail, we first propose to learn architectures specifically for binary networks via a new search space for binary architectures and a novel search objective. Our results exhibit that our searched architectures show non-trivial improvements over architectures derived from floating point networks.

We then propose an unsupervised representation learning method for binary networks that uses a moving target. Specifically, we jointly train a randomly initialized classifier attached to a pretrained floating point backbone as our moving target with additional feature similarity loss that is dynamically balanced to improve training. Through extensive validations, we show that the representations learned by binary networks in an unsupervised manner transfer to diverse datasets and tasks.