Image Classification with Binary Networks on Datasets of Long-tailed Distribution

Abstract

The demand for better performing deep learning architectures is exploding on theend-user side. However, few algorithms are applicable to end users. First problem is enduser’s data distribution difference to that of frequently used curated dataset; it is highlylikely that the end user dataset is not curated and hence the frequency within each classis different. This kinds of datasets are called as long-tailed (LT) since the class-wise datafrequency sorted in descending order is long-tail looking. Second problem is resourceconstraint. End users need to deploy light models since their resources are very limited(e.g. ARM(Advanced Risc Machine), FPGA(Field-Programmable Gate Array), etc.).Hence many try to implement algorithms with small networks. There are many waysto shrink the network; in particular, we focus on Binary Network due to its efficiencywhich meets end user’s harsh condition. Binary network, since the advent of XNOR-Net, became widely used thanks to its small FLOPs and high performance. We firstshow current LT methods on full precision networks applied to binary networks does notperform well. We propose binary network on LT dataset which outperforms the current– i – SOTA methods on full precision networks applied to binary networks. Specifically, wepropose a knowledge distillation method on ImageNet pretrained network as teachernetwork. This teacher network’s classifier is finetuned on downstream task. For learningstudent network, we used KL divergence and Balanced KL divergence to learn teacher’sdistribution. We empirically found that all dataset needs different amount of balanced-ness. Therefore we learn a multiplicative factor on both KL and Balanced KL divergencewith attention mechanism.