Fixed Non-negative Orthogonal Classifier: Inducing Zero-mean Neural Collapse with Feature Dimension Separation

Abstract

Fixed classifiers in neural networks for classification problems have demonstrated cost efficiency and even outperformed learnable classifiers in some popular benchmarks when incorporating orthogonality (Pernici et al., 2021a). Despite these advantages, prior research has yet to investigate the training dynamics of fixed classifiers on neural collapse. Ensuring this phenomenon is critical for obtaining global optimality in a layer-peeled model, potentially leading to enhanced performance in practice. However, the neural collapse cannot explain the collapse phenomenon in the fixed classifier when its shape is not a simplex ETF. To overcome the limits, we exploit additional constraints to the layer-peeled model: non-negativity and orthogonality. Then, we propose a fixed non-negative orthogonal classifier, which makes a layer-peeled model with the fixed classifier have the global optimality and the max-margin in decision by inducing zero-mean neural collapse. Building on this foundation, we exploit a feature dimension separation inherent in our classifier for further purposes: (1) enhances softmax masking by mitigating feature interference in continual learning and (2) tackles the limitations of mixup on the hypersphere in imbalanced learning. We conducted comprehensive experiments on various datasets and demonstrated significant performance improvements.