From Theory to Remedy: Addressing Representation Collapse for Enhanced Class-Discriminative Clarity

Abstract

Deep learning has achieved remarkable success across diverse domains, particularly in image classification, where models often surpass human-level accuracy. However, conventional deep learning models are typically trained under idealized assumptions—such as perfectly separable class features, full retention of previously learned knowledge, and balanced class distributions—which rarely hold in real-world scenarios. This dissertation investigates three fundamental collapse phenomena that arise when these assumptions are violated: intra-class collapse in angle-based representation learning, inter-class collapse in continual learning, and minority collapse in imbalanced learning. Each phenomenon reflects a distinct form of information degradation in representation learning, and this work aims to analyze their underlying causes and propose theoretically grounded and empirically validated solutions. First, the study of intra-class collapse reveals that when an angle-based approach (e.g., cosine similarity) is applied to feature representations learned through the inner product, the inherent information dispersion of the inner product leads to overlap-induced information loss in the Euclidean norm. Experimental results confirm that this loss significantly impairs representation learning, with error rates reaching up to 24.39% among overlapping samples. To address this, the dissertation proposes a Spherization Layer, which learns representations using only angular information by constraining features onto a hyperspherical surface. The proposed layer operates through three processes—angularization, conversion-to-Cartesian, and no-bias training—enabling complete elimination of norm interference during training. Empirical evaluations demonstrate that the Spherization Layer is compatible with existing models without degrading performance and effectively improves accuracy in angle-dependent tasks such as word analogy tests and few-shot learning. Second, in continual learning, where models incrementally learn new tasks without accessing data from previous ones, inter-class collapse emerges as previously learned old class vectors collapse due to interference from new class vectors, resulting in catastrophic forgetting. While prior works inspired by Neural Collapse (NC) introduced fixed ETF classifiers to mitigate such interference, these approaches remain limited when the classifier’s structure deviates from the simplex ETF. To address this limitation, this work proposes a Fixed Non-negative Orthogonal (FNO) Classifier, alongside a new property termed Zero-mean Neural Collapse (ZNC), in which the origin serves as the global mean. The FNO classifier enforces non-negativity and orthogonality constraints, inducing feature dimension separation that prevents shared dimensions across classes. Furthermore, the synergy between the FNO classifier and softmax masking effectively reduces class-wise interference and alleviates catastrophic forgetting, thereby enhancing continual learning performance. Third, in imbalanced learning, where long-tailed data distributions bias models toward majority classes,minority collapse occurs as minority-class representations degrade in both feature and classifier space. Although oversampling or class-balanced sampling can partially mitigate this issue, their efficacy diminishes under Mixup, a widely used data augmentation technique. This dissertation extends the analysis of Neural Collapse to Mixup under imbalanced settings for the first time and identifies that minority collapse in Mixup arises not only from imbalanced singleton labels but also from mixed labels. To counter this, a Balanced Mixed Label Sampler (BMLS) is proposed, which balances mixed-label frequencies across epochs by deterministically forming Mixup pairs. Furthermore, the Mixed-Singleton Classifier (MS-Clf) interprets mixed labels as singletons, replacing the mixup loss with linearly interpolated class vectors to preserve strong feature learning effects while reducing a potential adverse effect of Mixup on classifier learning. Theoretical and empirical analyses demonstrate that balancing mixed labels and interpreting them as singletons significantly alleviate minority collapse and enhance performance on standard imbalanced benchmarks. In closing, this dissertation provides an executive summary of this work: a unified perspective on three distinct collapse phenomena that hinder representation learning in realistic settings—each corresponding to a specific violation of deep learning’s idealized assumptions. By introducing the Spherization Layer, the Fixed Non-negative Orthogonal Classifier, and the Balanced Mixed Label Sampler with Mixed-Singleton Classifier, this work offers both theoretical insight and practical methodology for mitigating intra-class, inter-class, and minority collapse. Collectively, these findings contribute to the broader understanding of geometric, continual, and imbalanced representation learning, and pave the way toward more robust, generalizable, and interpretable deep learning systems.