Deep Neural Collapse: Theory and Applications

Prevalence of Neural Collapse During the Terminal Phase of Deep Learning Training. Vardan Papyan, X.Y. Han, David L. Donoho. Proceedings of the National Academy of Sciences.

                @article{papyan2020prevalence,
                  title={Prevalence of neural collapse during the terminal phase of deep learning training},
                  author={Papyan, Vardan and Han, XY and Donoho, David L},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={117},
                  number={40},
                  pages={24652--24663},
                  year={2020},
                  publisher={National Acad Sciences}
                }

Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training. Cong Fang, Hangfeng He, Qi Long, Weijie J. Su. Proceedings of the National Academy of Sciences.

                @article{fang2021exploring,
                  title={Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training},
                  author={Fang, Cong and He, Hangfeng and Long, Qi and Su, Weijie J},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={118},
                  number={43},
                  pages={e2103091118},
                  year={2021},
                  publisher={National Acad Sciences}
                }

Neural Collapse in Multi-label Learning with Pick-all-label Loss. Pengyu Li, Xiao Li, Yutong Wang, Qing Qu. 41st International Conference on Machine Learning.

                @inproceedings{
                  li2024neural,
                  title={Neural Collapse in Multi-label Learning with Pick-all-label Loss},
                  author={Pengyu Li and Xiao Li and Yutong Wang and Qing Qu},
                  booktitle={Forty-first International Conference on Machine Learning},
                  year={2024},
                  url={https://openreview.net/forum?id=y8NevOhrnW}
                  }

Generalized Neural Collapse for a Large Number of Classes. Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, Zhihui Zhu. ArXiv preprint.

                  @article{jiang2023generalized,
                    title={Generalized neural collapse for a large number of classes},
                    author={Jiang, Jiachen and Zhou, Jinxin and Wang, Peng and Qu, Qing and Mixon, Dustin and You, Chong and Zhu, Zhihui},
                    journal={arXiv preprint arXiv:2310.05351},
                    year={2023}
                  }

Neural Collapse with Unconstrained Features. Dustin G. Mixon, Hans Parshall, Jianzong Pi. Sampling Theory, Signal Processing, and Data Analysis.

                @article{mixon2020neural,
                  title={Neural collapse with unconstrained features},
                  author={Mixon, Dustin G and Parshall, Hans and Pi, Jianzong},
                  journal={Sampling Theory, Signal Processing, and Data Analysis},
                  year={2022}
                }

Extended Unconstrained Features Model for Exploring Deep Neural Collapse. Tom Tirer, Joan Bruna. International Conference on Machine Learning (ICML).

                @inproceedings{tirer2022extended,
                  title={Extended unconstrained features model for exploring deep neural collapse},
                  author={Tirer, Tom and Bruna, Joan},
                  booktitle={International Conference on Machine Learning},
                  pages={21478--21505},
                  year={2022},
                  organization={PMLR}
                }

Memorization-Dilation: Modeling Neural Collapse Under Label Noise. Duc Anh Nguyen, Ron Levie, Julian Lienen, Gitta Kutyniok, Eyke Hüllermeier. arXiv preprint.

                @article{nguyen2022memorization,
                  title={Memorization-Dilation: Modeling Neural Collapse Under Label Noise},
                  author={Nguyen, Duc Anh and Levie, Ron and Lienen, Julian and Kutyniok, Gitta and H{\"u}llermeier, Eyke},
                  journal={arXiv preprint arXiv:2206.05530},
                  year={2022}
                }

A Geometric Analysis of Neural Collapse with Unconstrained Features. Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, Qing Qu. Advances in Neural Information Processing Systems.

                @article{zhu2021geometric,
                  title={A geometric analysis of neural collapse with unconstrained features},
                  author={Zhu, Zhihui and Ding, Tianyu and Zhou, Jinxin and Li, Xiao and You, Chong and Sulam, Jeremias and Qu, Qing},
                  journal={Advances in Neural Information Processing Systems},
                  volume={34},
                  pages={29820--29834},
                  year={2021}
                }

An Unconstrained Layer-Peeled Perspective on Neural Collapse. Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, Weijie J. Su. International Conference on Learning Representations.

                @inproceedings{ji2022unconstrained,
                  title={An Unconstrained Layer-Peeled Perspective on Neural Collapse},
                  author={Ji, Wenlong and Lu, Yiping and Zhang, Yiliang and Deng, Zhun and Su, Weijie J},
                  booktitle={International Conference on Learning Representations},
                  year={2022}
                }

Neural Collapse under Cross-Entropy Loss. Jianfeng Lu, Stefan Steinerberger. Applied and Computational Harmonic Analysis.

                @article{lu2020neural,
                  title={Neural collapse under cross-entropy loss},
                  author={Lu, Jianfeng and Steinerberger, Stefan},
                  journal={Applied and Computational Harmonic Analysis},
                  volume={59},
                  pages={224--241},
                  year={2022}
                }

On the Emergence of Simplex Symmetry in the Final and Penultimate Layers of Neural Network Classifiers. Weinan E, Stephan Wojtowytsch. arXiv preprint.

                @article{e2020emergence,
                  title={On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers},
                  author={E, Weinan and Wojtowytsch, Stephan},
                  journal={arXiv preprint arXiv:2012.05420},
                  year={2020}
                }

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold. Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, Qing Qu. Advances in Neural Information Processing Systems.

                @inproceedings{yaras2022neural,
                  title={Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold},
                  author={Yaras, Can and Wang, Peng and Zhu, Zhihui and Balzano, Laura and Qu, Qing},
                  booktitle={Advances in Neural Information Processing Systems},
                  year={2022}
                }

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path. X.Y. Han, Vardan Papyan, David L. Donoho. International Conference on Learning Representations.

                @inproceedings{han2022neural,
                  title={Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path},
                  author={Han, XY and Papyan, Vardan and Donoho, David L},
                  booktitle={International Conference on Learning Representations},
                  year={2022}
                }

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features. Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, Zhihui Zhu. International Conference on Machine Learning.

                @inproceedings{zhou2022optimization,
                  title={On the optimization landscape of neural collapse under mse loss: global optimality with unconstrained features},
                  author={Zhou, Jinxin and Li, Xiao and Ding, Tianyu and You, Chong and Qu, Qing and Zhu, Zhihui},
                  booktitle={International Conference on Machine Learning},
                  pages={27179--27202},
                  year={2022},
                  organization={PMLR}
                }

Neural Collapse in Deep Homogeneous Classifiers and the Role of Weight Decay. Akshay Rangamani, Andrzej Banburski-Fahey. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

                @inproceedings{rangamani2022neural,
                  title={Neural collapse in deep homogeneous classifiers and the role of weight decay},
                  author={Rangamani, Akshay and Banburski-Fahey, Andrzej},
                  booktitle={ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
                  pages={4243--4247},
                  year={2022},
                  organization={IEEE}
                }

Are All Losses Created Equal: A Neural Collapse Perspective. Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, Zhihui Zhu. Advances in Neural Information Processing Systems.

                @article{zhou2022all,
                  title={Are All Losses Created Equal: A Neural Collapse Perspective},
                  author={Zhou, Jinxin and You, Chong and Li, Xiao and Liu, Kangning and Liu, Sheng and Qu, Qing and Zhu, Zhihui},
                  journal={Advances in Neural Information Processing Systems},
                  year={2022}
                }

A Law of Data Separation in Deep Learning. Hangfeng He, Weijie J. Su. Proceedings of the National Academy of Sciences.

              @article{he2023law,
                  title={A law of data separation in deep learning},
                  author={He, Hangfeng and Su, Weijie J},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={120},
                  number={36},
                  pages={e2221704120},
                  year={2023},
                  publisher={National Acad Sciences}
              }

Nearest Class-Center Simplification through Intermediate Layers. Ido Ben-Shaul, Shai Dekel. Topological, Algebraic and Geometric Learning Workshops 2022.

              @inproceedings{ben2022nearest,
                title={Nearest class-center simplification through intermediate layers},
                author={Ben-Shaul, Ido and Dekel, Shai},
                booktitle={Topological, Algebraic and Geometric Learning Workshops 2022},
                pages={37--47},
                year={2022},
                organization={PMLR}
              }

On the Implicit Bias Towards Minimal Depth of Deep Neural Networks. Tomer Galanti, Liane Galanti, Ido Ben-Shaul. ArXiv preprint.

              @article{galanti2022implicit,
                title={On the implicit bias towards minimal depth of deep neural networks},
                author={Galanti, Tomer and Galanti, Liane and Ben-Shaul, Ido},
                journal={arXiv preprint arXiv:2202.09028},
                year={2022}
              }

Feature Learning in Deep Classifiers through Intermediate Neural Collapse. Akshay Rangamani, Marius Lindegaard, Tomer Galanti, Tomaso A Poggio. 40th International Conference on Machine Learning.

              @InProceedings{rangamani2023feature,
                title = 	 {Feature learning in deep classifiers through Intermediate Neural Collapse},
                author =       {Rangamani, Akshay and Lindegaard, Marius and Galanti, Tomer and Poggio, Tomaso A},
                booktitle = {International Conference on Machine Learning},
                pages = 	 {28729--28745},
                year = 	 {2023},
                publisher =    {PMLR}
              }

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks. Liam Parker, Emre Onal, Anton Stengel, Jake Intrater. ICML Workshop in High-Dimensional Learning Dynamics 2023.

              @article{parker2023neural,
                title={Neural collapse in the intermediate hidden layers of classification neural networks},
                author={Parker, Liam and Onal, Emre and Stengel, Anton and Intrater, Jake},
                journal={arXiv preprint arXiv:2308.02760},
                year={2023}
              }

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination. Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, Qing Qu. ArXiv preprint.

                @article{wang2025understanding,
                            title={Understanding deep representation learning via layerwise feature compression and discrimination},
                            author={Wang, Peng and Li, Xiao and Yaras, Can and Zhu, Zhihui and Balzano, Laura and Hu, Wei and Qu, Qing},
                            journal={Journal of Machine Learning Research},
                            volume={26},
                            number={220},
                            pages={1--61},
                            year={2025}
                          }

Deep Neural Collapse: Theory and Applications

Neural Collapse in the Balanced Case (Left); Multi-label case (Middle); and Large classes (K>d) case (Right).

Neural Collapse in Last-Layer

Phenomena

Theoretical Insights

Proressive Neural Collapse across Layers

Phenomena

Theoretical Insights