Deep Neural Collapse: Theory and Applications

Neural Collapse in Last-Layer

Phenomena

Prevalence of Neural Collapse During the Terminal Phase of Deep Learning Training. Vardan Papyan, X.Y. Han, David L. Donoho. Proceedings of the National Academy of Sciences.

                @article{papyan2020prevalence,
                  title={Prevalence of neural collapse during the terminal phase of deep learning training},
                  author={Papyan, Vardan and Han, XY and Donoho, David L},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={117},
                  number={40},
                  pages={24652--24663},
                  year={2020},
                  publisher={National Acad Sciences}
                }

Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training. Cong Fang, Hangfeng He, Qi Long, Weijie J. Su. Proceedings of the National Academy of Sciences.

                @article{fang2021exploring,
                  title={Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training},
                  author={Fang, Cong and He, Hangfeng and Long, Qi and Su, Weijie J},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={118},
                  number={43},
                  pages={e2103091118},
                  year={2021},
                  publisher={National Acad Sciences}
                }

Neural Collapse in Multi-label Learning with Pick-all-label Loss. Pengyu Li, Xiao Li, Yutong Wang, Qing Qu. 41st International Conference on Machine Learning.

                @inproceedings{
                  li2024neural,
                  title={Neural Collapse in Multi-label Learning with Pick-all-label Loss},
                  author={Pengyu Li and Xiao Li and Yutong Wang and Qing Qu},
                  booktitle={Forty-first International Conference on Machine Learning},
                  year={2024},
                  url={https://openreview.net/forum?id=y8NevOhrnW}
                  }

Generalized Neural Collapse for a Large Number of Classes. Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, Zhihui Zhu. ArXiv preprint.

                  @article{jiang2023generalized,
                    title={Generalized neural collapse for a large number of classes},
                    author={Jiang, Jiachen and Zhou, Jinxin and Wang, Peng and Qu, Qing and Mixon, Dustin and You, Chong and Zhu, Zhihui},
                    journal={arXiv preprint arXiv:2310.05351},
                    year={2023}
                  }

Theoretical Insights

To be filled

Proressive Neural Collapse across Layers

Phenomena

A Law of Data Separation in Deep Learning. Hangfeng He, Weijie J. Su. Proceedings of the National Academy of Sciences.

              @article{he2023law,
                  title={A law of data separation in deep learning},
                  author={He, Hangfeng and Su, Weijie J},
                  journal={Proceedings of the National Academy of Sciences},
                  volume={120},
                  number={36},
                  pages={e2221704120},
                  year={2023},
                  publisher={National Acad Sciences}
              }

Nearest Class-Center Simplification through Intermediate Layers. Ido Ben-Shaul, Shai Dekel. Topological, Algebraic and Geometric Learning Workshops 2022.

              @inproceedings{ben2022nearest,
                title={Nearest class-center simplification through intermediate layers},
                author={Ben-Shaul, Ido and Dekel, Shai},
                booktitle={Topological, Algebraic and Geometric Learning Workshops 2022},
                pages={37--47},
                year={2022},
                organization={PMLR}
              }

On the Implicit Bias Towards Minimal Depth of Deep Neural Networks. Tomer Galanti, Liane Galanti, Ido Ben-Shaul. ArXiv preprint.

              @article{galanti2022implicit,
                title={On the implicit bias towards minimal depth of deep neural networks},
                author={Galanti, Tomer and Galanti, Liane and Ben-Shaul, Ido},
                journal={arXiv preprint arXiv:2202.09028},
                year={2022}
              }

Feature Learning in Deep Classifiers through Intermediate Neural Collapse. Akshay Rangamani, Marius Lindegaard, Tomer Galanti, Tomaso A Poggio. 40th International Conference on Machine Learning.

              @InProceedings{rangamani2023feature,
                title = 	 {Feature learning in deep classifiers through Intermediate Neural Collapse},
                author =       {Rangamani, Akshay and Lindegaard, Marius and Galanti, Tomer and Poggio, Tomaso A},
                booktitle = {International Conference on Machine Learning},
                pages = 	 {28729--28745},
                year = 	 {2023},
                publisher =    {PMLR}
              }

Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks. Liam Parker, Emre Onal, Anton Stengel, Jake Intrater. ICML Workshop in High-Dimensional Learning Dynamics 2023.

              @article{parker2023neural,
                title={Neural collapse in the intermediate hidden layers of classification neural networks},
                author={Parker, Liam and Onal, Emre and Stengel, Anton and Intrater, Jake},
                journal={arXiv preprint arXiv:2308.02760},
                year={2023}
              }

Theoretical Insights

Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination. Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, Qing Qu. ArXiv preprint.

                @article{wang2023understanding,
                  title={Understanding deep representation learning via layerwise feature compression and discrimination},
                  author={Wang, Peng and Li, Xiao and Yaras, Can and Zhu, Zhihui and Balzano, Laura and Hu, Wei and Qu, Qing},
                  journal={arXiv preprint arXiv:2311.02960},
                  year={2023}
                }

Deep Neural Collapse: Theory and Applications

Neural Collapse in the Balanced Case (Left); Multi-label case (Middle); and Large classes (K>d) case (Right).

Neural Collapse in Last-Layer

Phenomena

Theoretical Insights

Proressive Neural Collapse across Layers

Phenomena

Theoretical Insights