Lei Huang (黄 雷)

Associate Professor

State Key Laboratory of Software Development Environment

Institute of Artificial Intelligence, Beihang University, Beijing, China


Office: Room B1009, New Main Building, Beihang University, Haidian, Beijing

Email: huangleiai@buaa.edu.cn

Email (Optional): huanglei36060520@gmail.com


[Publication] | [Professional Activities] | [Students] |[Teaching] |[Education] | [Awards and Honors] | [Source Code]| [Miscellaneous] | [CV]

Lei Huang received his BSc and PhD degrees under supervision of Prof. Wei Li, respectively in 2010 and 2018, at the School of Computer Science and Engineering, Beihang University, China. From 2015 to 2016, he visited the Vision and Learning Lab, University of Michigan, Ann Arbor, as a joint PhD student supervised by Prof. Jia Deng. During 2018 to 2020, he was a research scientist in Inception Institute of Artificial Intelligence (IIAI), UAE. His current research mainly focuses on normalization techniques (involving methods, theories and applications) in training DNNs. He also has wide interests in deep learning theory (representation & optimization) and computers vision tasks. He serves as a reviewer for the top conferences and journals such as CVPR, ICML, ICCV, ECCV, NeurIPS, AAAI, JMLR, TPAMI, IJCV, TNNLS, etc.
  • NEWS July 2024: One paper is accepted by ECCV.
  • NEWS June 2024: One paper is accepted by IEEE TPAMI.
  • NEWS May 2024: The TinyLLaVA Factory project is open-sourced and the paper is available.
  • NEWS May 2024: One paper is accepted by IJCV.
  • NEWS May 2024: One paper is accepted by ICML.
  • NEWS Feb 2024: One paper is accepted by CVPR.
  • NEWS Feb 2024: One paper is accepted by IEEE TPAMI.
  • NEWS Feb 2024: The TinyLLaVA project is open-sourced and the technical report is available.
  • NEWS Jan 2024: One paper is accepted by ICLR.
  • NEWS Aug 2023: One paper is accepted by IJCV.
  • NEWS Feb 2023: One paper is accepted by IEEE TPAMI.
  • NEWS OCt 2022: My book Normalization Techniques in Deep Learning has been published in Springer Nature.
  • NEWS Sep 2022: Two papers are accepted by NeurIPS 2022 (Spotlights).
  • NEWS March 2022: Three papers are accepted by CVPR 2022.
  • NEWS Sep 2021: One paper is accepted by NeurIPS 2021.
  • NEWS July 2021: Two papers are accepted by ICCV 2021.
  • NEWS March 2021: Two papers are accepted by CVPR 2021.
  • NEWS Feb 2021: I will organize a tutorial named "Normalization Techniques in Deep Learning: Methods, Analyses, and Applications" at CVPR 2021.
  • NEWS Dec 2020: Two papers are accepted by AAAI 2021.
  • NEWS Sep 2020: Our survey on normalization: "Normalization Techniques in Training DNNs: Methodology, Analysis and Application" can be available on arXiv and GitHub .
  • NEWS July 2020: One paper is accepted by ACM Multimedia 2020
  • NEWS July 2020: Two papers (One Oral) are accepted by ECCV 2020
  • NEWS June 2020: One paper is accepted by ICML 2020
  • NEWS June 2020: Our paper "Controllable Orthogonalization in Training DNNs" is norminated as Best Paper Candidate by CVPR 2020
  • NEWS Feb 2020: Two papers (Orals) are accepted by CVPR 2020
  • NEWS Jan 2020: One paper is accepted by ECAI 2020
  • NEWS May 2019: I will co-organize The 1st Statistical Deep Learning in Computer Vision workshop in conjunction with ICCV 2019.
  • NEWS March 2019: Two papers are accpeted by CVPR 2019.
  • NEWS September 2018: I co-organized a tutorial named "Normalization Methods for Training Deep Neural Networks: Theory and Practice" at ECCV 2018. The slides can be found here .
  • NEWS Feb 2018: Our paper is accepted by CVPR 2018
  • NEWS November 2017: One paper is accepted by AAAI 2018 (oral)
  • NEWS  Sep 2017: I will atend the Doctoral Symposium at ICIP 2017, on 19 September at China National Convention Center (CNCC), BeiJing. I will present my research about 'Normalization Techniques in Training Deep Neural Networks'
  • NEWS  Jul 2017: One paper accepted by ICCV 2017.
  • Publications

    I am addicted to understanding and debugging the training of DNNs. I believe one avenue is delving into the basic modules of DNNs, e.g., normalization layer and linear layer. My contributions for the community are mainly on designing and understanding these basic modules and their compound for the training dynamics of DNNs (* indicates corresponding authors; # indicates equal contributions).

    Book




    Selected papers | All paper list


    • Normalization layer (algorithms and analyses):

    On the Nonlinearity of Layer Normalization.
    Yunhao Ni, Yuxin Guo, Junlong Jia, Lei Huang*
    ICML 2024
    Normalization Techniques in Training DNNs: Methodology, Analysis and Application.
    Lei Huang*, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    IEEE TPAMI (ESI Highly Cited Paper), 2023 (arXiv:2009.12836)
    Understanding the Failure of Batch Normalization for Transformers in NLP
    Jiaxi Wang, Ji Wu, Lei Huang
    NeurIPS 2022 (Spotlight)
    Delving into the Estimation Shift of Batch Normalization in a Network.
    Lei Huang*, Yi Zhou, Tian Wang, Jie Luo, Xianglong Liu.
    CVPR 2022
    Group Whitening: Balancing Learning Efficiency and Representational Capacity
    Lei Huang*, Yi Zhou, Li Liu, Fan Zhu, Ling Shao
    CVPR 2021
    An Investigation into the Stochasticity of Batch Whitening
    Lei Huang*, Lei Zhao, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    CVPR 2020 (Oral)
    Iterative Normalization: Beyond Standardization towards Efficient Whitening
    Lei Huang*, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    CVPR 2019
    Decorrelated Batch Normalization
    Lei Huang*, Dawei Yang, Bo Lang, Jia Deng
    CVPR 2018

    • Linear layer with constraints:

    Controllable Orthogonalization in Training DNNs
    Lei Huang*, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, Ling Shao
    CVPR 2020 (Oral, Best Paper Nomination )
    Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
    Lei Huang*, Xianglong Liu, Bo Lang, Admas Wei Yu, Bo Li
    AAAI 2018 (Oral)
    Projection Based Weight Normalization for Deep Neural Networks
    Lei Huang*, Xianglong Liu, Bo Lang, Bo Li
    preprint (arXiv:1710.02338), Pattern Recognition 2020
    Centered Weight Normalization in Accelerating Training of Deep Neural Networks
    Lei Huang, Xianglong Liu*, Yang Liu, Bo Lang, Dacheng Tao
    IEEE ICCV, 2017

    • Understanding the representation and training of DNNs.:

    Understanding Whitening Loss in Self-supervised Learning.
    Lei Huang, Yunhao Ni, Xi Weng, Rao Muhammad Anwer, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan.
    IEEE TPAMI. 2024
    Modulate Your Spectrum in Self-Supervised Learning
    Xi Weng#, Yunhao Ni#, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, Lei Huang*
    ICLR 2024 (preprint arXiv:2305.16789)
    An Investigation into Whitening Loss for Self-supervised Learning
    Xi Weng#, Lei Huang*#, Lei Zhao#, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan
    NeurIPS 2022 (Spotlight)
    Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
    Lei Huang*, Jie Qin, Li Liu, Fan Zhu, Ling Shao
    ECCV 2020 (Oral)
    On the Number of Linear Regions of Convolutional Neural Networks
    Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, Ling Shao
    ICML 2020
    Block-normalized Gradient Method: an Empirical Study for Training Deep Nerual Network
    Adams Wei Yu, Lei Huang, Qihang Lin, Ruslan Salakhutdinov, Jaime Carbonell.
    preprint (arXiv:1707.04822)

    Professional Activities

    • Conference Reviewer:ICLR2025, AAAI2025, NeurIPS2024, ECCV2024, ICML 2024, CVPR2024, ICLR2024, AAAI2024, NeurIPS2023, ICCV2023, ICML2023, CVPR2023, ICLR2023, AAAI2023, NeurIPS2022, ECCV2022, ICML 2022, CVPR 2022, AAAI 2022, NeurIPS2021, ICCV2021, ICML 2021, CVPR 2021, AAAI 2021, WACV 2021, NeurIPS 2020, ACM Multimedia 2020, ECCV2020, IJCAI2020, CVPR2020, AAAI2020, ICCV 2019, CVPR 2019, AAAI 2019, ACM Multimedia 2019

    • Journal Reviewer: IEEE TPAMI, JMLR, TMLR (Transactions on Machine Learning Research), IJCV, IEEE Transactions on Cybernetics, TNNLS (IEEE Transactions on Neural Networks and Learning Systems), PR.

    • Sub-Reviewer of CVPR 2016, NIPS 2016, IJCAI 2017


    Students

    • Graduate students officially: Dun Su (2021-now),         Xi Weng (2022-now),         Junlong Jia (2022-now),         Yunhao Ni (2023-now),         LuoChe Wang (2023-now)

    • Students supervised unofficially: Diwen Wan (Internship in IIAI of UAE between 2018-2020),        Lei Zhao (Beihang University, 2019 - now),        Jiaxi Wang (Tsinghua University, 2020 - 2022),        Jiawei Zhang (Beihang University, 2021 - now),        Ge Kan (Beihang University, 2021 - 2022),        Junzhu Mao (Nanjing University of Science and Technology, 2021 - 2022)

    Teaching

    • Deep Learning (B3J420160), Spring, Beihang University

    • Computer Vision (B3J424120); Machine Learning (B3I423170), Fall, Beihang University

    • Deep Learning (42113305, for postgraduate); Reinforcement Learning (42112114, for postgraduate), Spring, Beihang University

    • Pattern recognition and machine Learning (42112110, for postgraduate); Fall, Beihang University

    Project

    • 国家科技创新2030-“新一代人工智能”重大项目课题,基础模型的鲁棒性评测机理、方法与工具研究,2022.12-2025.11,主持(课题负责人)

    • 国家自然科学基金面上项目,深度神经网络机制可解释性研究,2025.01-2028.12,主持

    • 国家自然科学基金青年项目,深度神经网络中激活值标准化技术研究,2022.01-2024.12,主持

    • 北京航空航天大学青年拔尖人才计划项目,深度神经网络模型设计与分析,2022.07-2027.06,主持

    • 软件开发环境国家重点实验室探索性自选课题,分布可控的深度神经网络模型分析与设计,2021.01-2022.12,主持

    • 复杂关键软件环境全国重点实验室探索性自选课题,面向无人群智系统的深度神经网络技术研究,2023.01-2024.12,主持

    • 国家科技创新2030-“新一代人工智能”重大项目课题,人机智能持续学习与融合演化模型,2021.12-2024.11,参与

    • 北京航空航天大学理工交叉融合“十大科学问题” 项目,群体智能涌现机理与运行机制,2022.07-2025.06,参与



    Education

    • Sep. 2015 - Oct. 2016
      • Visiting Ph.D student in Vision & Learning Lab, at the University of Michigan, Ann Arbor
      • Research advisor: Profs. Jia Deng
    • Sep. 2011 - 2018
      • Ph.D student in Computer Science, School of Computer Science and Engineering, Beihang University
      • Advisor: Profs. Wei Li and co-advised by Profs. Bo Lang
    • Sep. 2010 - Sep. 2011
      • Master student in Computer Science, School of Computer Science and Engineering, Beihang University
      • Advisor: Profs. Wei Li and co-advised by Profs. Bo Lang
    • Sep. 2006 - Jun. 2010
      • B.Sc in Computer Science, School of Computer Science and Engineering, Beihang University
      • Thesis advisor: Profs. Wei Li

    Talk Slides

    • September 8th, 2018. Normalization Methods for Training Deep Neural Networks: Theory and Practice, ECCV 2018 Tutorial. Munich, Germany. [Slides]

    • August 17th, 2017. Normalization techiniques in deep learning. Multimedia Signal and Intelligent Information Processing Laboratory, Tsinghua University, Beijing. [Slides]

    • November 2016- January 2017. Deep Learning Seminar For graduated students in State Key Laboratory of Software Development Environment, Beihang University, Beijing. [slides1-Introduction] [slides2-MLP] [slides3-CNN] [slides4-RNN]

    • November 1th, 2014. Graph-based active Semi-Supervised Learning: a new perspective for relieving multi-class annotation labor. ICT International Exchange Workshop 2014, Laboratory of Advanced Research B, University of Tsukuba, Japan [Slides]


    Source Code

    • ONI: This project is the implementation of the paper "Controllable Orthogonalization in Training DNNs" (arXiv:2004.00917).

    • StochasticityBW: This project is the implementation of the paper "An Investigation into the Stochasticity of Batch Whitening" (arXiv:2003.12327).

    • IterNorm: This project is the implementation of the paper "Iterative Normalization: Beyond Standardization towards Efficient Whitening" (arXiv:1904.03441).

    • DBN: This project is the Torch implementation of the paper : Decorrelated Batch normalization (arXiv:1804.08450).

    • OWN: This project is the Torch implementation of the paper : orthogonal weight normalization method for solving orthogonality constraints over Steifel manifold in deep neural networks (arXiv:1709.06079).

    • CWN: This project is the Torch implementation of our accepted ICCV 2017 paper: Centered Weight Normalization in Accelerating Training of Deep Neural Networks

    • NormProjection: This project is the Torch implementation of the paper: Projection Based Weight Normalization for Deep Neural Networks (arXiv:1710.02338)

    • Ladder_deepSSL_NP: The reimplementation of Ladder networks with projection based weight normalization. We achieved test errors as 2.52%, 1.06%, and 0.91% on Permunate invariant MNIST dataset with 20, 50, and 100 labeled samples respectively, which is the state-of-the-art results.


    Miscellaneous

    • I also have great interest in hiphop dance (particularly in breakin dance), singing and guitar. you can find some videos of my show on my homepage of Meipai and YouTube