Lei Huang (黄 雷)

Associate Professor

State Key Laboratory of Software Development Environment

Institute of Artificial Intelligence, Beihang University, Beijing, China


Office: Room B1009, New Main Building, Beihang University, Haidian, Beijing

Email: huangleiAI@buaa.edu.cn

Email (Optional): huanglei36060520@gmail.com


[Publication] | [Professional Activities] | [Students] |[Teaching] |[Education] | [Awards and Honors] | [Source Code]| [Miscellaneous] | [CV]

Lei Huang received his BSc and PhD degrees under supervision of Prof. Wei Li, respectively in 2010 and 2018, at the School of Computer Science and Engineering, Beihang University, China. From 2015 to 2016, he visited the Vision and Learning Lab, University of Michigan, Ann Arbor, as a joint PhD student supervised by Prof. Jia Deng. During 2018 to 2020, he was a research scientist in Inception Institute of Artificial Intelligence (IIAI), UAE. His current research mainly focuses on normalization techniques (involving methods, theories and applications) in training DNNs. He also has wide interests in deep learning theory (representation & optimization) and computers vision tasks. He serves as a reviewer for the top conferences and journals such as CVPR, ICML, ICCV, ECCV, NeurIPS, AAAI, JMLR, TPAMI, IJCV, TNNLS, etc.
  • NEWS OCt 2022: My book Normalization Techniques in Deep Learning has been published in Springer Nature.
  • NEWS Sep 2022: Two papers are accepted by NeurIPS 2022.
  • NEWS March 2022: Three papers are accepted by CVPR 2022.
  • NEWS Sep 2021: One paper is accepted by NeurIPS 2021.
  • NEWS July 2021: Two papers are accepted by ICCV 2021.
  • NEWS March 2021: Two papers are accepted by CVPR 2021.
  • NEWS Feb 2021: I will organize a tutorial named "Normalization Techniques in Deep Learning: Methods, Analyses, and Applications" at CVPR 2021.
  • NEWS Dec 2020: Two papers are accepted by AAAI 2021.
  • NEWS Sep 2020: Our survey on normalization: "Normalization Techniques in Training DNNs: Methodology, Analysis and Application" can be available on arXiv and GitHub .
  • NEWS July 2020: One paper is accepted by ACM Multimedia 2020
  • NEWS July 2020: Two papers (One Oral) are accepted by ECCV 2020
  • NEWS June 2020: One paper is accepted by ICML 2020
  • NEWS Feb 2020: Two papers (Orals) are accepted by CVPR 2020
  • NEWS Jan 2020: One paper is accepted by ECAI 2020
  • NEWS May 2019: I will co-organize The 1st Statistical Deep Learning in Computer Vision workshop in conjunction with ICCV 2019.
  • NEWS March 2019: Two papers are accpeted by CVPR 2019.
  • NEWS September 2018: I co-organized a tutorial named "Normalization Methods for Training Deep Neural Networks: Theory and Practice" at ECCV 2018. The slides can be found here .
  • NEWS Feb 2018: Our paper is accepted by CVPR 2018
  • NEWS November 2017: One paper is accepted by AAAI 2018 (oral)
  • NEWS  Sep 2017: I will atend the Doctoral Symposium at ICIP 2017, on 19 September at China National Convention Center (CNCC), BeiJing. I will present my research about 'Normalization Techniques in Training Deep Neural Networks'
  • NEWS  Jul 2017: One paper accepted by ICCV 2017.
  • Publications

    I am addicted to understanding and debugging the training of DNNs. I believe one avenue is delving into the basic modules of DNNs, e.g., normalization layer and linear layer. My contributions for the community are mainly on designing and understanding these basic modules and their compound for the training dynamics of DNNs (* indicates corresponding authors; # indicates equal contributions).

    Book




    Selected papers | All paper list


    • Normalization layer (algorithms and analyses):

    Normalization Techniques in Training DNNs: Methodology, Analysis and Application.
    Lei Huang*, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    arXiv:2009.12836
    Understanding the Failure of Batch Normalization for Transformers in NLP
    Jiaxi Wang, Ji Wu, Lei Huang
    NeurIPS 2022 (Spotlight)
    Delving into the Estimation Shift of Batch Normalization in a Network.
    Lei Huang*, Yi Zhou, Tian Wang, Jie Luo, Xianglong Liu.
    CVPR 2022
    Group Whitening: Balancing Learning Efficiency and Representational Capacity
    Lei Huang*, Yi Zhou, Li Liu, Fan Zhu, Ling Shao
    CVPR 2021
    An Investigation into the Stochasticity of Batch Whitening
    Lei Huang*, Lei Zhao, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    CVPR 2020 (Oral)
    Iterative Normalization: Beyond Standardization towards Efficient Whitening
    Lei Huang*, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
    CVPR 2019
    Decorrelated Batch Normalization
    Lei Huang*, Dawei Yang, Bo Lang, Jia Deng
    CVPR 2018

    • Linear layer with constraints:

    Controllable Orthogonalization in Training DNNs
    Lei Huang*, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, Ling Shao
    CVPR 2020 (Oral, Best Paper Nomination )
    Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks
    Lei Huang*, Xianglong Liu, Bo Lang, Admas Wei Yu, Bo Li
    AAAI 2018 (Oral)
    Projection Based Weight Normalization for Deep Neural Networks
    Lei Huang*, Xianglong Liu, Bo Lang, Bo Li
    preprint (arXiv:1710.02338), Pattern Recognition 2020
    Centered Weight Normalization in Accelerating Training of Deep Neural Networks
    Lei Huang, Xianglong Liu*, Yang Liu, Bo Lang, Dacheng Tao
    IEEE ICCV, 2017

    • Understanding the representation and training of DNNs.:

    An Investigation into Whitening Loss for Self-supervised Learning
    Xi Weng#, Lei Huang*#, Lei Zhao#, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan
    NeurIPS 2022 (Spotlight)
    Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
    Lei Huang*, Jie Qin, Li Liu, Fan Zhu, Ling Shao
    ECCV 2020 (Oral)
    On the Number of Linear Regions of Convolutional Neural Networks
    Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, Ling Shao
    ICML 2020
    Block-normalized Gradient Method: an Empirical Study for Training Deep Nerual Network
    Adams Wei Yu, Lei Huang, Qihang Lin, Ruslan Salakhutdinov, Jaime Carbonell.
    preprint (arXiv:1707.04822)

    Professional Activities

    • Regular Conference Reviewer: ICLR2023, AAAI2023, NeurIPS2022, ECCV2022, ICML 2022, CVPR 2022, AAAI 2022, NeurIPS2021, ICCV2021, ICML 2021, CVPR 2021, AAAI 2021, WACV 2021, NeurIPS 2020, ACM Multimedia 2020, ECCV2020, IJCAI2020, CVPR2020, AAAI2020, ICCV 2019, CVPR 2019, AAAI 2019, ACM Multimedia 2019

    • Journal Reviewer: JMLR, TMLR (Transactions on Machine Learning Research), IEEE TPAMI, IJCV, IEEE Transactions on Cybernetics, TNNLS (IEEE Transactions on Neural Networks and Learning Systems), PR.

    • Sub-Reviewer of CVPR 2016, NIPS 2016, IJCAI 2017


    Students

    • Graduate students officially: Dun su (2021-now),         Xi Weng (2022-now),         Junlong Jia (2022-now)

    • Students supervised unofficially: Diwen Wan (Internship in IIAI of UAE between 2018-2020),        Lei Zhao (Beihang University, 2019 - now),        Jiaxi Wang (Tsinghua University, 2020 - now),        Jiawei Zhang (Beihang University, 2021 - now),        Ge Kan (Beihang University, 2021 - now),        Junzhu Mao (Nanjing University of Science and Technology, 2021 - now)

    Teaching

    • Deep Learning (B3J420160), Spring, Beihang University

    • Computer Vision (B3J424120); Machine Learning (B3I423170), Fall, Beihang University



    Education

    • Sep. 2015 - Oct. 2016
      • Visiting Ph.D student in Vision & Learning Lab, at the University of Michigan, Ann Arbor
      • Research advisor: Profs. Jia Deng
    • Sep. 2011 - 2018
      • Ph.D student in Computer Science, School of Computer Science and Engineering, Beihang University
      • Advisor: Profs. Wei Li and co-advised by Profs. Bo Lang
    • Sep. 2010 - Sep. 2011
      • Master student in Computer Science, School of Computer Science and Engineering, Beihang University
      • Advisor: Profs. Wei Li and co-advised by Profs. Bo Lang
    • Sep. 2006 - Jun. 2010
      • B.Sc in Computer Science, School of Computer Science and Engineering, Beihang University
      • Thesis advisor: Profs. Wei Li

    Awards and Honors

    • China Scholarship Council (CSC) Scholarship, 2015.

    • The National Scholarship for Ph.D. candidate, 2014.

    • The Guanghua Scholarship, 2013.

    • Outstanding graduate, Beihang University. 2010

    • The Second Class Prize of the 2nd Beihang University Dancing Contest, 2010

    • The Second Class Prize of the 1st Beihang University Dancing Contest, 2008.

    • The Third Class Scholarship for Academic Performance in Beihang University, 2009

    • The Third Class Scholarship for Contest Excellence in Beihang University, 2008

    • The Third Class Prize of the 18th Mathematics Contest for College Students in Beijing, 2007.

    • The Third Class Prize of the 24th National Physics Contest for College Students, 2007.

    • The Second Class Scholarship for Excellent Freshman in Beihang University, 2006.


    Talk Slides

    • September 8th, 2018. Normalization Methods for Training Deep Neural Networks: Theory and Practice, ECCV 2018 Tutorial. Munich, Germany. [Slides]

    • August 17th, 2017. Normalization techiniques in deep learning. Multimedia Signal and Intelligent Information Processing Laboratory, Tsinghua University, Beijing. [Slides]

    • November 2016- January 2017. Deep Learning Seminar For graduated students in State Key Laboratory of Software Development Environment, Beihang University, Beijing. [slides1-Introduction] [slides2-MLP] [slides3-CNN] [slides4-RNN]

    • November 1th, 2014. Graph-based active Semi-Supervised Learning: a new perspective for relieving multi-class annotation labor. ICT International Exchange Workshop 2014, Laboratory of Advanced Research B, University of Tsukuba, Japan [Slides]


    Source Code

    • ONI: This project is the implementation of the paper "Controllable Orthogonalization in Training DNNs" (arXiv:2004.00917).

    • StochasticityBW: This project is the implementation of the paper "An Investigation into the Stochasticity of Batch Whitening" (arXiv:2003.12327).

    • IterNorm: This project is the implementation of the paper "Iterative Normalization: Beyond Standardization towards Efficient Whitening" (arXiv:1904.03441).

    • DBN: This project is the Torch implementation of the paper : Decorrelated Batch normalization (arXiv:1804.08450).

    • OWN: This project is the Torch implementation of the paper : orthogonal weight normalization method for solving orthogonality constraints over Steifel manifold in deep neural networks (arXiv:1709.06079).

    • CWN: This project is the Torch implementation of our accepted ICCV 2017 paper: Centered Weight Normalization in Accelerating Training of Deep Neural Networks

    • NormProjection: This project is the Torch implementation of the paper: Projection Based Weight Normalization for Deep Neural Networks (arXiv:1710.02338)

    • Ladder_deepSSL_NP: The reimplementation of Ladder networks with projection based weight normalization. We achieved test errors as 2.52%, 1.06%, and 0.91% on Permunate invariant MNIST dataset with 20, 50, and 100 labeled samples respectively, which is the state-of-the-art results.


    Miscellaneous