Serop Baghdadlian

Advisors
An Nguyen (M.Sc.), Dr. Dario Zanca, Dr. Sebastian Lapuschkin (Frauenhofer HHI), Leander Weber (M.Sc.) (Frauenhofer HHI), Prof. Dr. Björn Eskofier

Duration
06/2021 – 12/2021

Abstract
Deep neural networks are the current state-of-the-art solution for many machine perception problems such as computer vision and speech recognition. Adding extra knowledge to an extensively trained neural network is a crucial step towards achieving general intelligence, however, this is still limited due to the problem of catastrophic forgetting (CF) [1]. Current work for minimizing this effect involves adding regularisation terms for the network parameters [2], sample replay techniques [3][4], genetic algorithms for finding the optimal paths for relearning [5]. Despite all these efforts the problem for CF still persists and a solution is therefore needed.

The goal of this thesis is to implement a robust toolbox for mitigating the CF problem using the explainable AI method Layer-Wise Relevance Propagation (LRP) [6], which decomposes the neural network output into proportionate contributions of different units within the network. The toolbox therefore should be able to:

• Assign relevance score to the different units within the neural networks.
• Neural network pruning for keeping the most relevant units w.r.t specific task and freeing up the rest network capacity.
• Costume training by assigning different learning rates to different parts of the network.

All implementations are developed, evaluated, and compared on benchmark datasets, which are used in literature. Real-world datasets could also be used for evaluating the applicability of the proposed approach. Additionally, the choice of the architecture should be comparable with literature and sequential learning tasks have to be chosen from literature for comparison.

References:
[1] Ronald Kemker. Measuring Catastrophic Forgetting in Neural Networks. In: Proceedings of the AAAI Conference on Artificial Intelligence 32.1, April 2018.
[2] James Kirkpatrick. Overcoming catastrophic forgetting in neural networks. In: Proceedings of the National Academy of Sciences 114.13, March 2017.
[3] Zhizhong Li, and Derek Hoiem. Learning without Forgetting. In: arXiv:1606.09282, February 2017.
[4] Hanul Shin. Continual Learning with Deep Generative Replay. In: arXiv:1705.08690 December 2017.
[5] Chrisantha Fernando. PathNet: Evolution Channels Gradient Descent in SuperNeural Networks. In: arXiv:1701.08734 January 2017.
[6] Montavon, Gregoire, Alexander Binder, Sebastian Lapuschkin,Wojciech Samek, and Klaus- Robert Mueller. Layer-Wise Relevance Propagation: An Overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning Cham. In Springer International Publishing, 2019.