Title: Efficient large-scale deep learning on the SpiNNaker2 Neuromorphic Supercomputer

Duration: 01.07.2022 – 30.06.2025

Research Area: Methods and Hardware for Neuro-Inspired Computing

The SpiNNcloud is a neuro-inspired supercomputer based on 5 million low-power ARM microcontrollers that operate asynchronously and communicate in a packet-based fashion. It is by far the largest brain-inspired computer worldwide and offers a unique potential for energy-efficient real-time AI processing. In this project, we want to develop and implement scalable deep learning models on the SpiNNaker2 supercomputer. In addition to vanilla models such as CNN, ResNet and Transformers, neuro-inspired models will be developed that e.g., make use of structural and temporal sparsity. This makes it much easier to implement these models on the SpiNNaker2 system, thus avoiding communication bottlenecks by a scalable event-based communication. The efficient deep learning models will be benchmarked against standard hardware for machine learning.

Further, a sustainable workflow will be established to ensure reproducible and scalable research on the SpiNNcloud for a wide user community.

Aims

The aim is to develop and implement state-of-the-art deep learning models on the SpiNNcloud supercomputer, requiring less energy than on conventional AI hardware like NVIDIA GPUs. This is to be shown for various architectures such as convolutional neural networks, recurrent neural networks and transformers. Technically, the aim is to develop modified DNN models that are suitable for the event-based SpiNNaker2 platform and to implement a scalable software stack for mapping these DNN models.

Problem

The brain processes information order of magnitudes more efficiently than AI models on GPUs. Yet, brain-inspired models like spiking neural networks are lacking behind DNNs in terms of performance. The challenge is to apply concepts from neuroscience into DNNs to achieve both high performance and energy efficiency. Another challenge is to quantize the models to 8-bit integers for efficient inference with the SpiNNaker2 DNN accelerators.

Practical Example

Event-based recurrent neural networks (EGRU) were developed in collaboration with D. Kappel (RUB) and A. Subramoney (RHUL) [Subramoney et al 2023]. EGRU models were implemented on SpiNNaker2 for language modelling and event-based vision, requiring 18x less energy than a NVIDIA A100 [Nazeer et al. 2024].

Technology

We use the SpiNNcloud, a massively parallel neuro-inspired supercomputer with highly efficient DNN accelerators for int8 operations. To increase efficiency, we leverage neuro-inspired concepts like event-based communication and structural sparsity. Post-training quantization and quantization-aware training will help to prepare DNN models for efficient inference on SpiNNaker2. State-of-the-art approaches from ML compilers are adopted for the efficient mapping of the models to SpiNNaker2.

Outlook

After implementing standard models such as ResNets and Transformers on single-chip SpiNNaker2 systems, we aim to scale up the network models to run on multiple 48-chip server boards. In addition, more efficient transformer models for natural language processing shall be developed and deployed on the SpiNNcloud supercomputer.

Publications

• Subramoney, A., Nazeer, K. K., Schöne, M., Mayr, C., & Kappel, D. (2023). Efficient recurrent architectures through activity sparsity and sparse back-propagation through time. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=lJdOlWg8td
• Gonzalez, H. A., Huang, J., Kelber, F., Nazeer, K. K., Langer, T., Liu, C., … & Mayr, C. (2023). Spinnaker2: A large-scale neuromorphic system for event-based and asynchronous machine learning. In  First Workshop on Machine Learning with New Compute Paradigms, 2023, https://openreview.net/forum?id=KAiPD1OwvF
• Nazeer, K. K., Schöne, M., Mukherji, R., Mayr, C., Kappel, D., & Subramoney, A. (2024). Language Modeling on a SpiNNaker 2 Neuromorphic Chip. Accepted for AICAS 2024, Preprint: https://doi.org/10.48550/arXiv.2312.09084

Team

Lead

• Prof. Dr.-Ing. Christian Mayr

Team Members

• Chen Liu
• Tim Langer
• Matthias Jobst
• Mark Schöne
• Khaleel Khan