Title: Scaling transformer neural networks to gigapixel images

Project duration: 5 years

Research Area: Applied AI and Big Data

Our project focuses on developing transformer neural networks for gigapixel image classification, tackling the challenge of processing large-scale pathology images using advanced AI techniques. Pathology images are gigapixel-sized, making them unsuitable for standard neural network architectures. Recently, transformer neural networks have achieved state-of-the-art performance in image processing. Therefore, we aim to develop new transformer-based architectures for gigapixel image classification and compare them with existing methods. We face technical challenges in designing an architecture that can handle images of varying sizes and can be trained efficiently on large datasets. Moreover, the architecture needs to process gigapixel images directly without tiling or using a sliding window. Our goal is to create an end-to-end solution that processes whole slide images and outputs classifications. Additionally, we will explore the explainability of transformer models and compare them to other architectures.

Aims

• Development of transformer neural networks to analyse gigapixel-sized images of histopathology slides from different types of cancer
• Application of the improved method for direct prediction of clinical biomarkers and cancer classification from histopathology images
• Investigation of the explainability of the transformer models and comparison to other architectures

Problems

Currently, workflows in computational pathology lack end-to-end solutions: Large histopathology slide images must be tiled and analyzed in patches. This process causes loss of information. Moreover, the lack of global context in individual patches limits classification performance. In addition, optimizing the tiling strategy can be challenging.

Technology

In this project, we will develop and adapt transformer neural networks for gigapixel images to address limitations in computational pathology. The advantage of this approach is that it processes the whole slide image at once, providing direct classification without the need for tiling.

Outlook

Overall, our novel analysis method for gigapixel images will enable the analysis and classification of large microscopic images as a whole. Additionally, our transformer neural network can be applied to analyze different cancer types and relevant biomarkers.

Publications

• El Nahhas, O.S.M., Loeffler, C.M.L., Carrero, Z.I. et al. Regression-based Deep-Learning predicts molecular biomarkers from pathology slides. Nat Commun 15, 1253 (2024). https://doi.org/10.1038/s41467-024-45589-1
• Tayebi Arasteh, S., Misera, L., Kather, J.N. et al. Enhancing diagnostic deep learning via self-supervised pretraining on large-scale, unlabeled non-medical images. Eur Radiol Exp 8, 10 (2024). https://doi.org/10.1186/s41747-023-00411-3

• Foersch, S., Glasner, C., Woerl, AC. et al. Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer. Nat Med 29, 430–439 (2023). https://doi.org/10.1038/s41591-022-02134-1

• Saldanha, O.L., Quirke, P., West, N.P. et al. Swarm learning for decentralized artificial intelligence in cancer histopathology. Nat Med 28, 1232–1239 (2022). https://doi.org/10.1038/s41591-022-01768-5

• Kather, J.N., Pearson, A.T., Halama, N. et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 25, 1054–1056 (2019). https://doi.org/10.1038/s41591-019-0462-y

• Kather JN, Krisam J, Charoentong P, Luedde T, Herpel E, Weis C-A, et al. (2019) Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med 16(1): e1002730. https://doi.org/10.1371/journal.pmed.1002730

• Kather, J.N., Heij, L.R., Grabsch, H.I. et al. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat Cancer 1, 789–799 (2020). https://doi.org/10.1038/s43018-020-0087-6

Team

Lead

• Prof. Dr. Jakob Nikolas Kather (TU Dresden)

Team Members

• Omar El Nahhas, MSc
• Marco Gustav, MSc

Partners

• Else Kröner Fresenius Center (EKFZ) for Digital Health
• Dresden University Hospital Carl Gustav Carus
• TUD Dresden University of Technology