Title: Reinforcement Learning with Active Particles 

Duration: 2 years 

Research Area: Reinforcement Learning, Physics and Chemistry

Living organisms adapt their behavior to their environment to achieve specific goals. They sense, process, and encode environmental information into biochemical processes, resulting in appropriate actions or properties. These adaptive processes occur within an individual’s lifetime, across generations, or over evolutionary timescales, leading to distinct behaviors in individuals and collectives. Examples include swarms of fish and flocks of birds developing collective strategies against predators and optimizing foraging tactics, birds learning to use convective air flows, sperm evolving complex swimming patterns for chemotaxis, and bacteria expressing specific shapes to follow gravity.

In this project, we implement learning in systems of microswimmers, which are tiny machines or microrobots that can explore micrometer length scales by self-propulsion. They provide potential applications in drug delivery but also provide model systems for many large-scale self-propelled systems studying collective motion, self-organization and adaptive behavior.

Aims

This project explores the adaptive navigation of active particles in real world environments by reinforcement learning algorithms. We would like to explore how specific behaviors in biological species appear based on specific environmental properties and sensorial inputs. We aim to extend the application of reinforcement learning techniques to multi-agent reinforcement learning with direct experimental control to address the reality gap between simulated learning in virtual environments and real-world learning.

Problem

The behavior of microorganisms is highly optimized by evolutionary pressures from past environmental conditions, yet the specific historical factors shaping these biochemical feedback circuits are often unknown. This lack of historical knowledge complicates the projection of current behaviors onto present environmental properties. A potential solution involves using reinforcement learning approaches to control microswimmers, which are artificial or engineered microorganisms representing model systems.

Reinforcement Learning and computer control allows these microswimmers to adaptively optimize their behaviors in real-time based on current environmental feedback, effectively learning and responding to stimuli in a manner like natural evolutionary processes. By training Reinforcement Learning algorithms to adjust the microswimmers’ responses to varying environmental conditions, we can achieve precise control over their movements and functions, improving their efficiency in applications such as targeted drug delivery, environmental sensing, and micro-manipulation. This approach bridges the gap between historical evolutionary optimization and current environmental adaptability, leveraging advanced computational techniques to enhance the functionality of microswimmers in complex and dynamic environments.

The information gained from studying these relations and transferring them to biological systems can be very useful for new biotechnological systems including bioreactors but also for drug delivery with microrobots or optimized traffic by studying collective behaviors in multi-agent systems.

Technology

Our experimental approaches implement various elements of machine learning and combine these algorithms with the hardware control of an optical microscopy setup to steer active particles.

1. Video-Rate Active Particle Tracking and Classification. We use and adapt single shot convolutional neural networks for the tracking and classification of single active particles at video rate.
2. The implementation of Active Particle Feedback Control. We develop optical feedback control algorithms to self-organize active particles into recurrent active units.
3. Reinforcement Learning Algorithms. We use simple Q-learning algorithms and develop more advanced Deep Q-, Actor Critic Reinforcement Learning and PPO algorithms to directly learn navigation and search strategies based on the dynamics of active particles in liquid environment in real-time.

Outlook

Our project employs advanced experimental control of optical microscopy to investigate the dynamics of active microparticles in liquids. Currently, the focus is primarily on single active particles, but we aim to extend this to ensembles of active particles in complex dynamic environments, including external flows and fields, to study their multi-agent response functions resulting from environmental interactions. These systems exhibit spatial and temporal correlations, and our research could be expanded to include transformer models, which are highly effective at uncovering temporal correlations. Exploring spatio-temporal representation learning in these systems and its connection to real-world physical mechanisms could significantly benefit future AI developments.

Publications

• Volpe, G. et al. Roadmap on Deep Learning for Microscopy. arXiv (2023) doi:10.48550/arxiv.2303.03793.
• Cichos, F., Landin, S. M. & Pradip, R. Intelligent Nanotechnology. 113–144 (2023) doi:10.1016/b978-0-323-85796-3.00005-6.
• Pradip, R. & Cichos, F. Deep reinforcement learning with artificial microswimmers. Emerg. Top. Artif. Intell. (ETAI) 2022 12204, 122040F-122040F–7 (2022).
• Muiños-Landin, S., Fischer, A., Holubec, V. & Cichos, F. Reinforcement learning with artificial microswimmers. Sci Robotics 6, (2021).
• Fränzl, M., Muiños-Landin, S., Holubec, V. & Cichos, F. Fully Steerable Symmetric Thermoplasmonic Microswimmers. Acs Nano 15, 3434–3440 (2021).
• Fränzl, M. & Cichos, F. Active particle feedback control with a single-shot detection convolutional neural network. Sci. Rep. 10, 12571 (2020).
• Cichos, F., Gustavsson, K., Mehlig, B. & Volpe, G. Machine learning for active matter. Nat Mach Intell 2, 94–103 (2020).

Team

Lead

• Prof. Dr. Frank Cichos 

Team Members

• M. Asif Hasan
• Dr. Diptabrata Paul

Partners

• Dr. Nico Scherf
• Prof. Dr. Ostap Okhrin