Large Language Models (LLMs) are increasingly deployed in research and real-world applications, where they interact with sensitive data, external tools, and human users. Despite their capabilities, LLMs introduce novel security and privacy challenges, including prompt injection, data leakage, and unintended disclosure of sensitive information. These risks stem from the fundamental design of LLMs, which interpret natural language instructions without reliable mechanisms to distinguish trusted from adversarial inputs. As a result, ensuring the resilience, robustness, and trustworthiness of LLM-based systems has become a critical research challenge. Therefore, on September 3–4, 2026, ScaDS.AI Dresden/Leipzig invited interested people to join the workshop Resilient AI: Securing Large Language Models in Dresden.

The workshop provided an overview of how LLMs work, introduced common attack scenarios, discussed privacy risks, and presented methods for adding safeguards and protective mechanisms when building LLM-based systems. 17 people participated, laying the foundation for a well-rounded discussion on resilient AI.

Program

Day 1

The first day of the workshop began with an introduction by Prof. Simon Razniewski (Chair of Knowledge-aware Artificial Intelligence). He motivated LLM security through real-world incidents, while giving an accessible introduction to LLM internals: tokenization, context windows, transformer architecture, training stages (pre-training, fine-tuning, RLHF), prompt roles, and how generation works. Dr. Himanshu Beniwal continued with a presentation of the main attack categories: prompt injection, jailbreaking techniques, training data extraction, hallucination, and supply-chain risks, such as system prompt theft and model poisoning.

Together with Dr. Paramita Mirza and Dr. Siavash Ghiasvand, the participants afterwards attacked a pre-built chatbot through staged challenges, including role-play attacks, delimiter manipulation, and indirect injection. All attacks aimed to extract the hidden system prompt or to bypass safety checks. In the following debrief, the group compared which attacks succeeded and why. A discussion of what the participants would do differently as developers followed, closing the first day.

Day 2

On the second day, Prof. Hermann Diebel-Fischer (Evangelische Hochschule Dresden) examined how LLMs expose sensitive information, for example by training data memorization, PII leakage, differential privacy and its limits, GDPR requirements, and re-identification attacks. Dr. Himanshu Beniwal continued with an overview of surveys defensive tools, including input validation, output filtering, alignment techniques, LLM-as-judge, and prompt firewalls, along with their limitations.

This was followed by a discussion of the design of secure LLM systems. Afterwards, Dr. Paramita Mirza and Dr. Siavash Ghiasvand explored with the participants how to exploit a deliberately vulnerable RAG pipeline, and how to fix this issue with access control and output filtering. The wrap-up consisted of group presentations of the findings and a practical pre-shipping checklist. The Resilient AI workshop closed with further reading and next steps.

Outlook

The Resilient AI workshop brought together experts and practitioners to discuss how to make large language models more secure. The whole event, and the Q&A session in particular, led to very engaging discussions about real life scenarios and problems. Building on the positive feedback of the participants, similar events will be offered in the future. Check out our event calendar to not miss out on similar events.