Investigating the Impact of Generative Language Models on Students‘ Learning

How can AI chatbots meaningfully support students in learning, organizing their studies, and navigating university life? This doctoral project combines Human–Computer Interaction, participatory design, and technical development to investigate how generative language models can complement existing university support structures.

This PhD-research follows three connected phases:

1. Understanding student needs: A scoping review and interviews with 30 female STEM students identify support needs and design requirements.
2. Designing with students: The findings are translated into six personas using empathy mapping and validated with 45 students. Participatory focus groups explore chatbot concepts and requirements such as transparency, trust, and accessibility.
3. Developing and evaluating prototypes: The course-specific chatbot ST-Buddy combines LLM-based dialogue with the retrieval of course materials. Its evaluation with 32 students examines usability, helpfulness, and response quality.

Further evaluations with broader student groups and support scenarios will provide a more comprehensive understanding of how educational chatbots can support diverse needs in higher education.

ST-Buddy

A course-specific conversational agent developed in 2024 in cooperation with the Chair of Software Technology at TU Dresden. It provides students with low-threshold academic and organizational support and remains in active use. ST-Buddy combines generative language models with retrieval-based access to course materials, enabling context-sensitive answers grounded in relevant learning content and course information. The system has been iteratively evaluated in an introductory Software Technology course to explore its usability, perceived helpfulness, and potential to complement existing teaching and tutoring structures. Its further development was funded by the TU Dresden Fonds DLL.

Cooperations

Try out our metis platform?

metis.scads.ai

click here