From October 24–29, 2026, the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026) was held in Budapest, Hungary. ScaDS.AI Dresden/Leipzig joined the conference with a wide variety of contributions.

SAC Highlight Award

The paper “Gold vs. Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?” received a Senior Area Chair (SAC) Highlight Award at EMNLP 2026 in Budapest. The work was developed within a joint research network involving the Alexandra Institute, TU Dresden, and ScaDS.AI Dresden/Leipzig.

The award is given by the Senior Area Chairs, who each oversee a research area of the conference, to highlight particularly noteworthy papers. This year, 22 papers received the distinction. In total, EMNLP, one of the leading international conferences on natural language processing, attracted more than 17,000 submissions.

About “Gold vs. Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?”

Many AI language models are evaluated in languages other than English using benchmarks translated from English. But translation does not simply add questions on Danish law, Swedish sports history, the geology of the Estonian island of Hiiumaa or the history of the Sámi Grand Prix if the source benchmark does not contain them. The study therefore asks: do translated benchmarks reveal anything about how well language models capture language- and culture-specific knowledge?

To investigate this, the study uses seven benchmarks originally developed in five Nordic and Baltic languages as its point of comparison. These benchmarks draw on school exams, university entrance exams, professional licensing exams and citizenship tests, or consist of quizzes and factual questions. Alongside general subject knowledge, they also test local knowledge of law, history, geography and culture, to varying degrees depending on the benchmark. The results of 29 language models on these benchmarks were compared with their scores on DeepL translations of MMLU, a multiple-choice benchmark originally written in English that covers a broad range of school and academic knowledge and in part requires familiarity with the US context.

At the heart of the work is a statistical approach that measures how much additional information a translated benchmark provides. It first accounts for what model size and language proficiency already tell us about the benchmark scores. It then determines how much more the translated benchmarks reveals about performance on the seven comparison benchmarks, and how reliable this additional information is.

This step matters because rankings can be misleading: strong models tend to perform well on very different benchmarks. A similar ranking alone therefore tells us little about how much additional information a translated benchmark provides. For example, the rankings on both Swedish exam questions and the Danish citizenship test closely matched those on the translated benchmarks. However, the translated benchmark scores differed in how much they helped explain performance differences between models on the comparison benchmarks. For the Swedish exam questions, this additional contribution was more than ten times as large as for the Danish citizenship test.

This approach makes it possible to measure what had previously only been suspected: translated benchmarks are most informative when the knowledge they test overlaps with that of the native benchmarks, as with academic content. For local knowledge, their value is often limited. Even within the same language, the same translated benchmark provides different amounts of additional information depending on the test it is compared with. This is particularly relevant when language models are intended for use in a country’s public administration, legal system or education sector. The approach helps assess whether translated benchmarks are sufficient for evaluating the knowledge needed in these settings, or whether locally developed test items are required.

Our Contributions to EMNLP 2026

Papers

• Politov, A., Vergunova, I., Jäkel, R., & Färber, M. (2026). Energy-Aware Benchmarking of Generative Information Extraction. In Findings of the Association for Computational Linguistics: EMNLP 2026 (pp. 42668-42691).
• Qu, Z., & Färber, M. (2026). Vital Trace: Protocol-Constrained Patient-State Reasoning for Longitudinal Clinical Trajectories. arXiv preprint arXiv:2602.12833.
• Saeed, M., & Razniewski, S. (2026). LLMpedia: A Transparent Framework to Materialize an LLM’s Encyclopedic Knowledge at Scale. arXiv preprint arXiv:2603.24080.
• Sallouh, A., Popovič, N., & Färber, M. (2026). Does Modern Standard Arabic (MSA) Dominate Arabic Dialects in LLMs? A Representation-Level Analysis. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp.49531–49543).
• Thellmann, K., Stadler, B., Smart, D. S., Färber, M., & Lehmann, J. (2026, October). Gold vs. Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 657-688).
• Wilinski, J., & Färber, M. (2026). SciEmbed: Citation-Context Supervision for Scientific Document Embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2026 (pp. 43793-43819).
• Yuan, S., Im, K., & Färber, M. (2026). Analyzing Bias in False Refusal Behavior of Large Language Models for Hate Speech Detoxification. In Findings of the Association for Computational Linguistics: EMNLP 2026 (pp. 6741-6769).

Workshop Contributions

11th Workshop on Automated Knowledge Base Construction (AKBC 2026)

• Ghosh, S., Giordano, L., Hu, Y., Nguyen, T. Razniewski, S. (2026). Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5. In Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026) (pp. 8-18).
• Razniewski, S., Ghosh, S., Giordano, L., Hu, Y., Kowalzik, J., Nguyen, T. (2026). Epistemic Twins: Enabling a Symbolic Science of Language Model Knowledge. In Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026) (pp. 1-7).
• Schrader, T., Friedrich, A., Razniewski, S., Lange, L. (2026). The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations. In Proceedings of the 11th Workshop on Automated Knowledge Base Construction (AKBC 2026) (pp. 45-55).

Third Workshop on Uncertainty-Aware NLP

• Müller, P., Popovič, N., Färber, M. & Steinbach, P. (2026). Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA. arXiv preprint arXiv:2602.00279.

System Demonstrations

• Hu, Y., Nguyen, T., Razniewski, S. (2026). GPTKB 2.0: Browsing, Querying, and Auditing a Disambiguated LLM-Derived Knowledge Base. arXiv:2608.06992

EMNLP 2026

EMNLP 2026 brought together researchers from all over the world, sharing research findings in a variety of natural language processing (NLP) related fields, including Information Retrieval and Text Mining, LLM Agents, Machine Learning for NLP, and more. The Special Interest Group on Linguistic Data & Corpus-based Approaches to Natural Language Processing (ACL’s SIGDAT) organized the conference.