Master Thesis: Calibrating LLM-as-a-Judge for Swedish Fluency vs. Adequacy

As Sweden's national center for applied AI, we're on a mission to accelerate the use of AI to benefit our society, our

competitiveness, and everyone living in Sweden. We drive impactful initiatives in areas such as healthcare, energy, and public

services while pushing the boundaries of AI research in fields such as natural language processing, machine learning and AI

security. Join us in harnessing the untapped value of AI to drive innovation and create sustainable value for Sweden.

Automated evaluation is vital for tracking LLM post-training progress. "LLM-as-a-judge" paradigms are highly scalable but often

exhibit severe biases, particularly in non-English languages where they may conflate grammatical fluency with factual adequacy,

heavily penalizing minor linguistic errors.

In WP5, OpenEuroLLM is relying on the JudgeArena framework. For Swedish, we face the challenge that English-centric judge models

misjudge local knowledge or nuanced fluency. This thesis explores whether providing an LLM judge with few-shot, human-calibrated

Swedish examples can effectively decouple fluency from adequacy, aligning the automated Elo scores with human expert ratings.

• Literature study: Review automated evaluation paradigms, LLM-as-a-judge biases, and metrics distinguishing fluency from

adequacy.

• Implementation: Annotate a high-quality Swedish dataset with separate scores for fluency and adequacy. Use these as calibration

prompts for the Judge model in JudgeArena.

• Evaluation: Correlate the calibrated JudgeArena outputs against human judgments on a holdout set of Prelude 9B's generated

responses, quantifying the improvement in Pearson/Spearman correlation.

We are seeking curious, self-driven MSc students eager to work at the frontier of open-weight European AI research (LLMs). You

thrive on empirical discovery, design rigorous experiments, and let data challenge your assumptions.

• Ongoing Master’s studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative

field.

• Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension, alongside comfort running

distributed GPU training in Linux/HPC environments.

At AI Sweden, we are committed to building diverse and inclusive teams. Some positions may be subject to export control

regulations, which means that specific requirements may apply.

Doing your thesis at AI Sweden means working alongside leading AI scientists and change leaders. AI Sweden is Sweden’s National

Center for AI, we drive research questions that have both a long shelf-life and are widely applicable to Swedish industry and the

public sector. We aim for publications at the most competitive venues and celebrate a culture of research excellence.

As an organization, we’re uniquely positioned at the sweet spot of governmental influence and startup agility. Small enough to

stay adaptive and have fun but backed by and in close contact with both the government, academia and private and public sector.

Practical details

Location: Hybrid (Gothenburg / Stockholm) or Remote.

Application Deadline: 2026-11-20 (rolling selection – position may be filled earlier).

Start Date: January 2027

AI Sweden does not accept unsolicited support and kindly ask not to be contacted by any advertisement agents, recruitment agencies

or manning companies.