Multilingual AI Quality Specialist
On-siteLondon, England, United Kingdom or Stockholm, Stockholm, Sweden
Job Summary
Define quality frameworks, evaluation rubrics, and methodologies for multilingual AI experiences, including structured evaluations for AI-generated, translated, and recommendation-driven content. Lead dataset curation, annotation, and ground-truth creation to support model development, while analyzing results to identify quality gaps and inform launch decisions. Partner with Product, Engineering, Data Science, and Localization teams to improve quality signals, document best practices across languages and markets, and build scalable evaluation capabilities. This role is based in London or Stockholm with flexibility for remote work, requiring experience in multilingual quality evaluation, LLM-as-a-judge workflows, and cross-functional collaboration.
Required Qualifications
- experience in multilingual quality evaluation, localization, data curation, annotation, AI evaluation, or related fields, including text-to-text and text-to-speech experiences
- understand language quality, cultural relevance, content quality, and user experience across multiple languages and markets
- experience designing or conducting structured evaluations using quality rubrics, audits, annotation projects, or review methodologies
- comfortable using qualitative and quantitative data to identify trends, measure quality, and make recommendations
- familiar with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies
- enjoy working through ambiguity and turning complex quality challenges into practical evaluation strategies
- communicate effectively and thrive in highly cross-functional environments, collaborating with technical and non-technical partners alike
- based in London or Stockholm
Desired Qualifications
- Experience with recommendation systems, personalization, search, ranking, machine translation, generative AI, dataset creation, annotation operations, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms is a plus
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.