Data Scientist / AI Models
On-siteMadrid, Madrid, Spain
Job Summary
Evaluate and compare LLM models from OpenAI and Anthropic regarding performance, latency, and costs, while monitoring changes and deprecations to assess their impact on production environments. Optimize prompting strategies based on model capabilities and implement testing frameworks for GenAI solutions. Execute regression, prompt, and output evaluations, designing pipelines for model assessment and monitoring. Develop dashboards for result tracking and collaborate with IAOps/MLOps teams on observability, logging, and model governance. Generate recommendations for model selection, cost optimization, and risk mitigation throughout the AI model lifecycle. Requires over two years of experience with LLMs in production, GenAI auditing expertise, and AWS proficiency including Bedrock, SageMaker, Lambda, and CloudWatch. Fluency in Spanish and English is required for this hybrid role in Madrid.
Required Qualifications
- Experiencia de más 2 años en posiciones similares
- Titulación universitaria en Ingeniería Informática, Telecomunicaciones, Matemáticas u otras titulaciones STEM relacionadas
- Experiencia sólida con LLMs en entornos productivos
- Experiencia en GenAI enfocada en auditing, evaluating y monitoring
- Experiencia con AWS: Bedrock, SageMaker, Lambda y CloudWatch
- Conocimiento en Prompt Engineering
- Experiencia con frameworks de evaluación de LLMs como RAGAS, DeepEval o similares
- Experiencia en A/B testing y experimentación
- Conocimiento de observabilidad y monitorización de soluciones IA
- Comprensión del gobierno de IA y ciclo de vida de modelos
- Nivel fluido de español e inglés
- Modalidad híbrda Madrid
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.