Principal Data Scientist - Agent Builder
Remote · The Netherlands
Job Summary
Principal Data Scientist role shaping the evaluation strategy for conversational and agentic search within Elastic’s agentic platform. You will define evaluation and quality metrics for RAG, agents, tools, and model routing, prototype and productionize evaluation pipelines, and influence roadmap choices. Responsibilities include turning experimental results into product decisions, collaborating with engineering to deploy telemetry and dashboards, mentoring other data scientists, and communicating trade-offs to product, engineering, and leadership. You will work across retrieval methods (dense/sparse, vector search), context enrichment, and grounding/citation quality, with emphasis on scalable, measurable improvements to chat quality and user value.
Required Qualifications
- 8+ years of applied DS/ML experience
- deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences
- strong track record defining and leading evaluation for production AI/ML systems (offline metrics, online experimentation, LLM-as-judge approaches)
- hands-on with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets
- strong understanding of retrieval systems (dense and sparse retrieval, re-ranking, vector search) and evaluation metrics (nDCG, MRR, Recall@k, precision)
- experience collaborating with engineering to move from prototype to production (telemetry, dashboards, CI guardrails, regression tracking)
- practical Elasticsearch experience or similar search/distributed data systems; ES|QL familiarity is a plus
- excellent written and verbal communication; ability to mentor and lead in a distributed team
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.