Principal Data Scientist - Agent Builder
Remote
Job Summary
Principal Data Scientist - Agent Builder leads the evaluation and quality strategy for Elastic’s conversational/agentic search platform. Defines evaluation metrics and decision frameworks for RAG, agents, tools, and model routing; guides improvements across retrieval, vector search, and context enrichment; translates experimental results into product decisions; partners with engineering to productionize evaluation pipelines and dashboards; mentors others and shapes roadmap with cross-functional teams. Requires extensive hands-on DS/ML experience (IR/NLP/semantic search), strong evaluation leadership,proficiency in Python and PyTorch/Transformers, experience with Elasticsearch or similar systems, and excellent communication to product/leadership audiences.
Required Qualifications
- 8+ years of applied DS/ML experience with deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences
- Strong track record defining and leading evaluation for production AI/ML systems including offline metrics, online experimentation, LLM-as-judge approaches, groundedness, citation quality, and model comparison
- Experience influencing product and technical strategy through data in ambiguous domains
- Hands-on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code
- Strong understanding of retrieval systems including dense and sparse retrieval, re-ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, Recall@k, precision, and latency/cost trade-offs
- Experience collaborating closely with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking
- Practical Elasticsearch experience, or experience with similar search and distributed data systems; ES|QL familiarity is a plus
- Excellent written and verbal communication, with ability to explain complex scientific and technical trade-offs to engineering, product, design, and leadership audiences
- A collaborative, low-ego style and a strong ability to mentor, raise standards, and develop transparency for others in a distributed team
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.