Senior Data Scientist - LLM Evaluation & Business Intelligence
$148,000–$184,000 year
RemoteUnited States
Job Summary
Design and run evaluations of production LLM pipelines, developing accuracy and quality metrics that tell us how well our systems perform on real clinical tasks. Measure and improve the effectiveness of our retrieval-augmented generation (RAG) systems, from retrieval quality through final output, while establishing a trustworthy business intelligence foundation with production data models and a shared semantic layer. Partner with AI Engineering to close the loop between evaluation findings and production improvements, and collaborate with non-technical stakeholders to translate ambiguous questions into well-defined data products. Build and maintain production-grade data models using SQL and dbt on Databricks, advancing data governance practices across dbt, Databricks, and Hex. Support both internal and external reporting needs with accurate, well-tested datasets, and mentor junior data scientists on evaluation approaches and technical best practices.
Required Qualifications
- Bachelor's degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a related field
- 4+ years of experience as a data scientist, analytics engineer, ML/data professional, or other highly analytical role in life sciences, biotech, healthcare, or a related industry
- Solid understanding of how production LLM pipelines work
- hands-on experience evaluating LLM or NLP systems — accuracy metrics, error analysis, RAG/retrieval quality, prompt evaluation, or similar
- Strong proficiency in SQL
- experience building production data models in dbt (or a comparable transformation framework)
- Proficiency in at least one programming language
- Python
- PySpark
- Fluency with data science best practices: reusability, version control, documentation, testing, code review, and reproducibility
- Excellent written and verbal communication
- ability to translate technical concepts into actionable insights for non-technical stakeholders
- ability to develop reporting logic collaboratively with them
- Comfort with AI tooling and agentic workflows in day-to-day work
- Comfort with ambiguity
- adaptability to a mission-driven, fast-paced startup environment
Desired Qualifications
- Master's degree or higher in a quantitative field
- experience establishing or working within a semantic layer for organization-wide reporting
- experience integrating LLM-based workflows into production systems
- working closely with ML/AI engineering teams
- Familiarity with clinical data elements (oncology a plus)
- familiarity with the clinical trial industry or research operations
- Strong statistical knowledge, including regression, classification, hypothesis testing, causal inference, or Bayesian methods
- Prior experience mentoring other data scientists or leading cross-functional projects (for senior candidates)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.