Product Data Scientist — AI Evaluation & Quality
HybridAmsterdam, North Holland, The Netherlands
Amsterdam, North Holland, The NetherlandsHybridFull TimeDoctorate Or Professional DegreeMedium
Full TimeDoctorate Or Professional DegreeMedium
Job Summary
Own and extend the offline eval suite across products by managing datasets, judges, and metrics. Build and maintain online quality dashboards tracking resolution rates, CSAT, and error signals. Close the production feedback loop by mining failure patterns from real traffic to create regression cases and propose fixes. Harden methodology for judge stability and non-determinism handling. Translate weekly metrics into clear product trade-offs with the AI Quality lead and domain experts. Leverage Python, SQL, and Claude Code to analyze end-to-end.
Required Qualifications
- Python
- SQL
- Solid foundation in statistics
- Analytical mindset
- 3+ years in analyst / data scientist roles
- at least one in a product context
Desired Qualifications
- Experience in quality analytics for ML systems
- Hands-on experience evaluating LLM applications
- Experience building LLM agents
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.