Human Data Quality Engineer (Founding Team)
RemoteUnited States
Job Summary
Design quality frameworks and evaluation rubrics for complex human data programs, translating frontier AI lab needs into robust annotation schemas and operational strategies. Build upstream quality systems using Python and SQL, covering recruitment, screening, training, and calibration sessions to engineer quality into the entire lifecycle. Partner with product and engineering teams to develop scalable infrastructure, while investigating data integrity issues to drive root cause analysis and automated checks. Architect dashboards for performance visibility and mentor junior analysts to raise company-wide quality capability. Define best practices for new AI domains and shape how Prolific approaches quality as a strategic priority.
Required Qualifications
- 5+ years of experience in building quality, evaluation or annotation systems within AI, machine learning, LLMs or human data environments
- Strong Python and SQL skills
- A solid understanding of machine learning pipelines and how human data impacts model performance
- Strong analytical and statistical thinking
- experience designing scalable quality frameworks
- The confidence and credibility to interact with stakeholders at frontier labs and act as a partner
- The ability to leverage your experience and expertise to influence and guide stakeholders at every level, both client side and internally
- The ability to turn your own data analysis and quality methodology into requirements that product and engineering can build into systems
- The ability to explain your data analysis and findings clearly to non-technical stakeholders, so they can act on them
- A proactive, builder's mindset -you enjoy creating new systems, navigating ambiguity and improving how things work
Desired Qualifications
- Experience with LLM evaluation, RLHF, AI safety or red teaming
- Experience translating vague model or evaluation goals into clear annotation specifications
- Experience working with human annotation programmes or human data operations
- Familiarity with calibration, inter-rater agreement, drift detection or other evaluation methodologies
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.