Strategic Project Lead
$150,000–$200,000 year
RemoteUnited States or San Francisco, California, United States
Job Summary
Design annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs while managing scope, evaluator staffing, and client relationships from kickoff to final readout. Ensure data quality and inter-rater reliability through real-time course correction, then translate qualitative and quantitative analysis into actionable findings, reports, and workshops. Partner with engineering to build scalable tooling and publish case studies that define market standards. Based in San Francisco, this role requires 3+ years in study design and AI evaluation, with a focus on human-centered research and creative quality judgment.
Required Qualifications
- Based in San Francisco, California
- 3+ years designing studies, running evaluations, analyzing data, and presenting findings
- Experience with data labeling, annotation, human-in-the-loop systems, or AI and ML evaluation workflows
- A track record managing several concurrent projects, including annotation, evaluation, or QA work
- A real point of view on creative quality and what 'good' looks like
Desired Qualifications
- A strong research background, qualitative and quantitative. A Master's or PhD in HCI, cognitive science, psychology, design research, or a related field is a plus, and so is equivalent depth from industry
- Statistical methods for evaluation work: inter-rater reliability, sampling, regression, significance testing
- RLHF, LLM evaluation, or benchmark design
- Ethnography, UX research, or design-study methods
- Startup or 0-to-1 experience
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.