Member of Technical Staff, Forward Deployed
RemoteUnited States or Brooklyn, New York, United States
Job Summary
Build high-fidelity environments, evals, and datasets for frontier AI research, translating researcher hunches into shipped deliverables that measure model capabilities. Transform loosely defined research questions into concrete task distributions, grading logic, and scoring harnesses, then generalize proven one-offs into core product features. Communicate directly with technical customers to push back on misaligned requests and validate that outputs measure intended failure modes rather than ease of construction. Own whole deliverables end-to-end, moving from custom builds to generalizable infrastructure while learning task design, LLM-judged scoring, and reward hacking detection. Work within a small team of ex-Stripe, Snap, and AWS engineers to solve the hardest problems currently unsolved in the field.
Required Qualifications
- Full-stack range
- Strong generalist engineer
- Comfortable across backend services
- Data pipelines
- Enough frontend to ship a usable interface
- TypeScript/Python or similar
- Ability to turn a loosely-defined research question into a concrete environment, task set, eval, or dataset
- Ability to validate that it measures what was meant
- Ability to communicate clearly with researchers and technical customers
- Ability to push back when they're asking for the wrong thing
- Ability to know the difference between what someone requests and what they need
- Experience shipping production software
- Interest in how agents fail
- Interest in how to measure agent failure
Desired Qualifications
- ML research background
- Experience with task design
- Experience with LLM-judged scoring
- Experience with reward hacking detection
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.