Task Development Engineer
$312,000–$624,000 year
Remote
RemoteContract$312,000–$624,000 year
Contract
Job Summary
Develop difficult, novel tasks for frontier AI models, ensuring they remain challenging as capabilities evolve. Verify that existing tasks are solvable and contain only the necessary information for models. Baseline and score task completions from AI systems or human baseliners within your domain of expertise. Improve task development infrastructure by identifying and fixing inefficient workflows or low-quality outputs. Build well-scoped tasks supporting the Time Horizons methodology used by policymakers and national security stakeholders.
Required Qualifications
- Several years of experience working on complex projects and codebases
- Experience building hard (ideally agent-based) AI evaluations
- Experience building AI evaluations using the Inspect framework
- Experience building AI evaluations such as RE-Bench, HCAST, SWE-bench Verified, Cybench, or GPQA
- High attention to detail
- Ability to read closely, spot misspecifications and ambiguity, and pay attention to fiddly minutiae
- Ability to develop difficult, novel tasks for models
- Ability to build well-scoped tasks that remain challenging as model time horizons grow
- Ability to verify that tasks are actually solvable as specified
- Ability to ensure models are given only the information they need
- Ability to baseline tasks within a domain of expertise
- Ability to score task completions from AIs or human baseliners
- Ability to notice when existing workflows are inefficient or produce low-quality output
- Ability to take responsibility for improving inefficient workflows
- Minimum of 1 hour of overlap with the Pacific Coast Time workday
Desired Qualifications
- Familiarity with METR infrastructure
- Prior experience with Hawk
- Familiarity with the methodology behind METR's Time Horizons work
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.