Research Engineer, Benchmarking - Member of Technical Staff
On-siteLondon, England, United Kingdom
Job Summary
Build a unified system for evaluating agentic and algorithmic solutions, measuring task success, quality, and robustness across motifs, agent topologies, and decomposition strategies. Mine real commits and traces to run sandboxed execution grading, ensuring results are reproducible and free from contamination or metric gaming. Design task suites reflecting real agentic workloads like code search, edit, repair, and tool use, then compare approaches honestly to determine which ship. Lead external benchmark co-publications and feed results directly into customer engagements and internal quality reviews. This research hire applies the rigor of a strong evaluation paper to a production system, working with the General Capabilities Team to establish trusted evidence for the next era of heterogeneous AI infrastructure.
Required Qualifications
- PhD in computer science, machine learning, or a related field, or an equivalent research track record
- Authorship or co-authorship of a benchmark or evaluation paper at a recognised venue - NeurIPS Datasets and Benchmarks, ICML, ICLR, ACL
- A working understanding of how LLM and agent evaluation goes wrong: contamination, overfitting to benchmarks, weak baselines, underpowered comparisons, irreproducible results
- The engineering ability to design, build, and curate these systems decisively - strong Python, and comfort with sandboxed and distributed execution and CI
- Hands-on experience building or rigorously evaluating agentic or multi-step LLM systems
Desired Qualifications
- Published agentic or tool-use benchmarks that use execution-based grading
- Experience running sandboxed execution grading at scale
- Open-source evaluation or harness tooling
- Familiarity with code-agent workloads such as search, edit, and repair
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.