Freelance Agent Evaluation Engineer
$104,000–$104,000 year
RemoteCanada
Job Summary
Build realistic developer environments containing codebases, infrastructure, and context to simulate believable development histories. Design challenging tasks from intermediate states, defining prompts and success criteria that test AI coding agents' capabilities on real-world developer tasks. Write tests to verify agent solutions, accepting valid approaches while rejecting incorrect ones to ensure evaluations remain fair and robust. Iterate on tasks and tests based on QA feedback, analyzing agent failures to refine evaluation criteria until the assessment is comprehensive. This project-based role involves creating datasets to evaluate AI coding agents for Mindrift, connecting specialists with leading tech companies. Candidates must possess deep understanding of where frontier models fail to craft tasks that genuinely challenge advanced systems.
Required Qualifications
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
- A Master's Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields
- Bachelor's degree is accepted if only candidate has 5 years of experience in the field
- Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.