Freelance Agent Evaluation Engineer
$83,200–$83,200 year
RemoteSouth Africa
Job Summary
Build realistic developer environments with codebases, infrastructure, and context to form believable development histories. Design tasks from intermediate states by crafting prompts and defining success criteria. Write tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones. Iterate on tasks and tests based on QA feedback to ensure evaluations are fair and robust. This project-based role focuses on testing and improving AI coding agents, not data labeling or prompt engineering. Candidates with 5+ years in software development and a Master's degree in CS or related fields apply. Compensation is paid per accepted task, up to $40/hr.
Required Qualifications
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
- A Master's Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields
- Bachelor's degree is accepted if only candidate has 5 years of experience in the field
- Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.