Freelance Agent Evaluation Engineer
$83,200–$83,200 year
RemotePortugal
EXPIREDPortugalRemotePart Time$83,200–$83,200 yearMasters DegreeMedium
Part TimeMasters DegreeMedium
Job Summary
Build realistic developer environments with codebases, infrastructure, and context to simulate believable development histories. Design challenging tasks from intermediate states, defining prompts and success criteria for AI coding agents. Write tests that accept valid solutions while rejecting incorrect ones, then iterate on evaluations based on QA feedback to ensure robustness. This project-based role requires creating tasks that genuinely test frontier models' capabilities in real-world developer scenarios. Compensation is paid per accepted task, with rates up to $40/hr depending on qualification tier and efficiency.
Required Qualifications
- 5+ years in software development
- Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
- Experience writing tests (functional, integration)
- English proficiency - B2+
- A Master's Degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields
- Bachelor's degree is accepted if only candidate has 5 years of experience in the field
- Candidates should have a minimum of 3 years of professional experience in related roles or domain - specifically for QA-automation/testing or cybersecurity roles
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.