Technical AI Evaluator - Python, Automation or Security

Build realistic developer environments with codebases, infrastructure, and context to form believable development histories. Design tasks from intermediate states by crafting prompts, defining success criteria, and ensuring solvability for AI agents. Write tests that verify agent solutions, accepting all valid approaches while rejecting incorrect ones with balanced rigor. Iterate on tasks and tests based on QA feedback, reviewing agent solutions and analyzing failures to refine evaluations until they are fair and robust. This project-based role requires 5+ years in software development with a core stack of Python, JavaScript, Docker, Postgres, Kafka, and Redis, plus B2+ English proficiency. Compensation reaches up to $50/hr equivalent for tasks estimated at ~20 hours each, with participants setting their own schedule.