AI Evaluation Engineer (Python, QA or Security)

Build realistic developer environments with codebases, infrastructure, and context to simulate believable development histories. Design challenging tasks from intermediate states, defining prompts, success criteria, and solvability for AI coding agents. Write functional and integration tests that verify agent solutions, accepting valid approaches while rejecting incorrect ones. Iterate on tasks and tests based on QA feedback, analyzing agent failures to refine evaluations until they are fair and robust. This project-based role focuses on testing and improving AI systems for leading tech companies, with compensation up to $50/hr equivalent.