AI Agent Quality Engineer
On-siteTaguig, Metro Manila, Philippines
Job Summary
Build and scale an automated evaluation suite that stress-tests AI agents against prompt injections, sycophancy, and multi-attempt attacks before production approval. Design adversarial scenarios to extract sensitive data or force agents outside tool boundaries, then integrate these into CI/CD pipelines for unattended fleet-wide execution. Own the "break it on purpose" pass by establishing automated thresholds for pass/fail criteria and maintaining a guardrail catalog for audit-ready evidence. Track drift across the entire agent fleet to catch slow-moving security issues, ensuring every release meets strict quality and compliance bars.
Required Qualifications
- At least 4 years in software quality, security testing, or a related discipline
- 1–2 years specifically evaluating or red-teaming LLM-based systems
- Strong Python skills
- Comfortable writing production-quality code
- Working knowledge of REST APIs and webhook/event-driven patterns
- Real experience building automated test infrastructure and integrating it into CI/CD
- Hands-on familiarity with at least one adversarial testing or LLM eval tool (DeepTeam, Garak, PyRIT, Promptfoo, or similar)
- Understanding of how these map to standards like the OWASP LLM Top 10 or NIST's AI risk framework
- Practical understanding of prompt injection, jailbreaking, and sycophancy failure modes
- Ability to design new test cases for prompt injection, jailbreaking, and sycophancy
- Willingness to block a release when an agent doesn't meet its bar
Desired Qualifications
- Enough understanding of data sensitivity and compliance classification to design tests that reflect real risk
- Experience in a regulated environment where evaluation results had to hold up to an external audit
- Familiarity with multi-attempt or persistent-attack testing methodology
- Some exposure to how agents are actually built (prompting, tool schemas, orchestration)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.