Senior Software Development Engineer in Test - AI Tooling & Automation
$148,500–$223,900 year
On-siteSan Francisco, California, United States or Palo Alto, California, United States
Job Summary
Design and maintain scalable, AI-first testing frameworks and automation suites to validate complex Product Led Growth workflows and agentic capabilities. Execute quality strategies by translating high-level plans into actionable automation focused on validating LLM integrations, prompt accuracy, and deterministic system behaviors. Perform deep white-box testing to identify quality gaps, fix code flakiness, and support evaluation processes for subjective AI dimensions like response relevance and user safety. Collaborate closely with developers, product managers, and AI engineers to integrate quality early throughout the agile delivery lifecycle.
Required Qualifications
- Strong Java Skills: Strong technical fundamentals of Java, particularly in data structures and algorithms
- Agile Delivery at Pace: Proven track record of delivering automation solutions in a fast-paced CI/CD environment, supporting rapid deployment schedules without compromising quality
- Collaboration & Communication: Excellent communication skills, with the ability to work cross-functionally with product managers, engineers, AI/ML engineers, and data scientists
- Problem-Solving & Analytical Skills: Strong analytical and problem-solving abilities with a focus on identifying quality gaps and driving improvement
Desired Qualifications
- AI/LLM Validation Mindset: Hands-on experience evaluating Large Language Models (LLMs) for quality response, including familiarity with GPT-like models and AI evaluation frameworks is a big plus
- A software engineer with a strong coding background and understanding of delivering software at pace in Agentic PDLC
- Strong experience with white-box testing and fixing code is a huge plus
- Experience working with public cloud infrastructure (AWS/Azure/GCP) and/or foundational knowledge of the Salesforce platform (Salesforce Certifications are a plus)
- Prior experience testing autonomous agents, conversational UIs, or generative AI workflows
- Prior experience working with agentic products is a plus
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.