Sr. Manager, Test Automation & AI Evaluation
$98,000–$135,000 year
On-siteNewark, New Jersey, United States
Job Summary
Lead an automation-first quality strategy that eliminates manual testing waste and establishes automated testing as the default across teams. Own modern test automation frameworks and end-to-end coverage using Playwright, while building evaluation methodologies for LLM-powered features including golden sets and LLM-as-judge approaches. Apply AI to QA itself through self-healing automation, intelligent defect triage, and AI-assisted test generation. Provide direct management and mentorship to the existing quality assurance function, driving accountability for test coverage, release readiness, and delivery quality. Partner with engineering, product, and DevOps to embed quality gates into the delivery lifecycle.
Required Qualifications
- 7+ years of experience in quality engineering or test automation, including team leadership or management
- Deep, current, hands-on expertise in modern test automation frameworks such as Playwright, Cypress, or Selenium
- Demonstrated experience building automated test suites and integrating them into CI/CD pipelines
- Experience designing evaluation or testing approaches for AI/ML or LLM-powered systems, or a strong data-and-metrics orientation with the ability to build one
- Strong understanding of test strategy, defect management, and release-readiness governance across multiple teams
- Proven ability to lead, mentor, and grow a quality engineering team
- Excellent communication skills and the ability to translate quality risk into business terms
Desired Qualifications
- Experience testing generative AI or agentic products, including evaluation harnesses, prompt testing, and regression testing for AI flows
- Experience in a healthcare, health IT, or other regulated delivery environment where quality standards are elevated
- Familiarity with quality management systems and traceability requirements
- Experience with AI-assisted testing tooling and self-healing automation platforms
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.