AI Automation Engineer - Test Generation & Agent Systems (Sr Staff)
On-siteBengaluru, Karnataka, India
Job Summary
Design and build the Claude-powered test generation pipeline that converts product specs or Jira tickets into first-draft Playwright tests. Construct the autonomous test maintenance agent to monitor the suite, detect flakiness, diagnose root causes, and propose or apply fixes with human-in-the-loop review. Own the prompt engineering layer by maintaining the library, building an evaluation harness, and scoring generated test quality systematically. Define the acceptance workflow to determine which AI-generated tests require engineer review before merge and which qualify for automatic acceptance. Instrument the agent layer to track cost per test, acceptance rate, time-to-fix, and overall flakiness reduction. Partner closely with automation engineers to understand product domain nuance and improve generation quality for complex business logic.
Required Qualifications
- Production experience with the Anthropic Claude API or OpenAI API — specifically tool use, multi-step agent patterns, and structured output
- 8+ years of Strong TypeScript or Python — comfortable choosing the right language for the task
- Ability to evaluate LLM output quality rigorously: you build evals, not just vibes checks
- Experience designing agentic systems that handle failure gracefully: retries, fallbacks, and human escalation paths
- Comfort with ambiguity and incomplete requirements — you are building something that does not exist yet in most organizations
Desired Qualifications
- Playwright or comparable E2E test framework experience — you don't need to be an expert but you must be able to read and reason about test code fluently
- Experience with test infrastructure: fixture factories, seed scripts, environment isolation
- Prior work building agents with persistent memory, multi-tool orchestration, or complex retry logic
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.