Software Test Automation Engineering Lead (Bangalore)
On-siteBengaluru, Karnataka, India
Job Summary
Own test automation for Platform Team services including multi-tenant data isolation, IAM and OAuth workflows, event-driven backbone integrity, API gateway contract testing, billing accuracy, and agent marketplace functionality. Design and maintain Pytest-based test suites validating API contracts, event schemas, and service behavior under multi-tenant conditions while ensuring immutable audit log integrity. Define and own QA standards across Agent Stream pods, standardizing LLM and agent testing approaches, review strategies, and drive capability building through Langfuse trace validation and eval harness patterns. Integrate automated test suites into Azure DevOps CI/CD pipelines as quality gates and contribute to QA hiring by defining technical bars and ensuring consistent assessment.
Required Qualifications
- 7+ years of software testing and test automation experience
- Strong experience in SaaS platform, backend services, or enterprise application testing
- Strong experience designing, building, and maintaining automated test frameworks using Pytest, Playwright, Selenium, or similar tools
- Strong Python experience for test framework development
- Experience with REST API testing, contract testing, schema validation, and tools such as Postman, HTTPX, or similar platforms
- Experience integrating automated test suites into CI/CD pipelines
- Experience with SQL and NoSQL databases for test data management
- Experience testing multi-tenant SaaS systems
- Experience testing event-driven systems
- Experience with API gateway testing, OpenAPI schema validation, versioning, rate limiting, and inter-service contract testing
- Experience testing IAM and authentication workflows, including OAuth 2.0, token lifecycle validation, and identity propagation across services
- Experience validating audit trails, immutable logs, correlation ID chains, and event emissions for compliance-grade platform services
- Experience defining QA standards, test framework conventions, coverage expectations, or quality gate patterns across multiple teams or squads
- Demonstrated ability to raise QA capability through mentoring, framework contributions, standards documentation, or introduction of improved testing practices
- Understanding of LLM and agent testing concepts
- Familiarity with Langfuse or equivalent LLM observability tools and their use in agent validation
- Strong communication, collaboration, technical leadership, problem-solving, and continuous improvement skills
- Hands-on experience with Langfuse, including trace validation, evaluation pipeline setup, or confidence score assertion
- Experience with LLM evaluation frameworks, such as Ragas or equivalent tools for RAG output quality testing
- Experience with performance or load testing for API or LLM inference endpoints
- Experience with Azure Monitor, Application Insights, or similar tools for observability-assisted debugging
- Contributions to open-source test tooling or automation frameworks
- Prior formal or informal QA leadership experience, such as tech lead, QA chapter lead, or equivalent
Desired Qualifications
- Hands-on experience with Langfuse, including trace validation, evaluation pipeline setup, or confidence score assertion preferred
- Experience with LLM evaluation frameworks, such as Ragas or equivalent tools for RAG output quality testing preferred
- Experience with performance or load testing for API or LLM inference endpoints preferred
- Experience with Azure Monitor, Application Insights, or similar tools for observability-assisted debugging preferred
- Contributions to open-source test tooling or automation frameworks preferred
- Prior formal or informal QA leadership experience, such as tech lead, QA chapter lead, or equivalent preferred
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.