Senior/Lead Quality Engineer - Chatbot
RemoteUnited Kingdom or United States
Job Summary
Develop and evolve quality practices, test strategy, and release processes for the Chatbot product. Establish an output-evaluation framework to assess quality, reliability, and regressions in dynamic, non-deterministic LLM-based systems. Design pragmatic approaches for validating behavior beyond conventional deterministic test cases and identify, test, and improve guardrails for AI-specific risks like prompt injection. Partner with engineers through code reviews and collaborative debugging to improve quality earlier in the development lifecycle. Investigate testing gaps, research methods and tools, and design proof-of-concepts for new testing approaches or tooling. Support the evolution of release testing, including clearer quality signals and release-readiness criteria, while sharing knowledge to build the Quality Chapter across the wider engineering organization.
Required Qualifications
- Strong hands-on experience in software quality engineering, SDET, test automation, or a closely related engineering role
- Experience testing chatbot, conversational AI, RAG, LLM-powered, or other systems with dynamic and non-deterministic behaviour
- Proficiency in at least one programming language
- An experimental, investigative mindset
- Clear written and verbal communication
- Strong collaboration skills
- Good judgement in prioritising quality work based on risk, user impact, and delivery goals
Desired Qualifications
- Experience with AI security and safety testing
- Experience testing conversational AI, chatbot platforms, or agentic workflows
- Experience as the first Quality Engineer in a team
- Experience mentoring engineers or influencing testing practices across a cross-functional team
- Keeps pace with industry developments and actively fosters knowledge sharing by bringing new tech trends and methodology back to the team
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.