AI QA & Evaluation Lead
On-site · Riyadh, Riyadh Region, Saudi Arabia
Job Summary
Lead enterprise-grade QA, validation, and evaluation for AI-powered systems and intelligent automation platforms. Design and maintain AI testing and evaluation frameworks; develop benchmarking datasets and output validation methodologies. Evaluate AI-generated outputs for consistency, accuracy, and reliability; define confidence scoring and quality measurement approaches. Implement automated regression testing for AI workflows; monitor prompt performance and behavioural consistency; detect model drift and output degradation. Validate multilingual AI outputs including Arabic and English content. Ensure AI outputs align with enterprise governance standards; support auditability and traceability of AI-generated results; implement review and approval workflows for AI systems; support compliance validation activities for enterprise deployments. Conduct AI security and prompt injection testing; validate protections against unsafe outputs and misuse scenarios; support secure AI deployment and operational practices. Produce quality assurance reports and evaluation summaries; recommend improvements to prompts, workflows, and AI models; support continuous optimisation of AI solution quality.
Required Qualifications
- Bachelor's degree in Computer Science, Information Systems, Software Engineering, or related field
- Minimum 5 years experience in enterprise quality assurance or software testing
- Experience with automation testing frameworks
- Experience evaluating AI-powered systems preferred
- Experience in enterprise or regulated environments preferred
- Quality Assurance methodologies
- Automation testing
- API testing
- Regression testing
- Test case development
- AI evaluation including large language model evaluation
- Prompt engineering concepts
- AI benchmarking methodologies
- AI governance and validation concepts
- AI security testing
- Auditability and traceability principles
- Risk assessment concepts
- Security testing
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.