AI Agentic Tester
RemoteDenver, Colorado, United States or Missouri, United States
Denver, Colorado, United States or Missouri, United StatesRemoteFull TimeSmall
Full TimeSmall
Job Summary
Perform functional testing of AI agent workflows and validate LLM responses for accuracy, relevance, consistency, and hallucination rates. Test RAG pipelines, prompt engineering, and AI agent orchestration while executing API tests using Postman, Swagger, and REST APIs. Develop automation scripts in Python with Selenium, Playwright, or PyTest to maintain multi-agent interactions and tool calling workflows. Apply AI evaluation metrics to ensure response quality across Azure OpenAI, AWS Bedrock, and Gemini platforms.
Required Qualifications
- AI Agentic Testing
- Functional Testing
- Generative AI Testing
- Large Language Models (LLMs)
- AI Agents & Agentic AI
- Retrieval-Augmented Generation (RAG)
- Prompt Engineering Validation
- AI Model Validation & Evaluation
- API Testing (Postman, REST APIs, Swagger)
- Python
- Test Automation (Selenium / Playwright / PyTest)
- AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
- End-to-End Workflow Testing
- Multi-Agent Testing
- Azure OpenAI / AWS Bedrock / Gemini
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.