LLM Reliability & Evaluation Engineer
On-siteMohali, Punjab, India
Job Summary
Design and implement LLM evaluation pipelines covering accuracy, robustness, safety, and bias. Develop automated systems for benchmarking models on enterprise-relevant tasks and conduct stress tests, adversarial testing, and edge-case evaluations. Build tools to measure latency, consistency, and error recovery in multi-turn interactions while defining KPIs such as factual accuracy, hallucination rate, and toxicity. Establish real-time monitoring for drift, anomalies, and performance regressions. Partner with ML engineers, product managers, and domain experts to align evaluation with business objectives and implement ethical, explainable, and compliant practices. Feed insights from evaluation into fine-tuning, RLHF/RLAIF pipelines, and model selection. Maintain a central repository of test cases, benchmarks, and evaluation results. Stay current with state-of-the-art LLM evaluation techniques and explore automated evaluation using agentic test harnesses and synthetic data generation.
Required Qualifications
- 3–6 years in AI/ML, NLP, or applied model evaluation
- Strong understanding of LLM architectures, prompt engineering, and failure modes
- Hands-on with evaluation frameworks (Eval harnesses, Ragas, OpenAI Evals, DeepEval)
- Proficiency in Python and libraries like LangChain, LangGraph, LlamaIndex, Hugging Face
- Experience with vector databases, RAG pipelines, and knowledge graph integration
- Familiarity with bias/fairness testing and Responsible AI frameworks
Desired Qualifications
- Experience with reinforcement learning (RLHF, RLAIF) and reward modeling
- Exposure to agentic evaluation frameworks (multi-agent stress testing, synthetic user simulators)
- Knowledge of compliance and safety requirements for BFSI, GRC, or SOC use cases
- Contributions to open-source evaluation libraries or research papers
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.