IN_Senior Associate_LLM Observability Engineer_App Tech_Advisory_Pan India (early joiners)

Senior Associate role at PwC India for an LLM Observability & Performance Test Engineer. You will design and execute performance tests for AI-powered LLM applications, instrument systems to capture telemetry (prompts, responses, token usage, latency, errors) using tools like Datadog, LangChain, or OpenTelemetry, and monitor KPIs (response time, throughput, cost per query, accuracy, resource use) via real-time dashboards. You’ll identify bottlenecks in LLM pipelines, implement automated quality checks (hallucination detection, toxicity classifiers, relevance scoring), troubleshoot production issues through tracing/logging data, ensure security and regulatory compliance, optimize costs, and collaborate with data scientists, ML engineers, and QA teams on model fine-tuning and system architecture improvements. Required: strong performance testing methodology background, experience with JMeter/LoadRunner/Gatling, familiarity with LLM observability platforms, programming in Python or Java, 3+ to 7 years of experience; degrees in engineering or related fields (B.E./B.Tech./MCA/M.E./M.Tech./MBA/PGDM). Preferred: certifications in LLM Observability & Performance Test Engineering; context for AppDynamics/DataDog a plus. The role is based in Gurugram, India (PwC India), with a full-time office location.