AI Observability Engineer
On-site · Atlanta, Georgia, United States
Job Summary
The AI Observability Engineer collaborates with AI product teams to ensure AI solutions align with enterprise Responsible AI principles and standards, and supports the design and execution of the AI Governance program. Responsibilities include monitoring model performance (KPIs like context relevance, accuracy, precision, recall, F1), data quality and bias detection, using observability tools to generate insights from AI system data, managing the AI model management platform and pipelines, conducting cross-business reviews for governance alignment, contributing to training content, producing executive reports and strategic recommendations, and staying updated on global AI regulatory frameworks to translate into practical updates. This role emphasizes collaboration with legal, compliance, and security teams, and involves creating awareness materials and contributing to AI assets and methodologies.
Required Qualifications
- Minimum 3 years’ experience and a Bachelor’s degree is required
- Machine Learning, Large Language Models knowledge of various models, algorithms, and evaluation metrics
- Data pipelines and analysis skills with the ability to handle, clean, and transform data for monitoring and analysis
- Experience with Responsible AI or AI ethics
- Ability to collaborate with cross-functional teams and drive implementation of Responsible AI practices
- Nice to have experience building production LLM, RAG or model inference monitoring pipelines
- Hands-on experience with AI Observability tools (e.g., OpenTelemetry, Fiddler, Arize, LangSmith)
- Ability to build dashboards and alerts tied to AI model behavior
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.