Senior Engineering Manager, AI Quality & Governance
$210,000–$230,000 year
RemoteUnited States
Job Summary
Own the AI evaluations and observability platform, managing tracing, logs, dashboards, and operational feedback loops for prompts, model calls, and user outcomes. Design and operationalize automated and human-in-the-loop evaluation strategies, integrating regression checks for quality, safety, latency, and cost into engineering and release processes. Establish standards defining production-ready AI across the organization, including evaluation criteria, release gates, incident playbooks, and long-term quality metrics. Build and operate AI guardrails and policy enforcement capabilities, implementing content controls, PII detection, redaction, and audit logging. Translate emerging governance and risk expectations into working engineering systems and platform controls rather than static documentation. Own platform-level SLOs and tier-two incident support for AI behavior issues, partnering with product teams who remain first-line owners for features they ship. Act as an internal authority on AI quality and governance, participating in customer-facing conversations where product quality, safety, observability, or governance posture must be explained credibly. Hire, mentor, and grow the team over time as the combined function evolves.
Required Qualifications
- 8+ years of software engineering experience
- Meaningful experience leading engineers as a manager or technical lead
- Shipped production AI systems
- Hands-on experience with the operational complexity of production AI systems
- Strong experience designing and operating production backend systems and APIs
- Demonstrated hands-on experience building, shipping, or operating production AI systems
- Experience with LLM-powered or agentic workflows
- Experience with AI observability, evaluation, or debugging systems
- Practical experience designing or operating AI guardrails
- Familiarity with AI governance and risk frameworks such as NIST AI RMF or ISO/IEC 42001
- Ability to engage thoughtfully with legal, compliance, and security stakeholders
- Strong bias toward hands-on execution, platform thinking, and creating paved roads for other teams
- Ability to write and review production code
- Ability to own platform-level SLOs and tier-two incident support for AI behavior issues
- Ability to act as an internal authority on AI quality and governance
- Ability to participate in customer-facing conversations regarding product quality, safety, observability, or governance posture
- Ability to hire, mentor, and grow the team
- Ability to work 100% remote anywhere in the US
Desired Qualifications
- Experience in legal technology, regulated SaaS, or other environments where auditability and defensibility matter
- Experience with privacy-sensitive systems and PII handling
- Experience participating in AI incident reviews, red-team exercises, or internal review boards for production AI systems
- LangFuse experience
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.