Staff AI Engineer
On-siteToronto, Ontario, Canada
Toronto, Ontario, CanadaOn-siteFull TimeSenior LevelSmall
Full TimeSenior LevelSmall
Job Summary
Build the ingestion path for agent traces at production volume and score behavior live from traces as they occur. Enforce quality and safety guardrails in the request path to stop requests before they reach users, while correlating spans across services to read one request as one trace. Own the experience of turning production traces back into datasets and test cases for the next version. Set technical direction for this field through open source contributions and conference talks. Work with RBC partners and employees across Canada and worldwide.
Required Qualifications
- 8+ years in software or platform engineering
- 5+ years in SRE, real-time data infrastructure, observability, or large-scale stream processing
- Track record running high-volume telemetry in production with hands-on work on ingestion, storage, and query at scale
- Distributed tracing and Open Telemetry: semantic conventions, collector configuration, span correlation across services
- Familiarity with routing traffic on live signal, whether that's weighted load balancing, canary rollouts, or multi-armed-bandit routing
- Turning telemetry into decisions in real time — scoring, anomaly detection, or rule/threshold evaluation on streaming data
- An LLM observability platform (Langfuse, MLFlow, or equivalent) and the trace-to-evaluation feedback loop
Desired Qualifications
- A feel for the latency and backpressure trade-offs of doing work in the live request path — collectors, proxies, sidecars
- Experience in a regulated industry (financial services, healthcare) and its constraints on AI infrastructure
- AI security controls in the request path: prompt-injection mitigation, output filtering, PII detection
- AI governance, model audit logging, and runtime drift detection
- Open-source contributions or published work in observability, tracing, or LLM evaluation
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.