Gen AI Architect
On-siteFremont, California, United States
Job Summary
Architect and deploy Langfuse v3 in production environments, including upgrades from v2 with backward compatibility. Design modular components for prompt management, tracing, metrics, evaluation, and playground features using Langfuse capabilities. Integrate Azure AI Evaluation SDK into enterprise pipelines for scalable evaluation of GenAI models. Build reusable components and templates for evaluation across diverse RAG, multi-agent, and GenAI lifecycle workflows. Ensure scalability, observability, and traceability of evaluation frameworks in both offline and online environments. Provide technical leadership to development teams, ensuring best practices, coding standards, and performance optimization in complex GenAI projects. Collaborate with cross-functional teams to align evaluation metrics with business KPIs. Conduct feasibility studies, recommend technical alternatives, and ensure compliance with architecture best practices. Stay current with emerging GenAI evaluation tools and methodologies.
Required Qualifications
- Hands-on expertise in Langfuse (including v3 features) and integrations
- Proficiency with Azure AI Services and Azure AI Evaluation SDK
- Strong understanding of LLMOps, prompt engineering, and GenAI lifecycle management
- Experience with Retrieval-Augmented Generation (RAG), fine-tuning, and multi-agent orchestration
- Proficiency in Python
- Experience with cloud-native architectures (Azure preferred)
- Familiarity with observability tools, tracing frameworks, and evaluation metrics for GenAI systems
- Excellent communication, documentation, and stakeholder collaboration skills
Desired Qualifications
- TypeScript or other GenAI-related languages
- Experience with additional observability tools such as TruLens, Weights & Biases (W&B), Helicone
- Exposure to evaluation standards for enterprise AI adoption
- Knowledge of multi-cloud and hybrid deployment strategies
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.