Senior Director, Data Engineering
On-sitePleasanton, California, United States
Job Summary
Lead the multi-year data platform roadmap and architectural evolution for lakehouse, Medallion, and streaming infrastructure while deploying agentic AI to accelerate engineering velocity and reliability. Own the technology budget, team of 50+ engineers, and outcomes across data platform, analytics, and intelligence domains. Define strategy for embedding agentic AI in code generation, automated testing, and pipeline monitoring to drive measurable gains. Enforce architectural standards for trusted, governed data through data mesh adoption, contracts, and domain ownership, overseeing feature stores, vector databases, and RAG pipelines. Deliver a unified, queryable knowledge layer for retail functions with governance guardrails and adopt enterprise governance frameworks for compliance and observability. Embed DataOps best practices, including CI/CD for data and cloud cost optimization across GCP and Azure, to build a production-grade organization. Lead hiring, performance management, and succession planning for directors and senior managers while partnering with C-suite stakeholders to align data strategy with enterprise priorities.
Required Qualifications
- Bachelor's degree in computer science, engineering, or a related quantitative discipline
- 15 plus years in data engineering, data architecture, or closely related technical disciplines
- 8 plus years in building and leading data engineering teams
- 5 plus years in senior leadership with direct accountability for large-scale data engineering organizations (25+ engineers) in a complex enterprise environment
- Proven track record building and scaling production-grade data platforms at Fortune 500 scale
- Experience owning technology budgets
- Demonstrated success leading organizations through technology modernization and methodology shifts
- Expert-level command of data engineering fundamentals — ELT/ETL, data integration, cataloging, wrangling, quality, governance, and lineage
- Expert-level command of cloud data platforms
- Deep expertise in data lakehouse, Medallion architecture, and large-scale warehouse engineering
- Proficient in Airflow, Kafka/Flink, Python, SQL, and Apache Spark
- Strong grasp of data analytics, BI tooling, and self-service reporting platforms (e.g., Power BI, Looker)
- Knowledge of MLOps tooling, feature stores, vector databases, and LLM API integration for production AI/ML systems
- Proficiency in leveraging AI tools for daily engineering tasks to enhance productivity, optimize effort, and ensure cost-aware AI assistance
- Familiarity with Retrieval Augmented Generation (RAG) architectures, including semantic layers for data retrieval and grounding Large Language Model (LLM) responses
- Conceptual understanding of LLM-based applications (e.g., chatbots, Q&A systems), encompassing prompt engineering, context management, and response generation
- Exposure to agentic and multi-agent AI systems, including agent roles, tool utilization, memory management, and workflow orchestration
- Practical experience with LLM application frameworks (e.g., LangChain, LangGraph, or Google GenAI tools) for prototyping and integrating AI-driven solutions
Desired Qualifications
- Master's degree or MBA
- Experience in retail, CPG, grocery, or a similarly complex, high-velocity transactional industry
- Familiarity with retail-specific data domains: supply chain, merchandising, loyalty, store operations, and pharmacy
- Strong preference for GCP (BigQuery, Dataflow, Pub/Sub, Composer)
- Deep understanding of data mesh principles, semantic layer design, and conversational AI architecture
- Strong grasp of data analytics, BI tooling, and self-service reporting platforms (e.g., Power BI, Looker)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.