Lead Principal Data Engineer
On-siteResearch Triangle Park, North Carolina, United States
Job Summary
Own the end-to-end architecture of Peter Millar's Microsoft Fabric environment, including Lakehouse and Warehouse design, the medallion pattern, and Direct Lake semantic datasets. Design and govern OneLake as the single, organization-wide data lake, managing domains, shortcuts, data mirroring, and Delta Lake/Parquet storage standards. Lead data ingestion and transformation at scale using Fabric Data Factory pipelines, Dataflows Gen2, and Spark notebooks while enforcing Git-based source control and CI/CD standards. Establish monitoring, alerting, and observability across pipelines to ensure SLAs for data freshness and availability. Structure governed data in OneLake to serve as grounding for generative and agentic AI use cases, integrating Azure AI services and Microsoft Foundry into production workflows. Direct technical agendas for outsourced engineering partners and serve as hiring manager for subsequent engineering hires.
Required Qualifications
- Microsoft Fabric — deep, hands-on experience with Lakehouse, Warehouse, Data Factory pipelines, Dataflows Gen2, Spark notebooks, and Direct Lake
- OneLake — demonstrated experience designing and governing OneLake, including domains, shortcuts, mirroring, and Delta Lake/Parquet standards
- Azure AI — experience integrating Azure AI services / Azure OpenAI into data and retrieval pipelines
- Microsoft Foundry (Azure AI Foundry) — working knowledge of the model catalog and RAG/agent grounding for GenAI workloads
- Azure data platform — Synapse, Azure Data Lake Storage, Azure SQL, Service Bus, and Spark in production
- Engineering practice — strong CI/CD, Git-based source control, and infrastructure-as-code
- 8+ years in data engineering / data architecture, including 2+ years in a lead or principal capacity
- Expert SQL and Python
- strong Spark (PySpark) experience
- Proven track record building and operating a Microsoft Fabric / OneLake platform in production
- Experience managing or directing outsourced/consulting engineering teams
Desired Qualifications
- Fabric-specific experience strongly preferred and commands a premium
- Exposure to Microsoft Purview, data governance, and security/compliance frameworks
- Bachelor's in Computer Science, Engineering, or related field
- Master's or relevant Microsoft certifications (e.g., Fabric Analytics Engineer, Azure Data Engineer Associate) a plus
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.