Principal Data Engineer / Solution Architect Fabric
On-siteNew York City, New York, United States
Job Summary
Design, build, and support scalable data pipelines using Microsoft Fabric Data Factory, PySpark in Fabric, and Azure Data Factory. Drive active migration efforts from traditional Azure infrastructure to Microsoft Fabric, including pipeline transition, validation, performance tuning, and operational stabilization. Manage Databricks Unity Catalog structures, permissions, and access control policies while developing optimized Microsoft SQL assets and ADLS Gen2 storage layers. Quickly diagnose and resolve issues across data pipelines, SQL queries, and cloud integration layers. Partner with technical and business stakeholders to translate complex requirements into reliable, compliant data solutions during US Eastern business hours. Assist with knowledge transfer, document technical designs, and mentor delivery team members in a high-velocity startup environment.
Required Qualifications
- 8+ years of hands-on data engineering, data architecture, or complex data platform development experience
- Strong, practical experience building, supporting, and troubleshooting data pipelines in Microsoft Fabric and Azure Data Factory (ADF)
- Advanced proficiency in Microsoft SQL (query development, stored procedures, schema design, indexing, performance tuning) and Python / PySpark in Fabric
- Hands-on experience managing and supporting Databricks Unity Catalog (catalog/schema/table organization, access controls, permissions, and governance administration)
- Deep familiarity with Azure Data Lake Storage Gen2 (ADLS Gen2), including data organization, security, and cloud service integrations
- Proven experience supporting or leading platform migrations, modernizations, or transitions (specifically Azure-to-Microsoft Fabric)
- Familiarity with medallion architecture, lakehouse patterns, and modern data warehouse design
- Experience with Azure administration activities supporting data platforms (access management, service configuration, and monitoring)
- Prior hands-on experience working with Life Sciences, pharmaceutical, biotechnology, medical device, clinical, or regulatory data environments
- Knowledge of appropriate data handling, quality, compliance awareness, and business context in regulated settings
- Based in Boston, MA or willing/able to travel to Boston, MA for client engagements and alignment meetings
- Full working time overlap with US East Coast business hours (ET)
- Demonstrated ability to operate effectively in a startup-speed environment with shifting priorities, tight timelines, and minimal ramp-up time
Desired Qualifications
- Microsoft Certified: Azure Solutions Architect Expert
- Microsoft Certified: Fabric Data Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate
- Microsoft Certified: Azure AI Engineer Associate: nice to have
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.