Senior Data Engineer (Databricks + PySpark + ADF + Unity Catalog)
On-siteSan Jose, California, United States or San José, San José, Costa Rica
EXPIREDSan Jose, California, United States or San José, San José, Costa RicaOn-siteContractSenior LevelEnterprise
ContractSenior LevelEnterprise
Job Summary
Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks to support large-scale enterprise data modernization. Build scalable ingestion frameworks for structured and unstructured data sources while implementing Unity Catalog for governance, security, and access controls. Develop transformation processes with PySpark and Spark SQL, manage Delta Lake architectures, and monitor production performance to ensure data quality and compliance. Collaborate with architects and stakeholders to implement enterprise data models and support CI/CD deployment processes.
Required Qualifications
- 4-7+ years of experience in Data Engineering or Big Data development
- Strong hands-on experience with Azure Databricks
- Proficiency in PySpark, Spark SQL, and distributed data processing
- Experience implementing and managing Unity Catalog
- Strong experience with Azure Data Factory (ADF)
- Experience working with Delta Lake and modern Lakehouse architectures
- Solid understanding of data modeling, ETL/ELT, and data warehousing concepts
- Experience with Azure cloud services and ecosystem
- Proficiency with Git, version control, and CI/CD processes
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management abilities
Desired Qualifications
- Experience with Azure Synapse Analytics
- Familiarity with Collibra or other data governance platforms
- Experience with Infrastructure as Code (Terraform, ARM Templates)
- Databricks Certification (Data Engineer Associate/Professional)
- Microsoft Azure Data Engineer Associate Certification
- Experience supporting enterprise-scale data migration projects
- Knowledge of healthcare, retail, or consumer products data environments
- Exposure to real-time data processing and streaming technologies
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.