Data Bricks DE with Azure
$120,000–$140,000 year
On-siteCalifornia, United States
Job Summary
Design, build, and optimize scalable data pipelines using Azure Databricks, developing ETL/ELT workflows for large-scale structured and unstructured datasets. Implement batch and real-time processing solutions with PySpark and Spark SQL while managing data ingestion frameworks from cloud and on-premises sources. Build and maintain data lake solutions on Azure Data Lake Storage Gen2, ensuring adherence to security, governance, and compliance standards. Optimize Spark jobs for cost, scalability, and performance, and automate deployment processes using CI/CD pipelines. Collaborate with stakeholders to support analytics and AI initiatives within a Fortune 500-facing strategic partnership environment.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field
- 6-10 years of experience in Data Engineering and Analytics
- Minimum 3+ years of hands-on experience with Azure Databricks
- Strong experience with PySpark, SQL, and cloud-native data platforms
- Experience building enterprise-scale data lakes and data warehouses
- Knowledge of performance tuning and optimization techniques in Spark
- Experience working in Agile delivery environments
Desired Qualifications
- Azure Data Engineer Associate Certification (DP-203)
- Experience with Unity Catalog and Databricks Workflows
- Knowledge of Delta Live Tables (DLT)
- Experience supporting AI/ML workloads and data science teams
- Exposure to Snowflake, Power BI, or Microsoft Fabric
- Healthcare, Life Sciences, or Pharma domain experience is a plus
- Python
- Scala (Preferred)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.