Databricks Engineer
On-siteMilpitas, California, United States
Job Summary
Design, develop, and maintain scalable data pipelines using Databricks Lakehouse Platform, building ETL/ELT workflows for batch and streaming data with PySpark, Spark SQL, and Delta Lake. Implement Medallion Architecture, integrate sources including relational databases and APIs, and optimize Spark jobs for performance and cost efficiency. Collaborate with Data Architects and stakeholders while ensuring data quality, security, and governance. Manage CI/CD pipelines and deployment automation for Databricks workloads. Monitor, troubleshoot, and optimize production data pipelines, documenting solutions and adhering to engineering best practices.
Required Qualifications
- 4–8 years
- Databricks Lakehouse Platform
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Microsoft Azure (preferred)
- AWS
- Google Cloud Platform
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- Azure Synapse Analytics
- Azure Key Vault
- Azure DevOps
- Data Engineering
- Data Warehousing
- Data Modeling
- ETL/ELT Development
- Batch Processing
- Streaming (Kafka/Event Hubs)
- Data Lake Architecture
- DevOps & Version Control
- Git
- Azure DevOps / GitHub
- CI/CD Pipelines
- Databricks
- PySpark
- Spark SQL
- Delta Lake
- Python
- SQL
- Azure/AWS/GCP (at least one cloud platform)
- ETL/ELT Development
- Data Lake Architecture
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Azure Fundamentals (AZ-900)
- Experience with real-time analytics
- Knowledge of Lakehouse architecture
- Experience with Agile/Scrum methodologies
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management abilities
Desired Qualifications
- Unity Catalog
- Databricks Workflows and Jobs
- Delta Live Tables (DLT)
- Exposure to MLflow is an added advantage
- Experience with data governance and security best practices
- Familiarity with Infrastructure as Code (Terraform) is a plus
- Azure DevOps/GitHub Actions
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.