Data Engineer (Databricks)
Remote · Singapore, Singapore
Job Summary
Data Engineer (Databricks) responsible for designing, developing, and maintaining scalable data pipelines on Databricks and cloud platforms. Build ETL pipelines for centralized data storage (Delta Lake), integrate data from databases, APIs, logs, and streaming sources, and transform data for analytics, reporting, and machine learning workloads. Collaborate with analytics, product, and infrastructure teams to enhance the data platform, apply governance and monitoring best practices, and support both real-time and batch processing. Develop dashboards, reports, and data visualizations; implement monitoring, error handling, and CI/CD processes; participate in requirement sessions and support UAT and production deployment. Required skills include PySpark, Spark SQL, Databricks notebooks, ADF/Airflow, SQL, Azure/AWS data experiences, streaming technologies (Kafka/Flink/Kinesis), strong problem-solving, and Scrum experience; Databricks certifications and cloud experience preferred.
Required Qualifications
- 3+ years of experience in data engineering with scalable pipelines
- Hands-on experience with PySpark, Spark SQL, and Databricks notebooks/jobs
- Experience orchestrating data pipelines with ADF, Airflow, or similar tools
- Experience with real-time and batch data processing
- Proficiency in SQL including window functions and performance optimization
- Understanding of DevOps tools, Git workflows, and CI/CD pipelines
- Familiarity with Scrum methodology and experience working in Scrum teams
- Experience with streaming technologies such as Apache Kafka, Apache Flink, or AWS Kinesis
- Ability to design and implement real-time data processing pipelines
- Databricks Certified Data Engineer Associate and Databricks Certified Data Engineer Professional are preferred
- Experience building pipelines on Azure, with AWS experience beneficial
- Knowledge of data governance and data quality practices
- Experience with monitoring, logging, and alerting for data pipelines
- Experience with Azure Databricks and Delta Lake
- Strong problem-solving and collaborative mindset
- SIT, UAT and production implementation support
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.