Databricks Unified Data Analytics Platform Engineer
On-siteTaguig, Metro Manila, Philippines
Job Summary
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks technologies including Delta Lake, Auto Loader, and DLT to migrate and deploy data across systems. Build secure, observable pipelines with Unity Catalog configurations, PII masking, and medallion layering while integrating Power BI/Tableau dashboards and GenAI-compatible datasets via Vector Search and MLflow. Package and deploy solutions through CI/CD pipelines in GitHub or GitLab, optimizing jobs with the Photon engine for cost efficiency and SLA reliability. Collaborate with stakeholders to implement Delta Sharing and monitor operational metrics, requiring strong Python and PySpark skills and proficiency in AWS, GCP, or Azure.
Required Qualifications
- Minimum 1 year of experience
- Excellent programming and debugging skills in Python
- Strong hands-on experience with PySpark
- Proficiency in at least one cloud platform: AWS, GCP, or Azure
- Experience with cloud-based services relevant to data engineering, data storage, data processing, data warehousing, real-time streaming, and serverless computing
- Hands on Experience in applying Performance optimization technique
- Understanding data modeling and data warehousing principles
Desired Qualifications
- Certifications: Databricks Certified Professional or similar certifications
- Knowledge of machine learning concepts and experience with popular ML libraries
- Knowledge of big data processing (e.g., Spark, Hadoop, Hive, Kafka)
- Data Orchestration: Apache Airflow
- Knowledge of CI/CD pipelines and DevOps practices in a cloud environment
- Experience with ETL tools like Informatica, Talend, Matillion, or Fivetran
- Familiarity with dbt (Data Build Tool)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.