Sr. Data Engineer - PySpark, Python & Cloudera (CDP)
On-siteDubai, Dubai, United Arab Emirates
Job Summary
Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python within the Cloudera Data Platform. Build robust batch and distributed data processing solutions while optimizing ETL/ELT frameworks for structured and unstructured enterprise datasets. Ensure data quality, integrity, and governance through profiling, cleansing, and validation activities. Monitor and troubleshoot production pipeline issues, collaborate with Data Scientists for analytics preparation, and participate in Agile ceremonies including sprint planning and retrospectives. Manage multiple priorities in a fast-paced environment to deliver secure, high-performance data solutions powering business intelligence and machine learning initiatives.
Required Qualifications
- 8–10+ years of experience in Data Engineering
- Strong hands-on expertise in Python
- Strong hands-on expertise in PySpark
- Extensive experience with Cloudera Data Platform (CDP)
- Strong understanding of distributed data processing frameworks
- Experience building enterprise-scale ETL/ELT pipelines
- Strong knowledge of Big Data technologies
- Experience with Hadoop ecosystem technologies
- Strong SQL programming and query optimization skills
- Experience with data modelling and data transformation techniques
- Knowledge of data quality, validation, and governance principles
- Experience working with Git and version control systems
- Understanding of CI/CD practices for data engineering
- Strong understanding of REST APIs and data integration patterns
- Experience working with structured, semi-structured, and unstructured datasets
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management skills
- Ability to work in fast-paced Agile delivery environments
- Strong ownership mindset with focus on quality and delivery
- Ability to collaborate effectively with business and technical stakeholders
- Strong debugging and performance optimization capabilities
- Ability to manage multiple priorities and deliver within tight timelines
Desired Qualifications
- Experience working with cloud-native data platforms
- Exposure to machine learning data pipelines
- Knowledge of feature engineering and data preparation for AI/ML workloads
- Experience with workflow orchestration tools
- Exposure to containerization technologies such as Docker and Kubernetes
- Experience with DevOps practices for Data Engineering
- Banking
- Financial Services
- Digital Products
- Transaction Banking
- Enterprise Data Platforms
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.