Custom Software Engineer
On-siteBengaluru, Karnataka, India
Job Summary
Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, Python, and PySpark. Optimize Spark jobs, cluster configurations, and data processing workflows for performance and scalability. Integrate Databricks with orchestration tools such as Apache Airflow for scheduling and monitoring data pipelines. Deploy and manage Databricks notebooks, jobs, workflows, and libraries across Development, QA, UAT, and Production environments. Troubleshoot data pipeline failures and ensure reliable data processing and delivery. Collaborate with cross-functional teams to deliver business-driven data solutions. This role requires a minimum of 8 years of experience in Data Engineering with a focus on Azure Cloud-based solutions.
Required Qualifications
- Minimum 5 year(s) of experience
- 15 years full time education
- Strong hands-on experience in Azure Databricks with Python and PySpark
- Good understanding of Apache Spark architecture, Spark configurations, performance tuning, and cluster management
- Experience with Azure Storage Accounts, Azure Data Lake Storage (ADLS Gen2), and Delta Lake
- Knowledge of authentication and authorization mechanisms, including Service Principals and Managed Identity
- Experience integrating Databricks with Apache Airflow for job scheduling and monitoring
- Understanding of Microservices architecture, REST APIs, and API integrations
- Experience deploying Databricks notebooks, workflows, and libraries across multiple environments
- Strong knowledge of Databricks Jobs, Workflows, and troubleshooting data pipeline failures
- Ability to develop scalable ETL/ELT pipelines using PySpark and Python
- Knowledge of ITSM processes and Production Support activities
- Minimum of 8 years of experience in Data Engineering
Desired Qualifications
- Experience in Azure Cloud-based Data Engineering solutions
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.