Sr.Data Engineering
On-siteBengaluru, Karnataka, India
Job Summary
Design, build, and optimize robust ETL/ELT data pipelines using Python, PySpark, and Azure Databricks. Develop processes for ingesting and transforming data from transactional databases, APIs, and streaming sources. Implement unit and integration tests, and establish monitoring solutions with Grafana for pipeline health and data quality. Contribute to OpenShift infrastructure setup and CI/CD automation using GitHub Actions. Develop complex SQL queries and apply data modeling principles for efficient storage in SQL Server. Identify and resolve performance bottlenecks through query optimization and indexing strategies. Collaborate with data scientists and analysts to understand requirements and document architecture. Requires 6–8 years of experience; open to sponsorship.
Required Qualifications
- 6– 8 years of relevant experience
- Strong proficiency in Python and PySpark for large-scale data processing and ETL development
- Expertise in SQL for complex querying, data manipulation, and schema design
- Demonstrable experience in designing, building, and maintaining robust ETL/ELT data pipelines
- Hands-on experience with Azure Databricks
- Proficiency with core Azure Analytics Services including Azure Data Factory, Azure SQL Server, and Azure Key Vault
- Experience implementing CI/CD pipelines from GitHub (including GitHub Actions) for automated testing (unit tests), build, and deployment processes
- Familiarity and practical experience with OpenShift (setup, deployment-ready configurations, and management)
- Experience with HELM for deploying applications on Kubernetes/OpenShift
- Experience in setting up and configuring Grafana for dashboards to monitor data quality and pipeline health
Desired Qualifications
- Proven experience in SQL optimization and performance tuning
- Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related quantitative field
- Relevant Azure certifications (e.g., Azure Data Engineer Associate)
- Experience with real-time data processing frameworks (e.g., Kafka, Azure Event Hubs)
- Understanding of data governance, data security, and compliance best practices
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.