IN_Senior Associate_Data Engineer Databricks_Data and Analytics_Advisory_Hyderabad
On-siteHyderabad, Telangana, India
Job Summary
Design, build, and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL) and Delta Lake for batch and incremental processing. Ingest, transform, and curate large-scale structured and semi-structured datasets while optimizing Spark jobs for performance and cost efficiency. Collaborate with analytics, AI/ML, and business teams to support downstream data consumption, ensure data quality and lineage, and troubleshoot production pipeline issues. Document technical designs, data flows, and operational runbooks while adhering to enterprise security and compliance standards. Mentor junior engineers and contribute to best practices within an Agile environment focused on transforming raw data into actionable insights.
Required Qualifications
- 4+ years of experience as a Data Engineer with strong Databricks expertise
- Hands-on experience with Apache Spark, PySpark, and Spark SQL
- Strong knowledge of Delta Lake and Lakehouse architecture
- Advanced SQL skills
- Experience with ETL/ELT patterns and data warehousing concepts
- Exposure to at least one cloud platform (Azure / AWS / GCP)
- Understanding of distributed computing concepts
- Experience working in Agile teams
- Bachelor of Technology, MBA (Master of Business Administration)
- 5 to 8 years
Desired Qualifications
- Experience with Unity Catalog and data governance
- Exposure to Auto Loader or streaming frameworks
- CI/CD for data pipelines
- Python for data engineering and automation
- Databricks certification (Associate / Professional)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.