Senior Data Engineer
$80,000–$100,000 year
On-site · Chennai, Tamil Nadu, India
Job Summary
Develop and optimize data pipelines; build, test, and maintain ETL/ELT data pipelines using Azure Databricks and Apache Spark (PySpark). Ensure data quality and governance through Unity Catalog, and integrate with Azure Data Lake Storage and Azure Blob Storage. Collaborate with stakeholders to translate business needs into data engineering solutions. Key skills required include strong proficiency in Azure Databricks, Python, Unity Catalog, SQL, Delta Lake, and familiarity with CI/CD and infrastructure as code.
Required Qualifications
- Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines
- Python – Proficiency in writing optimized and maintainable Python code for data engineering
- Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking
- SQL – Strong knowledge of SQL for data transformations and optimizations
- Delta Lake – Understanding of time travel, schema evolution, and performance tuning
- Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory
- CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles
- Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption
Desired Qualifications
- Experience with MLflow for model tracking & deployment in Databricks
- Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid)
- Hands-on experience with dbt (Data Build Tool) for modular ETL development
- Certification in Databricks, Azure is a plus
- Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
- Experience with Azure Synapse Link for Dynamics, dataverse
- Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc
Additional Requirements
- Equal opportunity employer with encouragement to apply for women, minorities, individuals with disabilities, and veterans
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.