Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - Chennai
On-siteChennai, Tamil Nadu, India
Job Summary
Design and implement federated query layers using Starburst or Trino to enable high-speed analytics across distributed data sources without unnecessary data movement. Build scalable, distributed ETL/ELT pipelines with Python and Apache Spark, while managing modern table formats like Delta Lake or Iceberg to enforce ACID transactions. Optimize Spark jobs and SQL queries to minimize latency and manage compute costs, alongside enforcing fine-grained access control and data masking for privacy. This role breaks down data silos to create a unified, high-performance data layer for analytics and AI teams. Candidates must possess 5+ years of experience with Python, Spark tuning, and cloud platforms such as AWS or Azure.
Required Qualifications
- 5+ years of experience with Python
- deep expertise in Apache Spark tuning (partitioning, shuffling, caching)
- Hands-on experience with Starburst Enterprise, Trino (Presto), or Dremio
- Proven track record working with Delta Lake or Iceberg architectures
- Extensive experience with AWS (EMR, S3, Glue), Azure (Databricks, ADLS), or GCP
- Expert-level SQL skills for complex analytical queries and query plan analysis
- Proficiency in designing Star/Snowflake schemas
- understanding 'Medallion Architecture' (Bronze, Silver, Gold layers)
Desired Qualifications
- Experience with Infrastructure as Code (IaC) like Terraform or Pulumi
- Familiarity with dbt (data build tool) for modeling within the federation layer
- Knowledge of Kubernetes (K8s) for deploying and scaling Spark/Trino clusters
- Background in Data Mesh or Data Fabric methodologies
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.