Data Engineer
On-siteBengaluru, Karnataka, India
Bengaluru, Karnataka, IndiaOn-siteFull TimeStartup
Full TimeStartup
Job Summary
Build highly performant large-scale data pipelines capable of handling 100K+ jobs and petabyte-scale daily data volumes. Design and implement robust, cloud-native data infrastructure on AWS using Kubernetes and Airflow for efficient resource management. Develop a comprehensive SQL intelligence system encompassing query optimization, dynamic pipeline generation, and granular column-level lineage tracking. Leverage expertise in SQL AST analysis and parsing to create adaptive data pipelines and enhance query performance. Contribute to open-source initiatives within the developer community.
Required Qualifications
- 2-4 years of experience in data engineering, with a focus on building scalable data pipelines and systems
- Strong proficiency in Python and SQL
- Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake
- Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs
- Experience in building data pipelines using Airflow or dbt
Desired Qualifications
- [Optional] Solid understanding of cloud platforms, particularly AWS
- [Optional] Familiarity with Kubernetes (K8s) for containerized deployments
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.