Engineer, Data Engineering
On-siteBengaluru, Karnataka, India
Job Summary
Develop and maintain scalable ETL/ELT pipelines in GCP using Cloud Composer and PySpark on Dataproc. Optimize query performance through BigQuery partitioning, clustering, and materialized views while managing costs. Implement BigLake components for unified governance of structured and unstructured data. Deploy and manage infrastructure using Terraform and CircleCI to ensure modular, reproducible environments. Establish robust logging, alerting, and data quality checks to monitor data flow reliability. Collaborate with cross-functional teams to translate business requirements into efficient data solutions. Proactively learn new GCP services to enhance processing capabilities.
Required Qualifications
- Mid Level - 2 - 4 years of hands-on Data Engineering experience
- Solid knowledge of GCP services with a focus on BigQuery, DataForm, Cloud Composer, Cloud Storage, Cloud functions and Dataproc
- Proficiency in Dataform for managing version-controlled SQL workflows and modeling within BigQuery
- Excellent Python and SQL skills
- Solid understanding of Data Modeling (Star Schema, Snowflake) and Data Operations
- Experience with CI/CD tools and Infrastructure-as-Code is essential
- Expert-level BigQuery (Partitioning, Clustering, Slot Management)
- Implementing BigLake for unified governance
- Processing large-scale datasets using PySpark on Dataproc
- Python, SQL, and Cloud Composer (Airflow)
- Terraform (IaC), CircleCI, and GitHub
- Automated data environments
- Thinking Data mindset - where AI and advanced analytics are simply part of your standard toolkit
Desired Qualifications
- Experience in the News or Media industry is a strong plus
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.