Data Engineer
On-siteBengaluru, Karnataka, India
Job Summary
Design, develop, and maintain robust data pipelines using Azure Data Factory (ADF) to ingest, transform, and load data into data lakes or warehouses. Create, schedule, and monitor complex workflows with dependency management, while integrating ADF with Blob Storage, Synapse, and Databricks. Quickly diagnose and resolve pipeline failures, performance bottlenecks, and data quality issues through root cause analysis and optimization. Collaborate with analysts and architects to translate business needs into technical implementations, document logic, and ensure data accuracy. Optimize execution time and cost, maintaining metadata repositories and dictionaries.
Required Qualifications
- 3-5 years of experience
- Azure Data Factory (ADF)
- ETL/ELT processes
- data lakes
- warehouses
- databases
- batch data flows
- near real-time data flows
- dependency management
- parameterization
- dynamic pipeline design
- Azure Blob Storage
- Azure Synapse
- Azure Databricks
- Azure Key Vault
- logging
- alerting
- retry logic
- SQL queries
- data quality issues
- pipeline failures
- performance bottlenecks
- root cause analysis
- Spark transformations
- data validation
- quality checks
- reconciliation
- sanity checks
- data dictionaries
- metadata repositories
- data infrastructure
- tooling
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.