Senior Data Engineer
On-siteBengaluru, Karnataka, India
Bengaluru, Karnataka, IndiaOn-siteFull TimeSenior LevelEnterprise
Full TimeSenior LevelEnterprise
Job Summary
Design and develop scalable cloud-based data infrastructure using SQL and PySpark, collaborating with cross-functional teams to define ingestion architecture and implement efficient pipeline frameworks. Build automated tests and integrate them into CI/CD workflows to ensure data quality and performance across data lakes and warehouses. Troubleshoot processing issues and document ETL processes while partnering with data scientists on global initiatives. This individual contributor role requires 5+ years of experience in big data technologies and is based on-site in Bengaluru with 40 weekly hours.
Required Qualifications
- Bachelor's degree in Computer Science, Data Engineering, or a related field
- Minimum 5+ years of experience in data engineering or full-stack development
- Advanced expertise in managing big data technologies (Python, SQL, PySpark, Spark)
- Strong Databricks experience
- Advanced database/backend testing with the ability to write complex SQL queries for data validation and integrity
- Strong streaming and real-time API/service validation including automation
- Experience with automated web services (WSDL) and microservices (REST) using custom scripts and assertions for data validation and data-driven testing
- Experience with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)
- Proficiency in object-oriented programming and software design patterns
- Experience working in DevOps model, including installing, configuring, and integrating automation scripts on continuous integration tools (CI/CD) and GitHub
- Experience with Unit, Functional, Integration, User Acceptance, System, and Security testing of data pipelines
- Strong experience in designing and implementing data pipelines, ETL processes, and workflow automation
- Familiarity with data warehousing concepts, dimensional modeling, data governance best practices, and cloud-based data warehousing platforms (e.g., AWS Redshift, Google BigQuery, Snowflake)
Desired Qualifications
- Master's degree
- Familiarity with cutting-edge AI technologies and demonstrated ability to rapidly learn and adapt to emerging concepts and frameworks
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.