Senior Data Engineer
On-sitePune, Maharashtra, India
Job Summary
Design, implement, and optimize scalable, secure, and resilient data solutions for customer engagements, integrating data from internal and external sources. Develop and maintain end-to-end data pipelines, including real-time and batch processing using Apache Spark, Flink, and Kafka, while performing feature engineering and model evaluation. Collaborate with cross-functional teams to define technology solution options, translate architecture into practical implementation plans, and share industry-leading standards to minimize risk. Identify and remediate technical debt, manage development and release processes, and mentor team members on best practices. Work without supervision to deliver high-quality analytics products that drive business insights and innovation within committed timeframes.
Required Qualifications
- Bachelors degree or above qualification in IT, Software Engineering or Computer Science
- 5+ years of data engineering experience
- Highly experienced in building enterprise data platforms and implementing industry leading data engineering standards and practices
- Extensive experience in implementing industry leading data engineering standards and practices with solid understanding of cloud data platform architecture and design
- Expert in designing, building, and maintaining robust and scalable real-time and batch data pipelines using frameworks such as Apache Spark and Apache Flink
- Strong understanding of data warehousing concepts, dimensional modelling, and data structures
- Adept in data platforms such as Hadoop, Teradata, Snowflake, and Databricks as well as NoSQL databases such as Cassandra, MongoDB, and ScyllaDB
- Experience with data profiling, cleansing, and validation techniques
- Experience with machine learning algorithms, model building, and evaluation
- Experience with leveraging foundational Gen AI models (LLMs) and their application to enable business-specific use cases
- Experience with stream processing technologies
- Expert in SQL, Spark, Scala, Java, Python, Kafka, HBase, Streaming, MLlib, Flink, Airflow
- Experience with data visualisation tools such as Tableau and PowerBI
- Cloud Environments: AWS (EMR, Redshift, Lambda, S3, Glue, DynamoDB, Athena), Azure Data Factory, GCP (BigQuery, Dataflow)
- Proficient with tooling such as Intellij IDEA, AutoSys, Git, Jenkins, GitHub, Jira, Confluence
- Experience with containerisation technologies such as Docker and Kubernetes
- Experience with automating tasks like code builds, testing, deployment, and monitoring
- Strong understanding of data management, security and privacy practices
- Highly experience in agile methodology, continuous integration, test automation, and issue tracking
- Experience working in financial services industry
Desired Qualifications
- Creative problem solver and drive continuous improvement
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.