Data Engineer
On-siteChennai, Tamil Nadu, India
Job Summary
Design, build, and maintain efficient and scalable data pipelines to support data integration and transformation across various sources. Work with relational and NoSQL databases to manage large datasets, utilizing Apache Spark for distributed processing and real-time analytics. Implement and manage Kafka for data streaming and real-time integration between systems. Monitor and optimize pipeline performance to ensure high availability, while ensuring data quality and consistency across all architectures. Collaborate with cross-functional teams to translate business requirements into technical solutions and define automation standards for quality processes.
Required Qualifications
- 5-8 years of experience in data engineering, with a focus on building and managing data pipelines
- Strong proficiency in relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra)
- Experience in building data pipelines with data warehouses like Snowflake, Redshift
- Experience in processing unstructured data stored from S3 using Athena, Glue etc
- Hands-on experience with Kafka for real-time data streaming and messaging
- Solid understanding of ETL processes, data integration, and data pipeline optimization
- Proficiency in programming languages like Python, Java, or Scala for data processing
Desired Qualifications
- Experience with Apache Spark for big data processing and analytics is an advantage
- Familiarity with cloud platforms like AWS, GCP, or Azure for data infrastructure is a plus
- Excellent problem-solving skills and attention to detail
- Strong communication and collaboration skills, with the ability to work effectively in a team environment
- Monitor and optimize the performance of data pipelines and architectures, ensuring high availability and reliability
- Ensure data quality, consistency, and integrity across all systems
- Stay up to date with the latest trends and best practices in data engineering and big data technologies
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.