Data Engineer 2
On-sitePune, Maharashtra, India
Job Summary
Implement and automate deployment of distributed systems for ingesting and transforming data from relational, event-based, and unstructured sources. Develop reliable, scalable data pipelines with monitoring and alert mechanisms using ETL/ELT tools or scripting languages. Monitor and troubleshoot data quality and integrity issues while implementing governance processes for metadata, access, and retention. Build physical data models and storage architectures on distributed platforms like Data Lakes, Hadoop, and cloud databases. Analyze complex data flows and dependencies to contribute to logical and physical data models. Participate in testing and troubleshooting within agile development cycles using DevOps and Scrum methodologies.
Required Qualifications
- Experience implementing and automating deployment of distributed systems for ingesting and transforming data from various sources (relational, event-based, unstructured)
- Ability to continuously monitor and troubleshoot data quality and data integrity issues
- Knowledge of data governance processes and methods for managing metadata, access, and retention for internal and external users
- Experience developing reliable, efficient, scalable, and quality data pipelines with monitoring and alert mechanisms using ETL/ELT tools or scripting languages
- Ability to develop physical data models and implement data storage architectures as per design guidelines
- Experience analyzing complex data elements, systems, data flow, dependencies, and relationships to contribute to conceptual, physical, and logical data models
- Experience developing and operating large-scale data storage and processing solutions using distributed and cloud-based platforms (e.g., Data Lakes, Hadoop, HBase, Cassandra, MongoDB, Accumulo, DynamoDB)
- Experience using agile development technologies such as DevOps, Scrum, Kanban, and continuous improvement cycles for data-driven applications
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.