Sr.Data Engineer
On-siteChennai, Tamil Nadu, India
Job Summary
Develop and maintain robust ETL/ELT data pipelines using PowerCenter, Snowflake, and Databricks to process large, disconnected datasets across procurement, inventory, and shop floor systems. Design complex SQL queries, optimize storage and processing, and implement data security within scalable data warehouse and lake architectures. Build resilient data services that support root cause analysis, demand forecasting, and quality system monitoring using Python, Scala, and Kafka. Manage the full lifecycle from business requirements gathering to deployment while applying agile methodologies and collaborating with cross-functional teams.
Required Qualifications
- thorough knowledge of star/snowflake schemas
- Hands on experience in PowerCenter/IDMC/IICS
- full Life Cycle experience including business requirements gathering, data sourcing, testing/data reconciliation, and deployment within Business Intelligence/Data Warehousing Architecture
- Designing and implementing data security
- Use of Object function/object-oriented scripting languages including Scala, C++, Java and Python
- Cloud data warehouse and data lake platforms (e.g., Snowflake, Databricks, Redshift, BigQuery, MS Synapse, MS ADLS, Apache Haddop, etc.)
- Advanced SQL knowledge and experience working with relational databases and working familiarity with various databases
- Demonstrated understanding and experience using software and tools including ETL, relational SQL and NoSQL databases, big data tools like Kafka, Spark, and Hadoop
- Experience building and optimizing 'big data' data pipelines, architecture, and data sets
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
- Strong analytic skills related to working with unstructured datasets
- Experience and knowledge of project management best practices and agile software development methodologies
- A successful history of manipulating, processing, and extracting value from large, disconnected datasets
- Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores
- Strong project management and organizational skills
- Experience supporting and working with cross-functional teams in a dynamic environment
- Understanding of data science concepts
- Familiarity with AI/ML frameworks and libraries
- Procurement / Supplier data
- Inventory and warehouse management data
- Demand planning and forecasting
- Production orders, BOMs, routings
- Shop floor / MES data
- Quality systems (defects, CAPA, nonconformance)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.