Data Engineer
On-siteChennai, Tamil Nadu, India
Job Summary
Analyze legacy data warehouse, SQL Server, Oracle, ETL, and T-SQL implementations to migrate them to Databricks. Interpret complex T-SQL scripts, stored procedures, and transformation rules, then convert business logic into optimized Databricks and PySpark workflows. Design, develop, and maintain scalable ETL/ELT pipelines while supporting data ingestion, validation, and reconciliation across platforms. Collaborate with architects and migration teams to map source-to-target requirements, perform data quality checks, and optimize PySpark jobs for performance. Document migration logic, transformation rules, and technical designs, participating in code reviews and adhering to Data Engineering best practices. Requires 10+ years of experience predominantly in Data Engineering roles with strong hands-on expertise in Databricks, PySpark, and SQL/T-SQL.
Required Qualifications
- 10+ years of experience
- hands-on expertise in Databricks
- PySpark
- SQL/T-SQL interpretation
- worked predominantly as a Data Engineer throughout their career
- ability to understand existing T-SQL logic
- migrate transformation rules into scalable Databricks and PySpark solutions
- 5+ years of relevant Data Engineering experience
- majority of career spent in Data Engineering roles
- Strong hands-on experience in Databricks development
- Strong PySpark programming experience for data transformation and pipeline development
- Strong SQL/T-SQL understanding, especially the ability to read and interpret existing T-SQL code
- Experience in legacy data technologies such as SQL Server, Oracle, traditional ETL tools, or Data Warehouse platforms
- Experience in ETL/ELT development, data ingestion, transformation, and data pipeline implementation
- Good understanding of Data Warehousing, Data Modeling, and dimensional concepts
- Ability to migrate or re-engineer legacy SQL/ETL logic into Databricks/PySpark
- Strong analytical, debugging, and problem-solving skills
- Good communication skills to work with technical and business teams
Desired Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline
- Azure Data Factory or Azure Data Platform exposure
- Delta Lake experience
- Unity Catalog exposure
- Azure Synapse Analytics experience
- CI/CD exposure for data pipelines
- Experience working in Agile/Scrum delivery models
- Data governance, security, and lineage awareness
- Relevant certifications in Databricks, Azure Data Engineering, or Cloud Data Platforms
- Strong, hands-on Data Engineering background throughout career, not a partial or recent transition
- Profiles that are primarily BI, reporting, dashboarding, analytics, or only recently shifted into data engineering should not be prioritized
- Avoid profiles that are mainly BI Developers, Power BI Developers, Data Analysts, or reporting-only candidates
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.