Lead Data Engineer
On-site · Pune, Maharashtra, India
Job Summary
Lead Data Engineer responsible for end-to-end data pipelines and Databricks platform ownership. Role includes ingestion, cleansing, transformation, and ML inference; design and operate scalable ETL/data pipelines using Databricks; manage workspace, clusters, security, governance, and best practices; contribute to full SDLC with CI/CD, testing, and production-readiness; collaborate with software engineers, data scientists, and product teams to deliver reliable data solutions; requires 6+ years in software engineering, a Bachelor's degree, hands-on Spark/Databricks with Python/Java, AWS experience, and knowledge of AI Agents; proactive, high-energy, detail-oriented, and able to work in Agile environments; knowledge of payments domain and Indian payment ecosystem is desirable; additional skills in Scala, data security/compliance, SQL/NoSQL, monitoring/observability tools, and cloud-native tooling are a plus.
Required Qualifications
- Bachelor’s degree in information technology, Computer Science or equivalent education
- Hands-on experience with Apache Spark or Databricks, preferably using Python or Java
- Proven experience building large-scale data or ETL pipelines that handle high-volume datasets
- Experience working in cloud-native environments, ideally on AWS
- Strong understanding of software craftsmanship, including: Writing clean, maintainable, and well-tested code; Designing effective testing strategies; Working with CI/CD pipelines and modern source control practices
- A strong mindset focused on reliability, scalability, and operational excellence, with ownership for production systems
- Ability to collaborate effectively within cross-functional, Agile teams
- 6+ years of experience in the software engineering field
- Must have knowledge of AI Agents and hands on using AI Agents
- Strong problem-solving skills and the ability to work independently on complex problems
- Experience working in Agile / Scrum environments
- Strong communication skills -- both verbal and written – and able to quickly learn and implement new technologies, application appropriate frameworks and tools
- Strong relationship, collaborative skills and organizational skills with a high degree of initiative and self-motivation and Able to work as a member of matrix based diverse and geographically distributed project team
- Willingness and ability to learn and take on challenging opportunities
- Knowledge of payments domain and Indian payment eco system is desirable
Apply with one swipe on Sorce. We auto-fill applications and apply on your behalf — no cover letters, no 40-minute forms.
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.