Lead Data Engineer
RemoteIndia
Job Summary
Design, develop, and manage scalable ETL/ELT pipelines for structured and unstructured data using AWS Glue, Redshift, S3, Athena, Lambda, and Kinesis/Kafka. Architect and implement data lake and data warehouse solutions, optimizing performance and cost efficiency while establishing standards for data quality, governance, and security. Own end-to-end delivery of data engineering initiatives, including automation, orchestration, and observability via Apache Airflow and AWS Step Functions. Enable self-service analytics by designing semantic layers that empower users with tools like ThoughtSpot and QuickSight Q. Mentor engineers and collaborate with Product, Engineering, and Business stakeholders to translate requirements into robust data solutions.
Required Qualifications
- 5+ years of professional experience in Data Engineering
- demonstrated ownership of complex data engineering projects
- Strong proficiency in Python
- Strong proficiency in SQL
- Hands-on experience with ETL/ELT frameworks
- Hands-on experience with orchestration tools such as AWS Glue, Apache Airflow, and AWS Step Functions
- Strong experience with the AWS data stack, including S3, Glue, Redshift, Athena, Lambda, and Kinesis or Kafka
- Proven experience designing and implementing data lakes
- Proven experience designing and implementing data warehouses
- Proven experience designing and implementing scalable data pipelines
- Strong understanding of data modeling
- Strong understanding of dimensional modeling
- Strong understanding of data warehousing
- Strong understanding of database design
- Experience with query optimization
- Experience with performance tuning
- Experience with cost optimization across data platforms
- Experience with Spark/PySpark
- Experience with EMR
- Experience enabling self-service analytics through well-designed data models
- Experience enabling self-service analytics through well-designed semantic layers
- Hands-on experience with BI/analytics platforms such as ThoughtSpot, Amazon QuickSight, or similar tools
- Strong understanding of data quality
- Strong understanding of governance
- Strong understanding of security
- Strong understanding of data lifecycle management
- Excellent problem-solving skills
- Excellent communication skills
- Excellent stakeholder management skills
- Excellent technical leadership skills
Desired Qualifications
- Experience leading a technical team
- Experience mentoring data engineers
- Experience owning data engineering projects from architecture and design through implementation and production
- Experience working in a SaaS or product-based environment
- Familiarity with modern data architecture patterns, including data lakes, lakehouses, streaming pipelines, and event-driven architectures
- Experience with Kafka/Kinesis and real-time or near-real-time data pipelines
- AWS certifications in Data Engineering, Solutions Architecture, or related areas
- Contributions to open-source projects
- Demonstrated involvement in the broader data engineering community
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.