Data Engineering and Cloud
On-siteBengaluru, Karnataka, India
Job Summary
Design, develop, and maintain scalable data pipelines and infrastructure to support autonomous driving research and development. Extract, transform, and load data from sensors, simulations, and databases while developing storage strategies that ensure data quality and integrity. Collaborate with data scientists and engineers to translate technical requirements into solutions, implement security measures, and monitor systems to resolve issues promptly. Continuously evaluate new technologies to improve processing efficiency and mentor junior engineers. Requires strong Python skills, experience with Apache Spark, Kafka, and Azure, and 6+ years of experience.
Required Qualifications
- Strong proficiency in Python or another programming language commonly used in data engineering
- Expertise in data engineering tools and frameworks, such as Apache Spark, Hadoop, Kafka, and Airflow
- Deep understanding of data warehousing and data lake concepts
- Experience with cloud platform (Azure) and cloud-based data services
- Knowledge of SQL and NoSQL databases
- Excellent problem-solving and analytical skills
- Ability to work independently and as part of a cross-functional team
- Strong communication and documentation skills
- Ph.D. or Master's degree in Computer Science, Data Engineering, or a related field
- BE M Tech CS
- 6+ years
Desired Qualifications
- Experience with machine learning and deep learning frameworks (TensorFlow, PyTorch)
- Familiarity with autonomous driving technologies and concepts
- Knowledge of data visualization tools (e.g., Tableau, Matplotlib)
- Experience with real-time data processing and streaming technologies
- Understanding of data quality and validation techniques
- Experience with containerization technologies (Docker, Kubernetes)
- Familiarity with CI/CD pipelines and DevOps practices
- Experience with data governance and compliance frameworks (e.g., GDPR, CCPA)
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.