Lead Data Engineer – Physical AI Platform, Data Engineering
$128,470–$208,770 year
On-siteChicago, Illinois, United States
Job Summary
Design and maintain scalable data pipelines, microservices, and cloud-based platforms using Python, AWS services (Kinesis, S3, DynamoDB, EventBridge), and SQL to support real-time and batch processing. Lead solution architecture and optimization for CI Autonomy initiatives, collaborating with Principal Engineers to define data workflows and enforce automated testing and data quality controls. Drive operational excellence through performance tuning, root-cause analysis, and monitoring using CloudWatch to ensure high availability across distributed data ecosystems. Work onsite five days a week with relocation available.
Required Qualifications
- Bachelor's degree in Computer Science, Computer Engineering, or related field
- 8+ years of experience in data engineering or related disciplines with increasing responsibility
- Extensive experience on modern, large scale, complex Caterpillar data platforms such as Helios Data Platform
- Strong foundation developing and deploying Python solutions to a production environment
- Experience leading teams to build high-throughput, scalable data pipelines
- Strong hands-on experience with AWS data services (Kinesis, S3, DynamoDB, EventBridge, etc.) at scale
- Strong in SQL, including data quality and validation practices
- Experience in deploying software using CI/CD tools such as Azure DevOps, Jira, Jenkins, etc.
- Experience developing microservices that support real-time data ingestion
- Experience developing software applications using relational and noSQL databases
- Ability to ensure data integrity across distributed and streaming systems
- Experience with monitoring, testing, and automation in large-scale data environments
- This position requires working onsite five days a week
- Relocation is available for this position
- Visa sponsorship is available for eligible applicants
- Offer of employment is conditioned upon the successful completion of a drug screen
Desired Qualifications
- Top Candidates Will Have: Bachelor's degree in Computer Science, Computer Engineering, or related field
- Top Candidates Will Have: 8+ years of experience in data engineering or related disciplines with increasing responsibility
- Top Candidates Will Have: Extensive experience on modern, large scale, complex Caterpillar data platforms such as Helios Data Platform
- Top Candidates Will Have: Strong foundation developing and deploying Python solutions to a production environment
- Top Candidates Will Have: Experience leading teams to build high-throughput, scalable data pipelines
- Top Candidates Will Have: Strong hands-on experience with AWS data services (Kinesis, S3, DynamoDB, EventBridge, etc.) at scale
- Top Candidates Will Have: Strong in SQL, including data quality and validation practices
- Top Candidates Will Have: Experience in deploying software using CI/CD tools such as Azure DevOps, Jira, Jenkins, etc.
- Top Candidates Will Have: Experience developing microservices that support real-time data ingestion
- Top Candidates Will Have: Experience developing software applications using relational and noSQL databases
- Top Candidates Will Have: Ability to ensure data integrity across distributed and streaming systems
- Top Candidates Will Have: Experience with monitoring, testing, and automation in large-scale data environments
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.