Senior Deep Learning Sofware Infrastructure Engineer
$224,000–$356,500 year
RemoteCalifornia, United States
Job Summary
Craft, scale, and harden deep learning infrastructure libraries and frameworks for training on multi-thousand GPU clusters. Improve efficiency throughout the training stack by optimizing data loaders, distributed training, scheduling, and performance monitoring. Build robust training pipelines and libraries to handle massive video datasets and enable rapid experimentation. Collaborate with researchers, model engineers, and internal platform teams to enhance efficiency, minimize stalls, and improve training availability. Own core infrastructure components such as orchestration libraries, distributed training frameworks, and fault-resilient training systems. Partner with leadership to ensure infrastructure scales with growing GPU capacity and dataset size while maintaining developer efficiency and stability.
Required Qualifications
- BS, MS, or PhD in Computer Science, Electrical/Computer Engineering, or a related field, or equivalent experience
- 12+ years of professional experience building and scaling high-performance distributed systems
- Extensive knowledge in deep learning frameworks (PyTorch is preferred)
- Large scale training (DDP/FSDP, NCCL, tensor/pipeline parallelism)
- Performance profiling
- Strong systems background: datacenter networking (RoCE, IB)
- Parallel filesystems (Lustre)
- Storage systems
- Schedulers (Slurm, Kubernetes, etc.)
- Proficiency in Python with experience writing production-grade libraries, orchestration layers, and automation tools
- Ability to work closely with multi-functional teams (ML researchers, infra engineers, product leads) and translate requirements into robust systems
Desired Qualifications
- Shown experience scaling large GPU training clusters with >1,000 GPUs
- Expertise in fault resilience and high availability, including elastic training and large-scale observability
- Tried leadership skills as a hands-on technical authority, encouraging others and establishing guidelines for ML systems engineering
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.