Research Engineer, Pretraining Scaling

Research Engineer on the ML Performance and Scaling team responsible for production pretraining pipelines. Own core aspects of the production pretraining process including model operations, performance optimization, observability, reliability, and incident response during model launches. Debug across the full stack—from hardware and networking to training dynamics and evaluation infrastructure. Design and run experiments to improve training efficiency, reduce step time, increase uptime, and enhance model performance. Build and maintain production logging, monitoring dashboards, and evaluation infrastructure; add capabilities to the training codebase (e.g., long context support, novel architectures). Collaborate with teams in SF and London and contribute to institutional knowledge by documenting systems and approaches. Candidates should have hands-on experience training large language models or deep expertise with JAX, TPU, PyTorch, or large-scale distributed systems; enjoy both research and engineering work; be comfortable on-call during launches; thrive in high-impact, variable work prioritization; and communicate effectively across time zones. This role is located in-office in San Francisco with a compensation range of $350,000–$850,000 USD annually.