Senior Software Engineer, Quantized Inference
$152,000–$241,500 year
On-siteSanta Clara, California, United States or Redmond, Washington, United States
Job Summary
Implement quantized and sparse recipes in inference engines including vLLM, TRT-LLM, and SGLang by writing Triton kernels and inserting quantize/dequantize nodes into prefill and decode paths. Own model export pipelines ensuring quantized checkpoints serialize correctly for downstream serving, while building prototypes and benchmarking harnesses to evaluate recipe throughput and interactivity. Develop data analysis tooling and visualizations for numerics debugging, and improve developer productivity across the team through CI, build systems, and training infrastructure enhancements. Participate in code reviews and incorporate feedback to maintain high code quality. This role is a core component of NVIDIA's productization effort across Megatron-LM, ModelOpt, and vLLM, requiring proficiency in Python and C++ with experience in ML accelerators and PyTorch internals.
Required Qualifications
- Proficient in Python
- familiarity with C++
- Strong software engineering fundamentals: concise, well-tested code; fluent with AI-assisted tooling
- Experience with ML accelerators with a basic understanding of how certain ML layers affect execution time
- Familiarity with PyTorch internals (custom ops, autograd, export) or equivalent framework
- Experience reading, modifying, or contributing to a large open-source codebase
- MS/PhD in Computer Science or related field, or equivalent experience
- 4+ years in a relevant software engineering role
- Demonstrated ability to move fast with ambiguous requirements, with strong written and verbal communication
Desired Qualifications
- Experience contributing to inference serving frameworks (vLLM, TRT-LLM, SGLang) or Triton kernel development
- Track record of debugging numerical issues across mixed-precision boundaries
- Deep experience with model compression techniques: PTQ, QAT, structured/unstructured sparsity
Hiring someone like this?
Get your role in front of qualified candidates on Sorce.