NVIDIA is hiring a Distinguished Engineer, Scaled Out Inferencing, to lead global strategy for scaled-out AI inferencing. You will architect high-throughput, low-latency distributed pipelines and model serving for massive-scale production AI across enterprise and cloud environments. You will define the technical roadmap for full-lifecycle deployment—from versioning and automated scaling to production reliability—and drive hardware-software co-optimization across NVIDIA's AI stack, including Dynamo and TensorRT-LLM. This high-impact role also involves guiding open-source efforts, shaping industry standards, and aligning cross-functional teams to advance inferencing strategy.
Overview
$320,000 – $488,750 / year
onsite
full-time
7+ years of experience (principal level)
Technologies
- TensorRT-LLMvLLMDynamoCUDAKubernetesDockerGPU architectureCloud-native infrastructureDistributed systemsAI/ML engineering
Responsibilities
- Lead architecture and development of distributed, high-performance inferencing pipelines for large-scale AI model serving across enterprise and cloud environments.
- Define technical strategy for model deployment, versioning, and scaling to support hundreds of thousands of requests per second with low latency and high reliability.
- Drive hardware-software co-optimization to ensure NVIDIA’s AI stack (including Dynamo and TensorRT-LLM) delivers peak performance at scale.
- Guide open-source efforts and community collaboration around Dynamo, TensorRT-LLM, and related inference projects.
- Build relationships with product managers, hardware engineers, and research teams to align inferencing strategy with broader business goals.
- Evaluate emerging inferencing technologies and integrate them into NVIDIA’s offerings.
Qualifications
- MS or PhD in Computer Science, Electrical Engineering, or a related field (or equivalent experience).
- 15+ years of experience in distributed systems, cloud infrastructure, or AI/ML engineering, with a focus on large-scale model serving and inference.
- Deep expertise in AI inferencing frameworks, including TensorRT-LLM, vLLM, and other model serving technologies.
- Strong understanding of GPU architecture, CUDA programming, and performance optimization techniques.
- Proven track record of leading cross-functional teams and driving technical strategy at an organizational level.
- Experience with Kubernetes, Docker, and cloud-native technologies for deploying and scaling AI workloads.
- Excellent communication skills and ability to influence technical direction across the company.