Infrastructure SRE – High Performance Computing

About the role

Sarvam operates a large, multi‑vendor GPU fleet that runs both massive training jobs and latency‑critical inference services. The role focuses on ensuring the health and reliability of this infrastructure, tackling challenges that span storage, networking, GPU systems, Kubernetes, and workload reliability.

Key responsibilities

• Operate the GPU fleet end‑to‑end across training and serving, handling provisioning, observability, capacity planning and fleet health.
• Participate in an on‑call rotation, author robust runbooks and drive post‑mortems that lead to lasting fixes.
• Develop internal tooling to support the team rather than relying solely on off‑the‑shelf solutions.
• Collaborate with ML and platform teams to keep large training runs alive and ensure serving latency remains predictable.

Required profile

• 5+ years of infrastructure or site reliability engineering experience, including at least 2 years operating GPU clusters at scale.
• Proven on‑call ownership with a track record of effective post‑mortems.
• Proficiency in Python or Go for building internal tools.
• Working fluency across the five focus areas (storage, fabric, GPU systems, Kubernetes platform, workload reliability).

Required skills

• Experience with parallel filesystems such as Lustre, GPFS, WEKA, or BeeGFS.
• Knowledge of InfiniBand or RoCE networking, NVLink/NVSwitch topology, and RDMA debugging.
• Expertise in GPU systems reliability, including NCCL debugging, driver/firmware lifecycle, and DCGM‑based health monitoring.
• Strong understanding of Kubernetes platform reliability, GPU operator stack, and scheduling (Slurm‑on‑K8s or native Kubernetes).
• Ability to detect and troubleshoot training and inference workload issues, including checkpoint/restart and high‑availability concerns.