Infrastructure Engineer – GPU Observability & Kubernetes

About the role

Aion, an enterprise AI platform that builds, fine-tunes and deploys AI at scale, is hiring an Infrastructure Engineer in Bengaluru. The role centres on observability and monitoring of GPU infrastructure, with deep expertise in secure multi-tenancy environments and isolation across hardware, network and storage layers for VM and container workloads.

Key responsibilities

• Build GPU observability with DCGM, NVML and custom exporters, tracking health, utilization, thermal management and performance across providers
• Deploy and operate Loki, Grafana, Tempo, Mimir, Prometheus and Thanos/VictoriaMetrics, with retention and aggregation strategies
• Write Prometheus exporters in Go or Python following OpenMetrics standards
• Develop custom Kubernetes controllers and operators for GPU workload management, scheduling and resource allocation
• Design observability for AI training and inference workloads, and integrate SLURM clusters with Kubernetes
• Write systemd services for bare-metal GPU nodes and manage infrastructure with ArgoCD and GitOps
• Build intelligent alerting, SLOs and multi-tenant metrics/log isolation with RBAC
• Create dashboards, APIs and alerting interfaces for providers and customers, and support incident response and performance tuning

Required profile

• 6-10 years of infrastructure engineering experience focused on observability, monitoring and production services
• Production experience monitoring NVIDIA GPUs with DCGM, NVML and nvidia-smi
• Experience with SLURM clusters and HPC batch workloads
• Experience with Kubernetes controllers, custom resource definitions and cluster-wide monitoring

Required skills

• Loki, Grafana, Tempo, Mimir, Prometheus, Thanos/VictoriaMetrics
• Go or Rust for production services and exporters
• Controller-runtime or client-go for Kubernetes controllers
• ArgoCD and GitOps
• OpenTelemetry integration and metrics architecture (cardinality, recording rules, retention)