Lead AI Ops Engineer

About the role

Mastercard's Operational Intelligence team is seeking a Lead AI Ops Engineer to own the production monitoring and reliability of AI agents. You will ensure agents remain accurate, safe, and trustworthy as they learn from real‑time outcomes.

Key responsibilities

• Own end‑to‑end production monitoring for AI agents, tracking metrics such as accuracy, latency, SLA breaches, guardrail triggers and out‑of‑scope rates via observability dashboards.
• Provide L1/L2 production support, triage incidents, and escalate to technical teams as new agentic services go live.
• Identify and diagnose issues by analyzing evaluation scores, drift alerts, failing traces and classifying root causes (intent, tool, parameter, hallucination errors).
• Partner with AI Engineering to prioritize fixes, run experiments on prompts, context and routing, and validate improvements before deployment.

Required profile

• Experience operating production AI/ML or agentic systems, including observability, evaluation and drift detection.
• Strong analytical skills to diagnose root causes across intent classification, tool selection, parameter errors and hallucination patterns.
• Excellent communication skills to translate production signals into clear, prioritized requests for AI Product and Engineering teams.

Required skills

• AI/ML monitoring and observability concepts (tracing, telemetry, guardrails).
• Evaluation engineering including benchmarks, automated evaluations, human review and regression testing.
• Governance and responsible AI practices such as PII detection, access control, audit logging and policy enforcement.