Staff Software Engineer – Frontier Security (AI)

About the role

Snowflake is building the Frontier Security AI team to deliver production‑grade LLM applications, intelligent agents, and AI infrastructure for enterprise customers. As a Staff Software Engineer you will lead the design and development of the Agentic Harness and evaluation platform, turning prototypes into reliable, secure services that operate on sensitive data at scale.

Key responsibilities

• Architect and build the Agentic Harness that executes complex, multi‑step AI workflows across models, tools, data, and services.
• Design stable interfaces for tool execution, context construction, state management, permissions, retries, fallbacks, and human review.
• Own end‑to‑end agent quality by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates.
• Convert ambiguous performance reports into measurable failure modes, reproducible tests, and durable fixes.
• Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code.
• Develop offline and online metrics for task completion, correctness, safety, latency, reliability, and cost.
• Build simulation and replay infrastructure for golden‑set tests, adversarial scenarios, model comparisons, and large‑scale experiments.
• Productionize new model capabilities as secure, observable, multi‑tenant services with clear operational controls.

Required profile

• 9+ years of software engineering experience with technical leadership of complex production systems.
• Hands‑on experience shipping and operating LLM applications, AI agents, or model‑backed workflows in production.
• Strong background in distributed systems, service architecture, high‑throughput APIs, concurrency, and failure handling.
• Experience building agent runtimes, workflow engines, developer platforms, or evaluation systems.
• Demonstrated ability to evaluate nondeterministic systems without relying on a single aggregate score.
• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience.

Required skills

• Python
• Java, Go, Rust, or TypeScript
• Distributed systems and service architecture
• High‑throughput APIs, concurrency, failure handling
• Agent runtime / workflow engine development
• Evaluation system design and implementation
• Production observability: structured tracing, metrics, logs, incident diagnosis
• Model APIs, prompt management, tool calling, retrieval, context engineering
• Kubernetes and cloud‑native infrastructure