AI Quality Engineer

About the role

Testing AI systems differs from traditional software testing because outputs are non‑deterministic and "correct" is often a spectrum. As an AI Quality Engineer you will design testing strategies, build evaluation frameworks, and create automated quality infrastructure to keep intelligent agents reliable in production.

Key responsibilities

• Define testing strategies for AI agents, conversational interfaces and agentic workflows, including behavioral test suites for non‑deterministic outputs.
• Build automated evaluation pipelines that run in CI/CD, create test harnesses for LLM‑based systems, and maintain regression suites.
• Develop monitoring, alerting and reporting dashboards for production agent quality (accuracy, latency, error rates).
• Collaborate with AI Solutions Engineers and Analysts to embed testability into design and advocate for a quality‑first culture.

Required profile

• 3–7 years of software or quality engineering experience.
• Strong programming skills in Python and/or TypeScript, with experience building automated test frameworks.
• Understanding of AI/ML systems and ability to design meaningful quality metrics for non‑deterministic outputs.

Required skills

• Python
• TypeScript
• Azure (Container Apps, Functions, AI Services)
• Microsoft 365
• pytest
• DeepEval
• Ragas
• REST APIs
• Dataverse
• SQL
• Git & GitHub
• CI/CD pipelines
• Docker
• Application Insights
• Microsoft Power Platform (Power Automate, Copilot Studio)
• Load testing tools