AI Platform Senior Engineer, QA

About the role

Accrete is seeking an AI Platform Senior Engineer, QA to lead quality assurance for its agent‑driven platforms and AI models. The role focuses on validating non‑deterministic AI outputs, multi‑agent workflows, and cross‑product integrations using advanced automation and evaluation techniques.

Key responsibilities

• Test multi‑agent workflows, orchestration layers, and cross‑domain intelligence systems, including groundedness, coherence, hallucination, and stability testing.
• Design and maintain automation frameworks with Python, JavaScript/TypeScript, Pytest, and Playwright; build end‑to‑end UI, API, and agent pipeline tests.
• Validate in‑house AI/ML models, implement LLM‑as‑Judge, perform statistical and comparative testing, and support AWS‑deployed models.
• Conduct API testing with Postman, design load and stress tests, monitor performance under AI‑heavy workloads, and identify scalability risks.
• Collaborate with engineering, product, and data‑science teams to define test strategies, track defects, and drive continuous QA improvement.

Required profile

• 6–8+ years of experience in QA, automation, or quality engineering.
• Strong proficiency in Python, JavaScript/TypeScript, Pytest, and Playwright.
• Experience testing AI systems, chatbots, or LLM‑based platforms and understanding non‑deterministic testing.
• Hands‑on API testing (Postman) and performance testing (K6 a plus).
• Agile/Scrum experience with solid analytical, debugging, and communication skills.

Required skills

• Python
• JavaScript / TypeScript
• Pytest
• Playwright
• Postman
• AWS
• K6 (optional)
• GitHub Actions, Jenkins, GitLab
• Docker, Kubernetes
• LangChain, AutoGen, CrewAI (preferred)
• RAG systems, vector databases (preferred)
• MLOps and AI observability tools (preferred)
• Statistical validation and NLP evaluation (preferred)

What we offer

• Competitive compensation aligned with market standards.
• Mediclaim coverage of ₹10 lakh for employee and family.
• Flexible PTO and hybrid work model.
• Professional growth opportunities and learning resources.
• Catered lunches, snacks, and company‑sponsored social events.