Data Scientist – Evaluation Frameworks (AI)

About the role

Sarvam is building a sovereign AI platform for India and needs a Data Scientist to own the evaluations function for its AI products. You will design, build, and maintain domain‑specific evaluation frameworks that ensure model quality in high‑stakes operational contexts.

Key responsibilities

• Design and implement evaluation frameworks for AI outputs such as document comprehension, command summarisation, geospatial reasoning, and enterprise workflow automation.
• Define quality metrics with domain experts and translate operational requirements into measurable signals.
• Run structured pre‑ and post‑deployment evaluation cycles, create dashboards, and monitor model performance in production.
• Identify failure modes, edge cases, and distribution shifts, focusing on uncovering weaknesses.
• Collaborate with MLOps engineers to operationalise automated, versioned evaluation pipelines.
• Build and manage domain‑specific datasets, including human annotation workflows when needed.

Required profile

• 3–6 years of experience in data science, ML research, or applied AI, with at least 2 years working on LLMs in production.
• Strong foundation in statistics and probability, understanding of valid versus misleading evaluations.
• Proven experience designing custom evaluation frameworks, including metrics, inter‑rater reliability, and red‑team methodologies.
• Ability to work with unstructured domain data such as PDFs, transcripts, and field reports.

Required skills

• Python programming
• pandas and NumPy
• HuggingFace datasets
• RAGAS, EleutherAI Eval Harness, LangSmith (or equivalents)
• Prompt engineering, model fine‑tuning, RLHF
• Statistical analysis and probability