Staff Software Engineer - Machine Learning (Search)

About the role

The Applied AI team at Databricks is advancing AI‑powered products, and the Search product is critical for helping customers find assets among millions of tables, notebooks, dashboards, and models. As a Staff Software Engineer you will lead enhancements to Search Quality, focusing on ranking, query understanding, evaluation frameworks, and scaling coverage of assets.

Key responsibilities

• Drive the development and deployment of ML‑based search and discovery relevance models and systems integrated with Databricks’ products and services.
• Design and implement automated ML and NLP pipelines for data preprocessing, query understanding and rewrite, ranking and retrieval, and model evaluation, enabling rapid experimentation and iteration.
• Collaborate with product managers and cross‑functional teams to drive technology‑first initiatives that enable novel business strategies and product roadmaps for the search and discovery experience.
• Contribute to building a robust framework for evaluating search ranking improvements – both offline and online.

Required profile

• BS+ (M.S. or PhD preferred) in Computer Science or a related field.
• 10+ years of experience developing search relevance systems at scale in production or high‑impact research environments.
• Experience applying large language models (LLM) to search relevance.
• Experience in one or more areas such as query understanding, NLP, text mining, recommendations, personalization, discovery, or conversational AI.

Required skills

• Machine Learning
• Natural Language Processing
• Large Language Models (LLM)
• Query understanding and rewriting
• Text mining
• Recommendation and personalization systems
• Discovery and conversational AI techniques