Data Engineer (Python, SQL, Azure Databricks, LLMs)

About the role

Beyondsoft, a mid-sized business IT and consulting company with offices across four continents, is looking for an experienced Data Engineer based in India. The role focuses on designing, building and maintaining scalable cloud-based data solutions on Azure, with a focus on production-grade data pipelines and an understanding of Large Language Models (LLMs) and Generative AI.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks, Python and SQL.
• Build ETL/ELT workflows that ingest, transform, validate and publish data from multiple sources.
• Maintain Databricks notebooks, workflows and jobs for production data processing.
• Develop reusable Python components for ingestion, transformation, automation and integration.
• Write and optimise complex SQL queries and work with large structured and semi-structured datasets.
• Integrate LLM and Generative AI capabilities, including LLM APIs, into data engineering solutions and data preparation workflows.
• Monitor pipelines, troubleshoot quality and performance issues, and implement logging, monitoring and data quality controls.
• Take part in design, code reviews, testing, deployment and production support, following source control, CI/CD, security and documentation best practices.

Required profile

• Strong hands-on experience with Azure Databricks and Python.
• Production-grade ETL/ELT pipeline experience and familiarity with large-scale structured and semi-structured datasets.
• Solid understanding of data modelling, data warehousing and cloud data architectures.
• Good understanding of LLMs and Generative AI, including prompt engineering, embeddings, vector search and RAG.
• Problem-solving, analytical, communication and collaboration skills.

Required skills

• Python, SQL and Azure Databricks
• ETL/ELT and data pipeline development
• LLM/generative AI fundamentals and LLM API integration
• Prompt engineering and RAG fundamentals

Preferred: Azure OpenAI, RAG applications, vector databases, Azure Data Factory, ADLS, Azure SQL/Synapse, REST APIs, CI/CD, Git and Azure DevOps; Azure, Databricks or AI certification is a plus. Remote candidates must complete a video screen with the recruiting team.