Senior/Lead Azure Databricks Engineer

About the role

We are seeking an experienced Azure Databricks Engineer to design, build, and optimise scalable data pipelines and analytics solutions on the Azure cloud platform. The role involves working with large, distributed datasets using Python, SQL and Apache Spark, and collaborating with data scientists and analysts to support advanced analytics and machine learning use cases.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks.
• Implement ETL/ELT workflows with PySpark, Spark SQL and Python.
• Optimise Spark jobs for performance, cost and scalability.
• Work with structured and semi‑structured data formats such as Parquet, Delta, JSON and CSV.
• Build and manage Delta Lake tables, including ACID transactions, time‑travel and schema evolution.
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory.
• Develop complex SQL queries and transformations, ensuring data quality and monitoring.
• Collaborate with data scientists, analysts and business stakeholders to support analytics and ML pipelines.
• Apply best practices for security, access control and governance in Azure.

Required profile

• 6+ years of experience in data engineering.
• Strong hands‑on experience with Azure Databricks and Apache Spark / PySpark.
• Proficiency in Python for data processing and advanced SQL skills.
• Experience with Delta Lake and Azure Data Lake Storage (ADLS Gen2).
• Familiarity with Git version control and CI/CD pipelines (Azure DevOps, GitHub Actions).
• Understanding of distributed computing concepts and cloud security/RBAC in Azure.

Required skills

• Azure Databricks
• Python
• SQL
• Apache Spark / PySpark
• Delta Lake
• Azure Data Lake Storage (ADLS Gen2)
• Git
• Azure Data Factory
• Azure DevOps / GitHub Actions
• Event Hub
• Kafka
• Spark Structured Streaming