Senior/Lead Databricks Engineer (Azure)

About the role

Blend is seeking an experienced Azure Databricks Engineer with strong hands‑on expertise in Python, SQL, and Apache Spark. You will design, build, and optimise scalable data pipelines and analytics solutions on the Azure cloud platform, working with large, distributed datasets and modern data‑engineering practices.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks.
• Implement ETL/ELT workflows with PySpark, Spark SQL and Python.
• Optimise Spark jobs for performance, cost and scalability.
• Work with structured and semi‑structured data formats such as Parquet, Delta, JSON and CSV.
• Build and manage Delta Lake tables, leveraging ACID transactions, time‑travel and schema evolution.
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2) and support analytics and ML use cases.

Required profile

• 6+ years of experience in data engineering.
• Strong hands‑on experience with Azure Databricks.
• Proficiency in Python for data processing and strong knowledge of SQL (joins, window functions, performance tuning).
• Hands‑on experience with Apache Spark / PySpark and Delta Lake.
• Familiarity with Azure Data Lake Storage (ADLS Gen2) and distributed computing concepts.

Required skills

• Azure Databricks
• Python
• SQL
• Apache Spark / PySpark
• Delta Lake
• Azure Data Lake Storage (ADLS Gen2)
• Git version control
• Azure Data Factory
• CI/CD pipelines (Azure DevOps, GitHub Actions)
• Spark Structured Streaming, Azure Event Hub, Kafka