Lead Data Engineer

About the role

We are seeking a self‑driven Lead Data Engineer with 7–10 years of experience to build and optimise scalable ETL/ELT pipelines using Azure Databricks, PySpark and Delta Lake. The role works across scrum teams, ensures data governance with Unity Catalog and supports both real‑time and batch processing in the insurance domain.

Key responsibilities

• Develop and maintain scalable ETL/ELT pipelines on Azure Databricks using PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution for real‑time and batch processing.
• Implement data governance, access control and lineage tracking using Unity Catalog.
• Collaborate with scrum teams, product owners and business stakeholders to deliver data solutions.
• Analyse, troubleshoot and resolve data issues end‑to‑end in real time.
• Write unit tests and apply SOLID development principles to ensure data integrity.

Required profile

• 7–10 years of experience as a Lead Data Engineer.
• Strong expertise in T‑SQL, Azure Databricks, Spark (PySpark/Scala), Delta Lake and Unity Catalog.
• Hands‑on experience with Azure cloud services such as ADLS Gen2, Azure Data Factory and Azure SQL.
• Self‑driven with minimal supervision and strong problem‑solving mindset.

Required skills

• Azure Databricks
• PySpark / Spark (Scala)
• Delta Lake
• Unity Catalog
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure SQL
• T‑SQL
• Microsoft TFS, Visual Studio, Azure DevOps