Azure Data Engineer – Databricks & Data Lake Specialist

About the role

We are looking for an experienced data engineer to design, develop and optimise data pipelines on Microsoft Azure. The role focuses on building scalable solutions using Azure Databricks, Azure Data Factory and Azure Data Lake Storage to support analytics and business intelligence teams.

Key responsibilities

• Design, develop and optimise end‑to‑end data pipelines with PySpark/Spark SQL on Azure Databricks.
• Build and manage data ingestion, transformation and processing workflows using Azure Data Factory and related services.
• Implement ETL/ELT processes and ensure data quality, reliability and monitoring.
• Develop and maintain data lake / lakehouse architectures on ADLS Gen2, including schema management and partitioning.
• Optimise Spark jobs for performance, cost and scalability, and apply best practices for data modelling.
• Collaborate with analytics, BI and business stakeholders to deliver data solutions.
• Support CI/CD pipelines, version control (Git) and automated deployments.
• Participate in Agile/Scrum ceremonies and contribute to continuous improvement.

Required profile

• 5–9 years of overall experience in data engineering or big data development.
• 3+ years of hands‑on experience with Azure Databricks.
• Strong problem‑solving abilities and effective communication skills.

Required skills

• Azure Databricks (PySpark, Spark SQL)
• Azure Data Factory
• Azure Data Lake Storage (ADLS Gen2)
• Azure SQL / Synapse (optional)
• SQL for data transformation and analytics
• Git and CI/CD pipelines
• Delta Lake
• Structured Streaming, Event Hubs
• Power BI or Tableau (exposure)
• Python (basic best practices)
• Agile/Scrum methodology