Software Engineer III – Data Engineer (Azure)

About the role

As a Data Engineer you will design, build, and optimise data pipelines and platforms that enable reliable, scalable, and secure data processing across enterprise environments. You will work with modern Azure technologies and collaborate with business analysts, data scientists, and application teams to deliver high‑quality data solutions that support analytics, reporting and operational needs.

Key responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Databricks and Azure storage services.
• Ingest data from relational databases, APIs and file‑based systems into Azure data platforms.
• Implement data transformations on HPCC using ECL and on Azure using SparkSQL and PySpark in Databricks.
• Develop and optimise notebooks and jobs in Azure Databricks.
• Write efficient Spark jobs for large‑scale data processing.
• Design and implement data models on HPCC with ECL and on Azure using Delta Lake, Parquet or other columnar formats.
• Optimise ECL code, SQL and Spark workloads for performance and scalability.
• Collaborate with business analysts, data scientists and application teams to understand requirements and promote data‑engineering best practices.

Required profile

• Strong proficiency in SQL and data‑transformation logic with solid understanding of data‑warehousing concepts and ETL/ELT patterns.
• Hands‑on experience with Azure Databricks, Spark SQL, PySpark, Azure Data Factory, Azure Data Lake Storage (ADLS Gen2) and Azure Synapse.
• Experience working with large‑scale structured and semi‑structured datasets in an Agile environment.
• Knowledge of the software development lifecycle, design patterns and best practices.
• Excellent communication skills and ability to work with cross‑functional teams.

Required skills

• Python
• PySpark
• SQL
• Azure Databricks
• Azure Data Factory
• Azure Data Lake Storage (ADLS Gen2)
• Azure Synapse / Azure SQL
• HPCC and ECL
• Delta Lake
• Parquet