IT Delivery Engineer – Databricks & Azure

About the role

We are looking for an experienced IT Delivery Engineer to lead the design and development of scalable data pipelines on Databricks within Azure. You will shape data lakehouse solutions, ensure high performance, and collaborate closely with data architects and business teams.

Key responsibilities

• Design and develop scalable ETL/ELT workflows and data pipelines using Databricks on Azure.
• Architect data lakehouse solutions, optimizing storage, compute, and query performance for large datasets.
• Create, optimize, and maintain Databricks notebooks, jobs, and workflows for batch and near‑real‑time processing.
• Define coding standards, best practices, and reusable frameworks for Databricks‑based engineering.
• Implement data quality checks, validation rules, and monitoring to ensure reliable data products.
• Optimize Spark jobs for performance and cost, including cluster sizing, caching, and partitioning.
• Ensure security, governance, and compliance, managing access control and data lineage.
• Troubleshoot complex data issues, perform root‑cause analysis, and drive long‑term improvements.
• Mentor junior engineers, conduct code reviews, and support knowledge sharing.
• Contribute to roadmap planning, effort estimation, and delivery management.

Required profile

• Strong analytical abilities and solid technical background.
• Good communication skills and ability to work in agile teams.
• Understanding of software quality assurance, SOLID design principles, and testing tools.
• Familiarity with project life‑cycle activities, estimation methodologies, and quality processes.

Required skills

• Databricks
• Microsoft Azure
• Apache Spark
• ETL/ELT pipeline development
• Data lakehouse architecture
• Data quality and monitoring
• Security and governance in cloud environments