Technical Data Engineer – Databricks

About the role

We are seeking a Technical Data Engineer with hands‑on experience building scalable data pipelines on the Databricks Lakehouse Platform. The role involves designing, developing, and maintaining ETL/ELT solutions that integrate data from diverse sources and support our Fortune 500 clients.

Key responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Integrate data from databases, Amazon S3, files, and REST APIs.
• Build data pipelines leveraging Databricks Unity Catalog.
• Implement business logic, data transformations, and dimensional models.
• Create, schedule, monitor, and optimise Databricks Jobs and Workflows.
• Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, Gold).
• Ensure data quality through validation, error handling, logging, and monitoring.
• Optimise Spark workloads for performance, scalability, and reliability.
• Collaborate with cross‑functional teams to deliver production‑ready data solutions.

Required profile

• Strong expertise in Python, PySpark, and advanced SQL.
• Hands‑on experience with the Databricks Lakehouse Platform, Unity Catalog, Delta Lake, and Medallion Architecture.
• Experience integrating with REST APIs for data ingestion and export.
• Solid knowledge of ETL/ELT development, batch processing, incremental loading, and data transformation.
• Familiarity with data modelling concepts such as star and snowflake schemas, SCD, and data‑warehousing best practices.
• Experience working with structured and semi‑structured data formats (CSV, JSON, Parquet, Delta).
• Understanding of partitioning, file optimisation, Spark performance tuning, and query optimisation.
• Proficiency with Git and CI/CD best practices.

Required skills

• Python
• PySpark
• SQL
• Databricks Lakehouse Platform
• Unity Catalog
• Delta Lake
• Databricks Workflows / Jobs
• Databricks Clusters, Notebooks, Repos
• Medallion Architecture
• REST API integration
• Amazon S3
• AWS
• Git
• CI/CD

What we offer

• Competitive salary
• Strong insurance package
• Extensive learning and development resources