This job is no longer available
This job expired on 16/10/2026. It no longer accepts applications.
Technical Data Engineer – Databricks
ShyftLabs · Coimbatore
Job description
About the role
We are seeking a Technical Data Engineer with hands‑on experience building scalable data pipelines on the Databricks Lakehouse Platform. The role involves designing, developing, and maintaining ETL/ELT solutions that integrate data from diverse sources and support our Fortune 500 clients.
Key responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
- Integrate data from databases, Amazon S3, files, and REST APIs.
- Build data pipelines leveraging Databricks Unity Catalog.
- Implement business logic, data transformations, and dimensional models.
- Create, schedule, monitor, and optimise Databricks Jobs and Workflows.
- Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, Gold).
- Ensure data quality through validation, error handling, logging, and monitoring.
- Optimise Spark workloads for performance, scalability, and reliability.
- Collaborate with cross‑functional teams to deliver production‑ready data solutions.
Required profile
- Strong expertise in Python, PySpark, and advanced SQL.
- Hands‑on experience with the Databricks Lakehouse Platform, Unity Catalog, Delta Lake, and Medallion Architecture.
- Experience integrating with REST APIs for data ingestion and export.
- Solid knowledge of ETL/ELT development, batch processing, incremental loading, and data transformation.
- Familiarity with data modelling concepts such as star and snowflake schemas, SCD, and data‑warehousing best practices.
- Experience working with structured and semi‑structured data formats (CSV, JSON, Parquet, Delta).
- Understanding of partitioning, file optimisation, Spark performance tuning, and query optimisation.
- Proficiency with Git and CI/CD best practices.
Required skills
- Python
- PySpark
- SQL
- Databricks Lakehouse Platform
- Unity Catalog
- Delta Lake
- Databricks Workflows / Jobs
- Databricks Clusters, Notebooks, Repos
- Medallion Architecture
- REST API integration
- Amazon S3
- AWS
- Git
- CI/CD
What we offer
- Competitive salary
- Strong insurance package
- Extensive learning and development resources
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in India.
Salaries by job title
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
ShyftLabs
Coimbatore
Related job offers
-
Senior Deployment Manager
Eurofins Coimbatore -
Principal Software Engineer - Java (eLIMS-Forensics)
Eurofins Coimbatore -
Senior Software Engineer (Java, Spring Boot, Azure)
Eurofins Coimbatore -
Développeur Front‑End UI/UX Scada (ReactJS) – GE Vernova
GE Vernova Paris -
Software Engineer - Backend Distributed Systems
salesforce