Senior Data Engineer (Databricks & Spark)
Koantek · Hyderabad
Job description
About the role
Koantek is looking for an experienced Full Stack Data Engineer with 5-6 years of industry experience, preferably in the manufacturing or energy sectors. The role focuses on developing and maintaining scalable data solutions on Databricks and related technologies.
Key responsibilities
- Develop and deploy end-to-end data pipelines and solutions on Databricks, integrating with various data sources and systems.
- Collaborate with cross-functional teams to deliver effective BI solutions.
- Implement data ingestion, transformation and processing workflows using PySpark/Scala, SQL and Databricks notebooks.
- Build and maintain data models, ETL/ELT processes and APIs and data services supporting analytics, reporting and application integration.
- Ensure data quality, integrity and security across the data lifecycle, and monitor, troubleshoot and optimise pipeline performance in the cloud.
- Integrate data from APIs, relational and non-relational databases, IoT devices and external providers, following data governance, security and compliance policies.
Required profile
- Bachelor's or Master's degree in Computer Science, Engineering or a related field.
- 5-6 years of hands-on data engineering experience, with a strong focus on Databricks and Apache Spark.
- Experience building and maintaining ETL/ELT pipelines in production environments.
- Strong communication skills and the ability to work independently or in a team.
Required skills
- Python/PySpark and/or Scala; strong SQL skills.
- Experience with Delta Lake, Unity Catalog or similar technologies.
- Knowledge of data structures, algorithms, data modelling, ETL/ELT and data warehousing.
- Familiarity with Kafka, Spark Streaming and Databricks Genie.
- CI/CD tools such as Azure DevOps and Git.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in India.
Salaries by job title
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 1 day ago
Expires 1 month from now
4 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Koantek
Hyderabad