Senior Data Engineer – Python, PySpark & Kafka

About the role

We are looking for a Senior Data Engineer to design, develop, and maintain high‑performance data pipelines for our global software development centre. You will work with cutting‑edge technologies to transform large‑scale datasets into actionable insights and real‑time streams.

Key responsibilities

• Develop and optimise Python scripts for data manipulation and application development.
• Build scalable PySpark jobs to process massive data volumes efficiently.
• Design and implement real‑time streaming pipelines using Apache Kafka, including Kafka Connect and Kafka Streams.
• Integrate with the Hadoop ecosystem (HDFS, YARN, Hive) for batch processing and storage.
• Write complex SQL queries against relational databases such as PostgreSQL and SQL Server.
• Collaborate with data architects to apply data‑warehousing concepts and dimensional modelling.
• Maintain code versioning using Git/GitHub and follow best practices for CI/CD.

Required profile

• B.Tech or M.Tech in Computer Science, Engineering or a related field.
• Strong hands‑on experience with Python, PySpark and Apache Kafka.
• Proven ability to work with Hadoop components and large‑scale data processing.
• Solid understanding of SQL, relational databases and data‑warehousing principles.
• Experience with version control systems, preferably Git.

Required skills

• Python
• PySpark
• Apache Kafka (Kafka Connect, Kafka Streams)
• Hadoop ecosystem (HDFS, YARN, Hive)
• SQL
• PostgreSQL
• SQL Server
• Data warehousing & dimensional modeling
• Git / GitHub