Senior Data Engineer (Spark, Kafka, NiFi)

About the role

We are seeking a highly skilled Senior Data Engineer with deep expertise in testing, validating, developing and supporting large‑scale batch and real‑time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi. The role involves designing innovative solutions, enhancing platform capabilities, and ensuring operational excellence in production environments.

Key responsibilities

• Develop, test, validate and maintain scalable batch and real‑time data processing applications using Apache Spark, Kafka, and NiFi.
• Build validation suites and reusable frameworks to ensure fault‑tolerant, high‑performance distributed systems.
• Design and support data pipelines for ingestion, transformation, routing, and real‑time processing, including replay and recovery mechanisms.
• Analyze performance bottlenecks, optimize throughput and resource utilization, and improve scalability and reliability of the platform.
• Collaborate with product, architecture, platform and DevOps teams, participate in code reviews, and mentor junior engineers.

Required profile

• Bachelor's degree in Computer Science, Engineering or related field.
• 6+ years of hands‑on experience developing large‑scale data‑intensive applications.
• Strong expertise in Apache Spark (batch and Structured Streaming) and Apache Kafka.
• Proficiency in Scala, Python (or PySpark) and strong SQL/data‑modeling skills.

Required skills

• Apache Spark
• Apache Kafka
• Apache NiFi
• Scala
• Python / PySpark
• SQL
• Distributed storage systems (Apache Ozone, Ceph, HDFS)
• Linux, Git, shell scripting
• CI/CD pipelines
• Docker, Kubernetes