Senior Hadoop & Spark Developer – Advisory (Bangalore)

About the role

PwC is seeking a Senior Hadoop & Spark Developer to join its Advisory practice in Bengaluru. You will work on large‑scale data engineering projects, building robust pipelines that enable clients to derive actionable insights from complex data sets.

Key responsibilities

• Design, develop, and maintain Hadoop‑based data pipelines using tools such as Hive, Sqoop, Pig, Oozie and Airflow.
• Implement high‑performance Spark solutions for batch, streaming and machine‑learning workloads.
• Create and optimise PySpark data‑transformation pipelines (JSON, CSV, RDBMS, streaming) on Kubernetes/on‑prem environments.
• Develop and manage data workflows with Apache Airflow.
• Integrate and support data sources across Oracle, MySQL, SQL Server, DB2, Teradata, PostgreSQL and Spark SQL.
• Provide application support and ongoing maintenance for Spark applications.
• Collaborate with cloud teams, preferably on GCP, to extend data solutions.

Required profile

• 6‑7+ years of hands‑on experience with Hadoop ecosystem components.
• 4+ years of professional experience with Apache Spark (batch, streaming, MLlib).
• 6‑7+ years of Python programming experience.
• 4+ years of PySpark development and deployment experience.
• 2+ years of experience designing workflows with Apache Airflow.
• Strong Unix/Shell scripting skills.

Required skills

• Hadoop (HDFS, Hive, Sqoop, Pig, MapReduce, Oozie, Airflow)
• Apache Spark (Spark Core, Spark SQL, Spark Streaming, MLlib)
• Python
• PySpark
• Kubernetes (on‑prem)
• Apache Kafka or equivalent
• SQL databases: Oracle, MySQL, SQL Server, DB2, Teradata, PostgreSQL
• Unix/Shell scripting
• Google Cloud Platform (preferred)

What we offer

• Opportunity to work on cutting‑edge data and analytics projects within a global advisory firm.
• Inclusive benefits, flexible work programmes and mentorship.
• Career growth in a collaborative, trust‑focused environment.