Senior Data Engineer – Azure & PySpark

About the role

PwC is looking for a Senior Associate to join its Data and Analytics advisory team in Noida. In this role you will help clients unlock enterprise insights by designing and delivering modern data pipelines and analytics solutions on the Azure cloud platform.

Key responsibilities

• Design, develop, and optimise data pipelines and ETL processes using PySpark or Scala to handle large volumes of structured and unstructured data.
• Implement data ingestion, processing, and storage solutions on Azure, leveraging Azure Databricks, Azure Data Lake Storage, and Azure Synapse Analytics.
• Develop and maintain data models, schemas, and metadata to ensure efficient data access and query performance.
• Monitor pipeline performance, troubleshoot issues, and optimise workflows for scalability, reliability, and cost‑effectiveness.
• Apply data security and compliance measures to protect sensitive information and meet regulatory requirements.

Required profile

• 4‑8 years of experience as a Data Engineer with proven expertise in building and optimising pipelines using PySpark, Scala, and Apache Spark.
• Hands‑on experience with Azure services, especially Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database.
• Strong programming skills in Python and Scala, with familiarity of version control and CI/CD practices.
• Knowledge of data‑warehousing concepts, dimensional modelling, and relational databases such as SQL Server, PostgreSQL or MySQL.
• Exposure to big‑data technologies like Hadoop, Hive or HBase is a plus.

Required skills

• PySpark
• Scala
• Python
• Apache Spark
• Azure Databricks
• Azure Data Lake Storage
• Azure Synapse Analytics
• Azure SQL Database
• CI/CD
• Version control (Git)
• SQL Server
• PostgreSQL
• MySQL
• Hadoop
• Hive
• HBase