PySpark Data Engineer – Media & Entertainment

About the role

We are seeking a Developer to design, build and optimise PySpark‑based data solutions that support analytics and content decision‑making for a global media and entertainment organisation. The position works in a hybrid model, collaborating with cross‑functional teams to deliver reliable data pipelines that improve audience insights and enable data‑driven strategies.

Key responsibilities

• Design robust PySpark pipelines to ingest, transform and aggregate large‑scale media datasets for analytics and reporting.
• Implement optimized PySpark code for batch and near‑real‑time processing of audience metrics and content consumption patterns.
• Collaborate with data engineers, analysts and product teams to translate business requirements into scalable technical solutions.
• Develop reusable data frameworks for streaming video‑on‑demand and advertising data, ensuring consistency and reliability.
• Apply data quality checks, validation routines and monitoring within PySpark workflows.
• Integrate data from multiple media platforms into unified data models for holistic audience and catalog views.
• Optimize storage formats, partitioning strategies and execution configurations to reduce processing time and infrastructure costs.
• Document technical designs, job configurations and data flows for maintainability.
• Ensure compliance with data governance, security and privacy standards.
• Troubleshoot production pipeline issues and contribute to continuous improvement initiatives.

Required profile

• Strong proficiency in PySpark programming and distributed data processing.
• Proven experience building scalable data pipelines for analytical workloads.
• Solid understanding of media and entertainment domain concepts such as audience measurement, content metadata, streaming events and advertising metrics.
• Ability to work in a hybrid environment and collaborate with cross‑functional teams.

Required skills

• PySpark
• Python
• SQL
• Distributed data processing
• Data pipeline development
• Data quality validation
• Data governance and security