PySpark Data Engineer – Media & Entertainment

About the role

The Developer will design, build, and optimise PySpark‑based data solutions that support analytics and content decision‑making for a global media and entertainment organisation. Working in a hybrid model, you will collaborate with cross‑functional teams to deliver reliable data pipelines that improve audience insights and enable data‑driven strategies.

Key responsibilities

• Design robust PySpark pipelines to ingest, transform, and aggregate large‑scale media datasets for analytics and reporting.
• Implement performance‑optimised PySpark code for batch and near‑real‑time processing of audience metrics and content consumption patterns.
• Collaborate with data engineers, analysts, and product teams to translate business requirements into scalable technical solutions.
• Develop reusable data frameworks that standardise processing of streaming video‑on‑demand and advertising data.
• Apply data‑quality checks, validation routines, and monitoring mechanisms within PySpark workflows.
• Integrate data from multiple media platforms and content‑management systems into unified data models.
• Optimise storage formats, partitioning strategies, and execution configurations to reduce processing time and infrastructure costs.
• Document technical designs, job configurations, and data flows for maintainability.
• Ensure compliance with data‑governance, security, and privacy standards relevant to media usage.
• Troubleshoot production pipeline issues, perform root‑cause analysis, and implement corrective actions.

Required profile

• Strong proficiency in PySpark programming and distributed data processing.
• Demonstrated experience building scalable data pipelines for analytical workloads.
• Solid understanding of media and entertainment domain concepts such as audience measurement, content metadata, streaming events, and advertising metrics.
• Ability to work effectively in a hybrid environment and collaborate with cross‑functional teams.

Required skills

• PySpark
• Python