PySpark Data Engineer – Media & Entertainment

About the role

The Developer will design, build, and optimise PySpark‑based data solutions that support analytics and content decision‑making for a global media and entertainment organisation. Working in a hybrid model, you will collaborate with cross‑functional teams to deliver reliable data pipelines that improve audience insights and enable data‑driven strategies.

Key responsibilities

• Design robust PySpark pipelines to ingest, transform, and aggregate large‑scale media datasets for analytics and reporting.
• Implement performance‑optimised PySpark code for batch and near‑real‑time processing of audience metrics and content consumption patterns.
• Collaborate with data engineers, analysts, and product teams to translate business requirements into scalable technical solutions.
• Develop reusable frameworks for processing streaming video‑on‑demand and advertising data, ensuring consistency and reliability.
• Apply data‑quality checks, validation routines, and monitoring within PySpark workflows.
• Integrate data from multiple media platforms into unified data models for holistic audience and catalog views.
• Optimise storage formats, partitioning, and execution configurations to reduce processing time and infrastructure costs.
• Document technical designs, job configurations, and data flows for maintainability.
• Ensure compliance with data‑governance, security, and privacy standards.
• Troubleshoot production issues, perform root‑cause analysis, and implement corrective actions.
• Contribute to continuous‑improvement initiatives and support testing activities.

Required profile

• Strong proficiency in PySpark programming and distributed data processing.
• Proven experience building scalable data pipelines for analytical workloads.
• Solid understanding of media and entertainment domain concepts such as audience measurement, content metadata, streaming events, advertising metrics, and catalog management.

Required skills

• PySpark
• Python
• Distributed data processing
• Data pipeline development
• Data quality validation
• Data integration
• Data modeling

What we offer

• Hybrid work environment
• Opportunity to influence data‑driven strategies for a global media brand
• Collaborative cross‑functional teams