Lead Software Engineer – Backend & AI Data Platform

About the role

Mastercard is seeking a Lead Software Engineer to drive the design and evolution of large‑scale backend data systems, real‑time and batch pipelines, and AI‑enabled services. You will work on massive transactional datasets, modern big‑data platforms, and cloud environments to transform how data is processed and operationalized at global scale.

Key responsibilities

• Design, develop, and maintain backend services and data pipelines using Scala, Python and Java.
• Build and optimise batch and streaming workloads on Cloudera Data Platform (CDP) with Spark.
• Configure, monitor and tune the platform using Cloudera Manager to ensure stability and performance.
• Create high‑quality datamarts and curated datasets with strong data integrity.
• Design, build and integrate AI‑powered agents for anomaly detection, workflow automation and generative‑AI use cases.
• Develop and support data and compute workloads in AWS, leveraging Databricks for large‑scale processing.
• Contribute to cloud‑native and hybrid architectures that blend on‑prem and cloud resources.
• Enable downstream consumers such as analytics, reporting and visualization tools (e.g., Qlik) through reliable data interfaces.
• Collaborate with analytics, fraud and business teams to align backend capabilities with evolving needs.
• Produce clear technical documentation, architecture diagrams and data flow specifications.

Required profile

• Proven experience in backend engineering for data‑intensive applications.
• Strong background working with distributed data platforms such as Cloudera, Spark and CDP.
• Hands‑on expertise in building real‑time and batch data pipelines.
• Experience designing AI‑driven automation and intelligent agents.
• Familiarity with cloud environments, particularly AWS, and hybrid architecture patterns.

Required skills

• Scala
• Python
• Java
• Apache Spark
• Cloudera Data Platform (CDP) / Cloudera Manager
• AWS
• Databricks
• AI/ML agent development (anomaly detection, generative AI)
• Data modelling and datamart design