Principal Data Engineer – Cloud, AI & Data Platform

About the role

Mastercard Foundry R&D is looking for a Principal Data Engineer to design and build the data foundations that enable advanced analytics, machine learning, generative AI and agentic AI solutions. The role combines hands‑on engineering with technical leadership, turning ambiguous research goals into secure, scalable production data platforms.

Key responsibilities

• Lead architecture, design and engineering of batch, streaming and event‑driven data platforms.
• Build reliable ingestion, transformation, enrichment and consumption pipelines.
• Develop governed lakehouse and modern data platform architectures using cloud‑native technologies.
• Create reusable data products, services, APIs and self‑service capabilities for rapid experimentation.
• Enable end‑to‑end AI/ML lifecycles with feature engineering, model training, evaluation, deployment and inference pipelines.
• Define engineering standards for data modeling, testing, CI/CD, observability and operational excellence.

Required profile

• 12–18 years of experience delivering complex enterprise data engineering solutions.
• Strong hands‑on expertise with Python, PySpark, advanced SQL and modern software engineering practices.
• Proven ability to design scalable batch and real‑time pipelines on AWS or Azure.
• Experience mentoring engineers and providing technical direction in an R&D environment.

Required skills

• Python
• PySpark
• SQL (data modeling, performance tuning)
• Apache Spark
• AWS (including Step Functions, S3, Glue)
• Azure (Data Factory, Machine Learning)
• Databricks
• Apache Airflow
• MLflow
• CI/CD pipelines and Infrastructure as Code (Terraform)
• Docker, Kubernetes
• Kafka or similar event‑streaming platforms