Senior Data Engineer – End‑to‑End Pipeline Delivery

About the role

Mastercard is looking for a Senior Data Engineer to join its Hyderabad team. This individual‑contributor position owns the full lifecycle of complex batch and real‑time data pipelines while mentoring peers.

Key responsibilities

• Design, develop and deliver end‑to‑end batch and streaming pipelines using Databricks, Spark, Python and PySpark.
• Drive dbt‑based transformation architecture with test‑driven development and modular design.
• Create and maintain CI/CD pipelines with GitLab and Jenkins, championing DataOps practices.
• Administer Databricks assets, optimize performance, manage access and control costs.
• Establish data observability frameworks covering quality, freshness, lineage and anomaly detection.
• Lead the design of real‑time streaming solutions using Kafka and Spark Structured Streaming.
• Mentor junior and mid‑level engineers, conduct code and design reviews.
• Collaborate with stakeholders on requirements, delivery planning and technical trade‑offs.

Required profile

• Bachelor's degree in Computer Science or related field (Master's a plus).
• 5+ years of data engineering experience with strong hands‑on expertise in Databricks, Spark and Python.
• Proven experience using dbt for testable transformation workflows.
• Experience with CI/CD tools (GitLab, Jenkins) and DataOps practices.
• Background in real‑time data processing and streaming pipelines.
• Strong SQL development skills and cloud infrastructure experience (AWS, Azure or GCP).
• Demonstrated ability to mentor engineers and lead technical delivery.

Required skills

• Databricks administration
• Spark (batch & streaming)
• Python & PySpark
• dbt and Test‑Driven Development
• GitLab, Jenkins (CI/CD)
• Kafka, Spark Structured Streaming
• SQL
• Cloud platforms: AWS, Azure, GCP