Python & PySpark Developer (5-7 years) – Chennai

About the role

Capco is seeking an experienced Python/PySpark developer to design, develop, test, and maintain high‑performance data‑processing pipelines. You will transform raw market, trade, and client data into trusted data‑sets that support analytics, reporting, and machine‑learning models for the Client Engagement and Financial Securities franchise.

Key responsibilities

• Design and develop robust ingestion, transformation, and enrichment pipelines using Python, PySpark, and SQL.
• Write and optimise complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
• Collaborate with data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
• Maintain CI/CD pipelines (Git, Jenkins, Docker) and monitor production workloads, troubleshooting performance bottlenecks.
• Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint.

Required profile

• 5–7 years of experience as a Python developer with Spark in a data‑engineering environment.
• Strong understanding of Agile methodologies (Scrum/Kanban) and DevOps practices.
• Ability to mentor junior engineers and contribute to innovation initiatives.

Required skills

• Python (NumPy, pandas, RESTful APIs)
• PySpark (DataFrames, Spark SQL, Structured Streaming, performance tuning)
• SQL (complex queries, stored procedures, optimisation)
• ETL processes
• Linux/Unix command line and shell scripting
• Version control and CI/CD tools: Git, Jenkins, Docker, Maven, Bitbucket
• Agile, Scrum, DevOps
• Optional: Flask for building APIs

What we offer

• A culture focused on innovation and creating lasting value for clients and employees.
• Ongoing learning opportunities to acquire new skills or deepen existing expertise.
• A flat, non‑hierarchical structure enabling direct collaboration with senior partners and clients.