Technical Lead – Big Data Production Support

About the role

We are seeking a highly skilled Technical Lead to guide a team of 7‑8 engineers in a 24/7 production support environment. The role focuses on solving complex Big Data challenges, improving system performance, and driving automation across our data ecosystem.

Key responsibilities

• Act as the primary technical expert, resolving complex Big Data and data‑engineering issues for a team of 7‑8 members.
• Maintain stability, scalability, and performance of Spark, Hadoop, EMR, and SQL‑based pipelines in a high‑availability setting.
• Troubleshoot performance bottlenecks, conduct root‑cause analysis, and implement preventive actions.
• Identify, propose, and execute improvements in data pipelines, architecture, and cost optimization.
• Collaborate with Data Engineering, DevOps, Architecture, and other stakeholders to deliver scalable solutions.
• Own incident management, RCA, and optimization initiatives.
• Develop and review SQL queries, ETL logic, and data‑transformation code.
• Monitor and optimize workloads on AWS EMR, Redshift, and related Big Data components.
• Drive automation to reduce manual intervention and boost operational efficiency.

Required profile

• 7‑10 years of hands‑on experience with Big Data technologies.
• Proven ability to support and guide a 7‑8 member engineering team in an individual‑contributor capacity.
• Strong background in production support or DevOps‑oriented environments.
• Excellent problem‑solving skills and a proactive mindset for system improvement.

Required skills

• AWS EMR and Amazon Redshift
• Hadoop ecosystem
• Apache Spark and Spark SQL
• Hive, Presto
• Advanced SQL development
• ETL workflow design and performance tuning
• Job orchestration, scheduling, monitoring, and log analysis
• Python (preferred)
• Apache Airflow or similar orchestration tools (preferred)
• CI/CD pipelines and automation tools (preferred)
• Data modeling, partitioning, and cost‑optimization strategies