Principal Data Processing Engineer – OSS

About the role

DataPelago is building a next‑generation data processing engine that accelerates Apache Spark, Flink, Ray and other platforms. As a Principal Data Processing Engineer – OSS you will drive the integration and advancement of open‑source components such as Apache Gluten and Velox within this engine, influencing both architecture and performance.

Key responsibilities

• Define and influence the architecture that connects DataPelago’s engine with open‑source data processing platforms.
• Lead design and implementation of functional and performance enhancements for Apache Gluten, Velox and their integration.
• Independently develop, test, optimize and maintain core engine components.
• Analyze roadmaps of Apache Gluten, Velox and related projects to identify strategic opportunities.
• Collaborate with engineering, product, open‑source communities and customer‑success teams.

Required profile

• BS/MS in Computer Science or related field with 6+ years of relevant experience.
• 3+ years of deep experience optimizing data‑processing engine components on benchmark and customer workloads.
• Strong knowledge of the architecture of Apache Spark, Apache Flink, Presto or Trino.
• Proven track record delivering high‑performance data processing engines at scale.
• Excellent communication and problem‑solving abilities.

Required skills

• C
• C++
• Java
• Linux development
• Apache Spark
• Apache Flink
• Presto / Trino
• Apache Gluten
• Velox

What we offer

• Technical leadership role shaping core engine architecture.
• Opportunities to work on cutting‑edge accelerated computing and data‑processing challenges.
• Direct impact on performance and scalability of a mission‑critical platform.
• Mentorship, growth, competitive compensation, stock options and comprehensive benefits.