Staff Software Engineer - Ingestion

About the role

Databricks is building Lakeflow Connect, a point‑and‑click ingestion platform that brings data from enterprise applications, databases, cloud storage and message queues into the Lakehouse. As a Staff Software Engineer you will lead the design and implementation of low‑latency, fault‑tolerant ingestion engines that serve thousands of customers worldwide.

Key responsibilities

• Design and deliver a highly scalable engine that processes hundreds of terabytes of data daily across many customers.
• Implement incremental data capture, log parsing and other techniques to extract data from OLTP systems with minimal production impact.
• Perform low‑level systems debugging, performance measurement and optimization on large production clusters.
• Define architecture, influence the product roadmap and own end‑to‑end projects.
• Lead technical reviews, mentor engineers and drive cross‑team initiatives.

Required profile

• 15+ years of experience building and supporting large‑scale distributed systems.
• Deep expertise in database replication, backup, and transaction recovery for major vendors such as Microsoft SQL Server, Oracle or IBM.
• Proven ability to work toward a multi‑year vision with incremental deliverables.
• Strong foundation in algorithms and data structures and a track record of delivering customer value.

Required skills

• Database replication, backup and transaction recovery.
• Incremental data capture and log parsing.
• Experience with Microsoft SQL Server, Oracle or IBM databases.
• Design of fault‑tolerant, high‑throughput distributed systems.
• Algorithmic problem solving and performance optimization.