Senior Software Engineer - Ingestion

About the role

At Databricks we enable data teams to solve the world’s toughest problems by building the world’s best data and AI infrastructure platform. Ingesting data into the Lakehouse is a strategic investment, and Lakeflow Connect provides ready‑to‑use, point‑and‑click connectors for many sources such as enterprise applications, databases, cloud storage, message queues, and local files.

Key responsibilities

• Solve real business needs at large scale by applying software engineering.
• Deliver a highly scalable, available, and fault‑tolerant engine processing hundreds of TB of data daily across thousands of customers.
• Perform low‑level systems debugging, performance measurement and optimization on large production clusters.
• Design architecture, influence the product roadmap, and own new projects.
• Use deep experience to prevent and investigate production issues.
• Plan and lead complex technical projects that collaborate with multiple teams.
• Break down complex problems into solutions, knowns and unknowns, and de‑risk through prototyping and validation.

Required profile

• BS (or higher) in Computer Science or a related field.
• 5+ years of production‑level experience in Python, Java, Scala, C++ or a similar language.
• Experience developing large‑scale distributed systems from scratch.
• Experience with database replication, backup, or transaction recovery at major vendors (e.g., Microsoft SQL Server, Oracle, IBM) is a plus.
• Hands‑on experience developing and operating backend systems.
• Ability to contribute throughout all project phases, from design to ongoing operations, with guidance from senior team members.

Required skills

• Python
• Java
• Scala
• C++
• Database replication
• Backup
• Transaction recovery
• Incremental data capture
• Log parsing