Senior Software Engineer (Data Platform)

About the role

Databricks is building a Data Intelligence Platform that automates decision‑making across the company. As a Senior Software Engineer on the Data Platform team you will collaborate with product, data science, and applied AI groups to create tools for logging, orchestration, data transformation, metric storage, governance, and data consumption.

Key responsibilities

• Design and operate the Databricks metrics store that aggregates detailed metrics from all business units with high quality and query performance.
• Build and run the cross‑company Data Intelligence Platform, balancing data protection with shareability as Databricks transitions to a public company.
• Develop tooling and infrastructure to manage Databricks at scale across multiple clouds, geographies, and deployment types, including CI/CD processes and test frameworks.
• Design the base ETL framework used by all pipelines within the organization.
• Partner with engineering teams to shape the long‑term vision and requirements for Databricks products.

Required profile

• 6+ years of industry experience with technical leadership on large data projects.
• 4+ years leading initiatives such as ETL frameworks, metrics stores, infrastructure management, or data security.
• Proven experience building and operating reliable multi‑geo data pipelines at scale.
• Hands‑on experience with workflow/orchestration tools like Airflow or DBT.
• Experience with large‑scale messaging systems such as Kafka or RabbitMQ.
• Excellent cross‑functional communication and consensus‑building skills.

Required skills

• Airflow
• DBT
• Kafka
• RabbitMQ
• CI/CD pipelines
• Infrastructure‑as‑code tooling
• ETL framework design
• Metrics store implementation
• Data security and governance