Data Engineer – Batch & Streaming Pipelines

About the role

We are looking for a Data Engineer to design, build and own both batch and streaming data pipelines for our AI‑driven platform that serves an underserved workforce. You will work on end‑to‑end data ingestion, processing, storage and quality assurance in a fast‑moving Singapore‑based tech company.

Key responsibilities

• Develop and maintain batch and streaming ingestion pipelines using Kafka, Spark or Flink.
• Orchestrate ETL/ELT workflows with Airflow, Prefect or Dagster.
• Design schemas and manage storage across PostgreSQL, S3/Parquet and columnar warehouses (BigQuery, Snowflake, ClickHouse).
• Implement feature stores and dataset versioning with DVC or LakeFS.
• Process audio and text data for Indic languages, including resampling, VAD, transcription alignment and tokenization.
• Ensure DPDP‑compliant handling of PII, including partitioning, retention and lineage.
• Apply data‑quality checks such as validation, deduplication and drift detection using Great Expectations.

Required profile

• 4+ years of professional data‑engineering experience.
• Strong ownership of production pipelines (batch and streaming).
• Expertise in distributed processing frameworks (Spark, Flink, Beam) and workflow orchestration.
• Deep understanding of data modeling, partitioning, indexing and normalization trade‑offs.
• Experience with audio/speech or NLP pipelines and change‑data‑capture (Debezium).
• Ability to work with infrastructure‑as‑code tools.

Required skills

• Python
• SQL (window functions, query optimisation)
• Kafka
• Spark / Flink / Beam
• Airflow / Prefect / Dagster
• PostgreSQL
• S3, Parquet
• DVC, LakeFS
• Great Expectations
• Debezium
• BigQuery, Snowflake, ClickHouse
• Message queues
• Infrastructure‑as‑code (e.g., Terraform)