This job is no longer available
This job expired on 25/08/2026. It no longer accepts applications.
Data Engineer – Batch & Streaming Pipelines
Zillwork · Coimbatore
Job description
About the role
We are looking for a Data Engineer to design, build and own both batch and streaming data pipelines for our AI‑driven platform that serves an underserved workforce. You will work on end‑to‑end data ingestion, processing, storage and quality assurance in a fast‑moving Singapore‑based tech company.
Key responsibilities
- Develop and maintain batch and streaming ingestion pipelines using Kafka, Spark or Flink.
- Orchestrate ETL/ELT workflows with Airflow, Prefect or Dagster.
- Design schemas and manage storage across PostgreSQL, S3/Parquet and columnar warehouses (BigQuery, Snowflake, ClickHouse).
- Implement feature stores and dataset versioning with DVC or LakeFS.
- Process audio and text data for Indic languages, including resampling, VAD, transcription alignment and tokenization.
- Ensure DPDP‑compliant handling of PII, including partitioning, retention and lineage.
- Apply data‑quality checks such as validation, deduplication and drift detection using Great Expectations.
Required profile
- 4+ years of professional data‑engineering experience.
- Strong ownership of production pipelines (batch and streaming).
- Expertise in distributed processing frameworks (Spark, Flink, Beam) and workflow orchestration.
- Deep understanding of data modeling, partitioning, indexing and normalization trade‑offs.
- Experience with audio/speech or NLP pipelines and change‑data‑capture (Debezium).
- Ability to work with infrastructure‑as‑code tools.
Required skills
- Python
- SQL (window functions, query optimisation)
- Kafka
- Spark / Flink / Beam
- Airflow / Prefect / Dagster
- PostgreSQL
- S3, Parquet
- DVC, LakeFS
- Great Expectations
- Debezium
- BigQuery, Snowflake, ClickHouse
- Message queues
- Infrastructure‑as‑code (e.g., Terraform)
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in India.
Salaries by job title
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
Zillwork
Coimbatore
Related job offers
-
Senior Deployment Manager
Eurofins Coimbatore -
Principal Software Engineer - Java (eLIMS-Forensics)
Eurofins Coimbatore -
Senior Software Engineer (Java, Spring Boot, Azure)
Eurofins Coimbatore -
Développeur Front‑End UI/UX Scada (ReactJS) – GE Vernova
GE Vernova Paris -
Software Engineer - Backend Distributed Systems
salesforce