Python & Data Engineering Associate

About the role

We are building a data collection and storage infrastructure from the ground up and need a Python‑focused Data Engineer to design, implement and maintain it. You will integrate third‑party APIs, create custom web crawlers, and shape a clean, unified data store ready for analysis.

Key responsibilities

• Integrate multiple third‑party data APIs, handling authentication, rate limiting, pagination, retries and varying response schemas.
• Develop custom web crawlers using tools such as Scrapy, BeautifulSoup or Playwright for sources without APIs.
• Design, implement and manage a relational database schema to store collected data reliably.
• Build standardisation and normalisation logic to transform raw data from diverse sources into a consistent internal structure.
• Handle deduplication and data‑quality management across all inputs.
• Create scheduled and on‑demand data collection pipelines using APScheduler, cron or equivalent.
• Write clean, well‑documented Python code and maintain the data infrastructure over time.

Required profile

• 2+ years of professional experience using Python as the primary development language.
• Proven track record building end‑to‑end data collection pipelines (APIs, crawlers, storage).
• Strong ownership mindset and ability to work independently on a long‑term project.

Required skills

• Python
• REST API integration
• Scrapy, BeautifulSoup, Playwright (web crawling/scraping)
• Relational database design and SQL
• Data normalisation and deduplication
• APScheduler or cron for scheduling
• Cloud basics (AWS, GCP or Azure)
• FastAPI or Flask (preferred)