Python/Spark/AI Developer_TT
First seen by Alion on Nov 6, 2026.
We're looking for a mid-to-senior Python data engineer to join the data team of a fast-growing, privacy-focused fintech (Series A). The company is backed by a global credit bureau and a major retailer, and has teams across South Africa, the UK, the US, the UAE and Brazil.
Its platform lets banks and lenders work with retailer and telco data without raw personal data ever changing hands. The main goal is to help people with no credit history get access to credit.
Your purpose in this role is to lead the move of legacy SQL Server / Azure Synapse pipelines onto open-source Apache Spark and Delta Lake. You'll write production software, not notebook scripts.
Logistics
Location: fully remote, Europe
Working hours: South Africa time zone (UTC+2, the same as Kyiv and one hour ahead of Warsaw). Senior leadership is on US Eastern time.
Employment: full-time. Contract type (B2B or permanent) and salary band to be confirmed.
Level: Mid-Senior, about 4–8 years of experience
What you'll do
Rebuild legacy T-SQL stored-procedure logic as typed, tested PySpark pipelines
Prove that the new pipelines match the legacy outputs
Build FastAPI services that run and report on pipeline jobs
Add provider-neutral LLM features (local and hosted models) to the data workflows
Work in a spec-first codebase: design docs and ADRs, with tests and documentation shipped alongside every change
Core stack
Python, PySpark, Spark SQL, Delta Lake, FastAPI, pytest, Docker, T-SQL, LLM integration
Requirements
Python as your primary production language: modern typed Python (3.10+, ideally 3.12), with type hints that pass pyright or mypy, and Pydantic models
2+ years running production Apache Spark / PySpark pipelines, using the DataFrame API and Spark SQL, partitioning and performance tuning, and with a solid grasp of the driver/executor architecture
Maintainable, tested code (pytest), with tooling such as uv or Poetry and a CLI framework such as Click
Based in Europe or South Africa and happy to work UTC+2 hours
Strong advantages
Experience migrating SQL Server / Synapse / stored-procedure logic to Spark and proving output parity with the legacy system
Open-source or self-managed Spark (Spark on Kubernetes, EMR, Dataproc, Spark Connect), not only Databricks notebooks; Hive Metastore, Trino, Apache Ranger
FastAPI services with request validation, HMAC-signed auth, idempotency and job-status semantics
LLM features: prompt construction, structured-output validation, local models (Ollama, vLLM, llama.cpp), evaluation and guardrails
Privacy engineering: PII tokenisation and hashing, k-anonymity, GDPR/POPIA, multi-tenant isolation
Fintech, banking, credit-bureau, adtech-measurement or retail-data background
Observability: structlog, OpenTelemetry, OpenLineage
Azure (ADLS, Synapse) and a Databricks Certified Spark Developer credential