Python/Spark/AI Developer
First seen by Alion on Nov 3, 2026.
Responsibilities:
Data Engineering: Replatform T-SQL into Spark SQL and build production PySpark pipelines on Delta Lake.
API Development: Build secure, idempotent FastAPI services with robust job-status semantics and authentication.
Quality and Validation: Write type-hinted, tested Python (pytest) with CI gates; validate migrated pipelines against legacy systems with parity evidence.
Spec-First Delivery: Work autonomously from design docs and ADRs, keeping documentation current alongside code changes.
Minimum Requirements:
Experience: 4+ years in professional Python development, with 2+ years building production Spark/distributed data pipelines.
Education: Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
Certifications: Databricks Certified Developer for Apache Spark (or equivalent) is advantageous.
Python (3.12+): Modern idiomatic Python, strict static analysis (mypy/pyright), Pydantic, ABC patterns, modern packaging (uv/Poetry), and CLI tooling.
Apache Spark / PySpark: Production OSS Spark experience (DataFrame API, Spark SQL, tuning, driver/executor architecture, Spark Connect).
Lakehouse Formats: Delta Lake (or Iceberg/Hudi) including MERGE INTO, schema evolution, time travel, and idempotent writes.
SQL: Advanced, dialect-portable SQL skills to translate complex legacy T-SQL logic into Spark SQL.
Testing and CI/CD: Pytest (fixtures, tiered suites), Git/GitHub flow, PR-driven workflows with linting (ruff) and type-check gates.
APIs and Containers: FastAPI development and daily development using Docker Compose stacks.
- Preferred Skills
AI/LLM Engineering: Provider-neutral LLM integration, structured output validation, local/hosted inference, and guard-railing.
Governance and Security: PII tokenization, hashing schemes, and data privacy awareness (POPIA/GDPR).
Cloud and Platform: Azure data estate (ADLS Gen2, Key Vault), Trino, Hive Metastore, and OpenTelemetry observability.
Benefits:
Competitive salary based on experience (salary can potentially be more based on experience/skills)
IF you meet the above requirements and want to make a career-changing move, apply today by emailing your CV to [email protected]
Package & Remuneration R750 000 - R1 050 000 per annum About Us Hire Resolve is a top recruitment agency in South Africa, connecting businesses with talented professionals across a wide range of industries and locations.We help companies recruit and hire skilled professionals across Engineering, Mining, IT, Finance, Manufacturing, Construction, Supply Chain, Logistics, Renewable Energy, Sales, Marketing and more. Our recruitment expertise spans entry-level, specialist, senior management and executive positions.
With recruitment solutions covering major South African cities including Johannesburg, Cape Town, Pretoria, Durban, Gqeberha and Bloemfontein, as well as locations nationwide, we help businesses find the right talent wherever they need it.
Whether you're hiring for a single position or building an entire team, Hire Resolve provides tailored recruitment solutions designed around your hiring needs.
Ready to find your next hire? Get in touch with Hire Resolve today.