Data Engineer

Job Title : Data Engineer Python & PySpark - Remote

Working Hours : Working full-time in UK hours.

Skills : PySpark, Azure Databricks, real time streaming data.

About Company :

We're a UK-based talent partner for small to medium sized businesses. We work with clients in the UK, the United States, South Africa, Australia and across Europe, helping them access and enjoy the benefits of global talent.

Role Overview :

We would ideally like someone who is very tech savvy, has PySpark experience, real time streaming and ideally experience with trading data or teams.

Role & Responsibilities :

Streaming Data Expertise :

- Proven experience designing, building, and maintaining real-time data streaming solutions using modern event-driven architectures.

- Strong understanding of low-latency data processing for energy trading, market data feeds, telemetry, and operational systems.

Python & PySpark Proficiency :

- Advanced hands-on experience with Python and PySpark for large-scale data engineering, transformation, and analytics workloads.

- Ability to optimize distributed processing pipelines and develop robust, production-grade data solutions.

Business Engagement & Solution Design :

- Strong communication and stakeholder management skills with the ability to translate complex business requirements into scalable technical solutions.

- Experience working directly with trading, risk, operations, and business teams to deliver measurable business value.

Data Governance & Medallion Architecture :

- Deep understanding of data governance principles, including data quality, lineage, cataloging, security, and compliance.

- Proven experience implementing and managing Medallion Architecture (Bronze, Silver, Gold) to support trusted and scalable analytics platforms.

Security-First Engineering Mindset :

- Designs and implements data platforms based on rigorous security principles, including least-privilege access, data protection, auditing, and regulatory compliance.

- Ensures security is embedded throughout the data lifecycle and solution architecture.

Independent Delivery & Ownership :

- Self-motivated and capable of working independently with minimal supervision.

- Demonstrates strong ownership, problem-solving ability, and the capacity to drive initiatives from design through implementation and operational support.

Scope of Work :

- Deliver PySpark-based development against defined requirements.

- Work within an existing Fabric-based architecture.

- Amend and extend existing pipelines (not building from scratch).

- Support optimisation, fixes, and enhancements.

- Contribute to data transformation logic for multiple data flows.

- Support export of data to downstream systems.

Skills

Data Engineering, Python, PySpark, Data Governance, Medallion Architecture, Data Quality, Data Lineage, Microsoft Fabric