Senior Data Engineer

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics solutions using Azure and Databricks ecosystems, and to help set technical direction for our Data Engineering team.

Responsibilities:

• Own architecture and technical design decisions for client data platforms, and represent those solutions directly in client conversations.
• Set engineering standards for code review, testing, and CI/CD practices across the pipeline codebase, and mentor junior data engineers.
• Design and implement robust data pipelines using Azure Data Factory (ADF) and Databricks.
• Develop scalable ETL/ELT processes to ingest, transform, and load data from various sources.
• Write clean, efficient, and reusable Python code for data processing and automation.
• Optimise and manage SQL queries for data extraction, transformation, and reporting.
• Collaborate with business stakeholders to understand data requirements and deliver solutions aligned with business goals.
• Ensure data quality, integrity, and security across all data platforms.
• Monitor and troubleshoot data pipelines to ensure high availability and performance.
• Implement logging, alerting, and error-handling mechanisms for proactive issue resolution.
• Apply domain knowledge of trading and financial markets to enhance data models and analytics capabilities.

Requirements:

• 8+ years of experience in Data Engineering or related roles.
• Strong, hands-on experience with the Azure and Databricks tech stack: Azure Data Factory, Azure Databricks, Delta Lake, and Unity Catalogue.
• Strong programming skills in Python for data engineering tasks.
• Strong SQL skills and a solid understanding of relational databases.
• Strong communication and collaboration abilities, including the confidence to work directly with business and client stakeholders.
• Demonstrated experience owning architecture decisions and providing technical direction on data engineering projects.
• Production experience with real-time data processing frameworks (e. g., Spark Streaming, Kafka).
• Experience in maintaining and supporting production-grade data pipelines.
• Hands-on experience with Power BI, including data modelling, DAX, and building interactive dashboards.
• Experience with version control systems (e. g., Git) and CI/CD pipelines.
• Excellent problem-solving skills and attention to detail.

Good to have (not mandatory):

• Prior experience in ETRM/CTRM, energy, or commodity trading platforms, including market data, trade lifecycle, and financial instruments.
• Relevant certifications (e. g., Databricks Certified Data Engineer Associate/Professional, or current Microsoft Azure data engineering certification).

Skills

adf, azure, azure data factory, azure databricks, data engineer, data engineering, databricks, kafka, power bi, python, spark, sql