Senior Data Engineer

Responsibilities:

• Design, build, and maintain scalable data pipelines and Delta Lake architectures using Databricks on AWS.
• Develop and enhance data lakehouse/data warehouse solutions supporting front office trading and commodity analytics.
• Build production-grade ETL/ELT pipelines using Python, PySpark, and SQL.
• Work with data scientists, ML engineers, quants, and business stakeholders to prepare ML-ready datasets and support ML/AI pipelines.
• Perform Databricks administration, including cluster management, jobs, permissions, workflows, and platform resources.
• Implement and maintain CI/CD pipelines, automated testing, deployment automation, and monitoring for data engineering workflows.
• Implement MLOps practices using Databricks and MLflow, including model tracking, deployment, and monitoring.
• Provision and manage data infrastructure using Terraform / Infrastructure as Code (IaC).
• Integrate AWS services such as S3, Lambda, Glue, API Gateway, IAM, EC2, and EKS with Databricks.
• Participate in data modeling, architecture, code reviews, and cross-functional solutioning.
• Ensure data governance, security, access control, performance, reliability, and data quality standards.
• Optimize Databricks workloads, Spark jobs, clusters, and cloud infrastructure for performance and scalability.
• Identify opportunities to improve Databricks and AWS data platform capabilities.
• Support streaming and real-time data pipelines using Kafka or Spark Structured Streaming where required.
• Work closely with Front Office Trading and quantitative and analytics teams to deliver reliable data solutions.
• Take end-to-end ownership of data engineering solutions from design and development through deployment and production support.