Data Engineer

Responsibilities:

• Own the architecture and design of enterprise lakehouse-based data platforms, including Delta Lake, medallion architecture, and unified analytics layers.
• Serve as the senior technical authority on data platform design, guiding architecture, tooling, modelling, and engineering standards.
• Design and build ingestion, transformation, and orchestration pipelines using Python, SQL, PySpark, and workflow orchestration tools.
• Architect data mesh and data product strategies, defining domain ownership, data contracts, and self-service data consumption.
• Establish best practices across ingestion, transformation, modelling, quality, observability, governance, and data consumption.
• Drive integration with AI/ML capabilities, including feature engineering pipelines, vector data infrastructure for RAG, and ML lifecycle management.
• Lead complex data migration, platform modernisation, and consolidation initiatives.
• Evaluate emerging technologies such as Apache Iceberg, Delta Lake, Apache Hudi, Unity Catalogue, Delta Live Tables, and Mosaic AI to inform architectural decisions.
• Solve complex enterprise-scale data architecture challenges across multiple teams and platforms.
• Drive cross-functional collaboration across data engineering, analytics, AI/ML, platform engineering, security, and product teams.
• Mentor senior data engineers and analysts.
• Engage with client stakeholders on data strategy and technical roadmaps.
• Define and enforce governance, compliance (GDPR, HIPAA, SOC2), and responsible data management practices.
• Drive FinOps maturity through compute optimisation, storage lifecycle management, cluster governance, and cost optimisation.

Requirements:

• 11+ years of experience in data engineering, analytics engineering, or data architecture.
• Mandatory hands-on expertise in Python, SQL, and enterprise lakehouse platforms.
• Strong knowledge of the modern data stack: dbt, Airflow/Dagster, Spark, Kafka, Fivetran/Airbyte, and cloud-native data services.
• Expertise in data modelling (Kimball, Data Vault, Activity Schema, OBT).
• Experience with cloud platforms (AWS, Azure, or GCP).
• Strong understanding of data governance, quality, lineage, cataloguing, and compliance.
• Experience supporting AI/ML workloads, including feature engineering, vector data, and model pipelines.
• Proven technical leadership and stakeholder management.

Preferred Qualifications:

• Experience architecting enterprise data platforms supporting LLMs, RAG pipelines, and Agentic AI.
• Deep expertise in Delta Lake, Apache Iceberg, and Apache Hudi.
• Experience implementing data mesh and federated data governance.
• Experience with Kafka, Flink, and Spark Structured Streaming.
• Experience with Unity Catalogue, Delta Live Tables, MLflow, Mosaic AI, and workflow orchestration.
• Background in financial services, healthcare, SaaS, or enterprise consulting.
• Experience leading geographically distributed engineering teams.

Must-Have Skills (Non-Negotiable):

• Lakehouse Platform Architecture: Hands-on architecture and engineering experience on a production-grade, enterprise-scale lakehouse platform.
• Python: Strong professional proficiency for data engineering and pipeline development.
• SQL: Advanced proficiency for data modelling, transformation, and performance tuning.
• Candidates without demonstrable hands-on experience in all three of the above will not be considered.