Lead Data Engineer

About the role

The Lead Data Engineer is responsible for the bank's target data platform and the data engineering team. The scope includes the ingestion, processing, storage, and serving layers; batch and streaming pipelines; data contracts; governance; and SLAs for data freshness, completeness, and correctness.

Responsibilities

• Design and evolve the target lakehouse/DWH architecture, from ingestion to serving.
• Own the architecture of batch and streaming pipelines and the standards for their implementation.
• Build streaming-first integrations with banking domains using Kafka.
• Implement change data capture (CDC) from the core banking system (CBS) and other operational systems, with controls for completeness, ordering, and redelivery.
• Develop the Data Lake, ClickHouse, analytical data marts, and the serving layer for BI, ML, and AI agents.
• Design real-time data marts and streams for fraud prevention and risk management.
• Implement data contracts, a schema registry, schema compatibility rules, and a change management process.
• Ensure data lineage, a data catalog, data quality monitoring, and classification of personally identifiable information (PII).
• Define SLAs/SLOs for data freshness, completeness, and correctness, and coordinate the response to breaches.
• Develop and maintain ETL/ELT pipelines using Airflow, dbt, Spark, and Kafka.
• Optimize performance, storage and compute costs, the data lifecycle, and data consumption.
• Implement CI/CD, automated testing, and observability for data pipelines.
• Develop data mart architecture for analytics, feature stores, ML/GenAI, and RAG use cases.
• Manage the data engineering team, including planning, hiring, performance reviews, professional development, and technical mentoring.
• Collaborate with Analytics, ML, Product, Engineering, Security, and banking domain owners.

Technology context

• Kafka, schema registry, and CDC.
• Airflow, dbt, and Spark.
• ClickHouse MergeTree, S3 Data Lake, and lakehouse/DWH.
• SQL, Python, and/or Scala.
• Kubernetes and GitLab CI/CD.
• OpenTelemetry, Prometheus, Grafana, and dedicated data quality monitoring.
• Data contracts, lineage, catalogs, and PII classification.
• PostgreSQL and other operational data sources from banking domains.

Requirements

What we are looking for

• At least three years of hands-on experience in Data Engineering and a proven track record of technical leadership.
• Experience designing production data platforms, from ingestion through data marts and serving.
• Excellent SQL and Python skills.
• Hands-on experience with Apache Airflow, Spark, and Kafka.
• Experience with batch and streaming processing, CDC, and handling late and duplicate events.
• An understanding of lakehouse/DWH approaches, data modeling, and data platform layers.
• Practical experience with data contracts, schema evolution, and schema registries.
• Experience implementing CI/CD, testing, and monitoring for data pipelines.
• An understanding of data governance, lineage, catalogs, data quality, and PII protection.
• Experience managing a team or a sustained track record of technical leadership across multiple data projects.
• English B2 level or above.

Nice to have

• Experience with ClickHouse MergeTree and optimizing high-load analytical queries.
• Practical experience with MLOps, feature stores, and collaborative development with ML engineers.
• Experience with production GenAI/RAG pipelines and an understanding of their data and observability requirements.
• Experience building real-time data streams for fraud prevention, risk management, or other critical decisions.