Lead AI Data Engineer

Job Summary

A senior data engineer who builds the pipelines and data infrastructure that feed AI/ML and agentic applications, owning ingestion through to retrieval-ready, feature-ready data.

Key Responsibilities

Must-have

• Production data pipelines at scale. You've built and operated batch and/or streaming pipelines that other teams depend on, and owned them through schema changes, backfills, and on-call.
• Spark and Databricks (or equivalent distributed compute). Comfortable tuning jobs for skew, shuffle, partitioning, and memory under real workloads, and working within a lakehouse platform like Databricks (Delta, Unity Catalog, workflows).
• Object stores as the data backbone. S3 / ADLS / MinIO, with strong grasp of partitioning, file formats (Parquet / Delta / Iceberg), compaction, and the cost and performance tradeoffs that come with them.
• Data integration across heterogeneous sources. Databases, APIs, event streams, and files stitched into reliable, coherent flows. This is the core of the role.
• Unstructured data. Parsing, chunking, and normalizing documents and multimodal content into something AI systems can actually consume.
• System design. You reason clearly about throughput, latency, consistency, idempotency, and cost, and can defend your decisions.

Strongly preferred

• Feature engineering for ML, building and serving features with offline/online consistency.
• Streaming and big-data stack, including Kafka / Kinesis / Pulsar, Flink, and lakehouse patterns.
• Open-source contributions, a signal of depth beyond day-job delivery.
• Coach and mentor our team as we build scalable machine learning solutions
• Strong communication skills and an easy-going attitude
• Build and manage strong relationships with stakeholders and various teams internally and externally

You'll pick up on the job (preferred, not required)

• RAG pipelines, including embedding generation, vector stores, and retrieval quality.
• Agentic integrations, the tool and data interfaces for autonomous AI workflows.

Strategy

• As the Squad member of AI team, the candidate is expected to participate and drive deliverables associated with Business Use cases.

Business

• Understand the Business requirement and execute the solutioning and ensue the delivery commitments are delivered on time and schedule.

Processes

• Design and Delivery of AI ML Use cases
• RAI, Security & Governance
• Model Validation & Improvements
• Stakeholder Management

People & Talent

• Manage the team in terms of project assignments and deadlines
• Manage a team dedicated for reviewing models related unstructured and structured data.
• Hire, nurture talent as required.

Risk Management

• Ownership of the delivery, highlighting various risks on a timely manner to the stakeholders.
• Identifying proper remediation plan for the risks with proper risk roadmap.

Governance

• Awareness and understanding of the regulatory framework, in which the Group operates, and the regulatory requirements and expectations relevant to the role.

Regulatory & Business Conduct

• Display exemplary conduct and live by the Group’s Values and Code of Conduct.
• Take personal responsibility for embedding the highest standards of ethics, including regulatory and business conduct, across Standard Chartered Bank. This includes understanding and ensuring compliance with, in letter and spirit, all applicable laws, regulations, guidelines and the Group Code of Conduct.
• Effectively and collaboratively identify, escalate, mitigate and resolve risk, conduct and compliance matters.

Key Stakeholders

• Business Stakeholders
• AIML Engineering Team
• AIML Product Team
• Product Enablement Team
• SCB Infrastructure Team
• Interfacing Program Team

Skills and Experience

• Build and maintain pipelines using Python, SQL, Spark
• Write ETL/ELT ingestion into Delta Lake or Iceberg tables
• Implement data quality checks and route failures
• Work with cloud storage and compute (ADLS, S3, Spark)
• Prepare feature-ready datasets for ML and embedding pipelines
• Support data warehouse and data lake environments
• Work with AI platform and data science teams on data needs
• SQL
• Python
• ETL and ML concepts
• Apache Spark
• Azure/AWS fundamentals
• Datawarehouse
• Datalake knowledge
• Distributed systems design, and architecture.
• Good to have: Flink, Vector

Qualifications

• Masters with specialisation in Technology with certification in AI and ML
• 6-10 years relevant of Hands-on Experience in developing and delivering AI solutions

About Standard Chartered

We're an international bank, nimble enough to act, big enough for impact. For more than 170 years, we've worked to make a positive difference for our clients, communities, and each other. We question the status quo, love a challenge and enjoy finding new opportunities to grow and do better than before. If you're looking for a career with purpose and you want to work for a bank making a difference, we want to hear from you. You can count on us to celebrate your unique talents and we can't wait to see the talents you can bring us.

Our purpose, to drive commerce and prosperity through our unique diversity, together with our brand promise, to be here for good are achieved by how we each live our valued behaviours. When you work with us, you'll see how we value difference and advocate inclusion.

Together we:

• Do the right thing and are assertive, challenge one another, and live with integrity, while putting the client at the heart of what we do
• Never settle, continuously striving to improve and innovate, keeping things simple and learning from doing well, and not so well
• Are better together, we can be ourselves, be inclusive, see more good in others, and work collectively to build for the long term

What we offer

In line with our Fair Pay Charter, we offer a competitive salary and benefits to support your mental, physical, financial and social wellbeing.

• Core bank funding for retirement savings, medical and life insurance, with flexible and voluntary benefits available in some locations.
• Time-off including annual leave, parental/maternity (20 weeks), sabbatical (12 months maximum) and volunteering leave (3 days), along with minimum global standards for annual and public holiday, which is combined to 30 days minimum.
• Flexible working options based around home and office locations, with flexible working patterns.
• Proactive wellbeing support through Unmind, a market-leading digital wellbeing platform, development courses for resilience and other human skills, global Employee Assistance Programme, sick leave, mental health first-aiders and all sorts of self-help toolkits
• A continuous learning culture to support your growth, with opportunities to reskill and upskill and access to physical, virtual and digital learning.
• Being part of an inclusive and values driven organisation, one that embraces and celebrates our unique diversity, across our teams, business functions and geographies - everyone feels respected and can realise their full potential.