Staff Engineer – AI Data Engineering

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

We are looking for an experienced AI Data Engineering Lead to define and lead the engineering of enterprise-scale data platforms and data products that power Artificial Intelligence, Machine Learning, Generative AI and advanced analytics.

This individual will build the organisation’s AI-ready data foundation by bringing together modern data engineering, cloud platforms, batch and real-time processing, unstructured data, vector and semantic technologies, data governance, and AI/ML engineering practices.

The role requires a senior technology leader with 15+ years of experience, strong hands-on technical depth, architecture capability, and a proven ability to lead engineering teams and deliver complex data platforms at enterprise scale.

Key Responsibilities

AI Data Engineering Strategy & Architecture

· Define the enterprise AI Data Engineering strategy, architecture and engineering standards.

· Design scalable data architectures supporting AI/ML, GenAI, LLM, analytics and enterprise data products.

· Establish reusable architectural patterns for structured, semi-structured and unstructured data.

· Define patterns for data lakes, lakehouse architectures, data warehouses, streaming platforms, vector databases and AI knowledge layers.

· Drive architectural decisions covering performance, scalability, resiliency, security and cost.

AI-Ready Data Platforms

· Lead engineering of data platforms providing trusted and reusable data for ML models, LLMs, AI agents and GenAI applications.

· Build ingestion and transformation frameworks for data from databases, APIs, documents, events, logs and enterprise applications.

· Establish pipelines for batch, near-real-time and streaming data processing.

· Drive automated data validation, quality controls, metadata management and lineage.

· Establish engineering approaches for preparing text, documents, images and other unstructured data for AI consumption.

GenAI & LLM Data Engineering

· Provide technical leadership for Retrieval-Augmented Generation (RAG), document ingestion, parsing, chunking and enrichment.

· Lead embedding generation and management, vector and hybrid search, semantic retrieval and reranking.

· Design knowledge repositories, metadata and access-control filtering, and enterprise search integration.

· Create data pipelines for LLM evaluation, AI grounding, authorised retrieval, fine-tuning where applicable, and AI agents.

· Understand how poor retrieval, incomplete or stale data, weak metadata and inappropriate chunking affect enterprise AI reliability.

Modern Data Engineering

· Provide leadership across ETL/ELT, distributed processing, lakehouse and data warehouse architectures, streaming, event-driven design, data APIs and data services.

· Drive data modelling, data contracts, observability, quality engineering, lineage and metadata management.

· Embed Infrastructure-as-Code, CI/CD, DevOps and DataOps practices in the engineering lifecycle.

· Improve reusability, automation, reliability and developer productivity through common frameworks and standards.

Cloud & Technology Platforms

· Demonstrate significant expertise across modern cloud and data platforms such as Microsoft Azure, Azure Databricks, Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Snowflake and Databricks.

· Bring strong capability in Python, SQL, Apache Spark/PySpark, Kafka, Delta Lake, Airflow and dbt.

· Apply AI and GenAI data technologies including Azure AI Search, vector databases, embedding models, semantic search, RAG frameworks, MLflow and feature stores.

· Experience with Azure OpenAI, enterprise LLM platforms and agent architectures is highly valuable. Deep expertise is expected across a relevant enterprise-scale stack, rather than every technology listed.

Data Governance, Security & Responsible AI

· Embed data governance, privacy, security and regulatory controls into data engineering architecture.

· Implement appropriate access controls for sensitive and confidential information.

· Ensure lineage, traceability and auditability across AI data pipelines.

· Ensure AI solutions retrieve and process only appropriately authorised data.

· Partner with Cybersecurity, Risk, Compliance, Data Governance and AI Governance teams, while promoting Responsible AI engineering practices.

· Experience handling PII, sensitive customer information, data residency and regulatory requirements is highly valuable.

Data Quality & Observability

· Define enterprise standards for AI data quality and automated validation across ingestion, transformation and retrieval pipelines.

· Monitor data freshness, completeness, accuracy, schema changes and pipeline failures.

· Drive observability across data pipelines and AI knowledge systems.

· Establish measurable engineering SLAs and SLOs for critical data services.

Engineering Leadership

· Lead and mentor senior data engineers, AI engineers and platform engineers.

· Provide technical direction across multiple engineering squads and programmes.

· Establish engineering standards, reference architectures and reusable frameworks.

· Conduct architecture and engineering design reviews, and resolve complex technical issues.

· Build capability through coaching, communities of practice and technical mentoring.

· Partner with Enterprise Architecture, Cloud, Cybersecurity, Data, AI/ML and application engineering teams while remaining technically credible and sufficiently hands-on.

Required Experience

· 15+ years of overall technology and engineering experience, with substantial depth in data engineering, data platforms or related disciplines.

· Significant experience leading enterprise-scale data engineering programmes and teams.

· Strong experience designing and implementing cloud-native data platforms.

· Deep understanding of distributed data processing and modern data architectures.

· Demonstrated experience supporting AI/ML or GenAI workloads through data engineering.

· Strong understanding of structured and unstructured data processing.

· Experience with enterprise-grade security, governance and data-quality practices.

· Strong coding capability, particularly in Python, SQL and PySpark or equivalent technologies.

· Experience with CI/CD, DevOps/DataOps and automated engineering practices.

· Proven ability to operate across architecture, engineering and senior stakeholder environments.

Preferred AI-Specific Experience

· GenAI and LLM solutions

· RAG architectures, vector databases and embedding pipelines

· Semantic and hybrid search, knowledge graphs and AI agents

· Feature engineering and feature stores

· MLOps and LLMOps

· Model evaluation datasets and data preparation for fine-tuning

· Unstructured-document processing

· Responsible AI and AI governance

Leadership Competencies

· Technology Vision: Converts AI strategy into practical data-engineering architecture.

· Engineering Excellence: Drives scalable, resilient and maintainable engineering practices.

· AI Mindset: Understands how data architecture changes for GenAI, LLMs and AI agents.

· Enterprise Thinking: Designs reusable capabilities rather than point solutions.

· Technical Leadership: Guides senior engineers and architects through complex technology decisions.

· Stakeholder Influence: Communicates complex concepts effectively to senior business and technology stakeholders.

· Execution: Converts architecture into production-grade platforms and measurable outcomes.

· Talent Development: Builds engineering capability and develops future technical leaders.

Education & Certifications

· Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, Information Technology or a related discipline.

· Relevant certifications from Microsoft Azure, Databricks or equivalent cloud/data technology providers are desirable.

· Advanced AI/ML, Data Engineering or Cloud Architecture certifications are advantageous.

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex, race, age, ethnic origin, educational, social and cultural background, marital status, pregnancy and maternity, religion or belief, disability or part-time / fixed-term work, or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade, job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.