Data Engineering Manager

The Ads Data team, part of Disney's Ad Platforms organisation, is on a mission to transform the advertising landscape across TV and streaming video through the power of data and AI. We design and build innovative solutions that measure and optimise the entire advertising lifecycle from planning and activation to measurement and insights. At the heart of our work is a highly collaborative, cross-functional team that delivers end-to-end solutions across a wide spectrum of technologies, including machine learning, big data, microservices, and data visualisation.

We are hiring a Product Dev Data Engineering Mgr to lead the teams responsible for building and scaling the data foundations that power AI adoption across Disney Ad Technology. In this role, you will guide how data is architected, governed, and delivered into AI-ready systems spanning streaming pipelines, embedding pipelines, vector stores, and monitoring frameworks while ensuring engineering teams execute with operational excellence.

You will partner closely with the AI Core Engineering team to deliver reliable, real-time data into shared AI agents, registries, developer tools, and platform services. As a people and technical leader, you will shape data engineering direction, manage team performance, and ensure your engineers deliver resilient, highquality pipelines that accelerate the development of high-profile AI applications across Ad Tech.

This role is ideal for leaders who combine deep technical experience with strong organisational leadership, someone who thrives on developing talent, driving clarity in ambiguous environments, and delivering data platforms that create measurable business impact. If you are energised by scaling teams, maturing systems, and building the foundation for the future of advertising, we'd love to meet you.

The candidate will have responsibilities across the following functions:

Execution, Delivery and Organisational Impact:

• Drive delivery of high-impact AI data initiatives by ensuring strong execution frameworks: sprint planning, estimation accuracy, roadmap forecasting, and predictable milestone delivery.
• Remove roadblocks, align priorities across partner teams, and ensure cross-functional collaboration results in timely and highquality outcomes.
• Manage team capacity, optimise resource allocation, and balance execution speed with longterm technical health and production stability.
• Collaborate with product, governance, and engineering leadership to ensure AI data systems evolve with business needs and regulatory requirements.

AI Data Foundations and Cross-Team Integration:

• Guide teams in designing and scaling streaming and batch data pipelines that power AI applications, ensuring low-latency, high-throughput, and high-reliability performance across Ad Tech.
• Oversee development of embedding pipelines, vector store integrations, and retrieval architectures to support semantic search, RAG systems, and AI assistants.
• Own the strategic roadmap for AI-optimized data stores, ensuring they integrate cleanly with enterprise-wide data architecture and evolving AI platform needs.
• Partner at the managerial level with AI Core Engineering, Infra, and Data Science.
• Product to ensure your teams deliver reliable, real-time data into shared agents, registries, and platform services.
• Drive adoption of reusable ingestion and transformation frameworks, enabling consistency and accelerating delivery across partner engineering teams.

Big Data Engineering:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Process large-scale datasets using Apache Spark and Apache Flink.
• Build distributed data processing applications using Scala, Python, or Java.
• Develop optimised ETL/ELT pipelines for structured and semi-structured data.
• Build and maintain modern Data Lake/Lakehouse architectures.

Data Processing:

• Develop high-performance Spark jobs for large-scale data transformation.
• Build streaming applications using Kafka and Apache Flink.
• Optimise Spark applications for memory utilisation, partitioning, caching, and shuffle performance.
• Process TB/PB-scale datasets efficiently.

Cloud and Data Platform:

• Work with Databricks, AWS EMR, Azure Databricks, or GCP Dataproc.
• Build scalable cloud-native data platforms.
• Design Delta Lake or Apache Iceberg-based architectures.
• Develop reusable data ingestion frameworks.

Pipeline Development:

• Build reusable ETL frameworks.
• Develop metadata-driven pipelines.
• Implement CDC (Change Data Capture).
• Build batch and near-real-time processing systems.
• Create fault-tolerant workflows.

Workflow Orchestration:

• Develop Airflow DAGs.
• Automate data pipelines.
• Schedule and monitor workflows.
• Handle pipeline failures and recovery.

Skills

AWS, Airflow, Apache Flink, Apache Spark, Databricks, ELT, EMR, ETL, Flink, GCP, Kafka, Spark, data, pipeline, pipelines